Problem description
DR jobs failing with error AWS_DR128
Cause
AWS DR Proxy instance's date and time settings are inaccurate, it can result in a discrepancy between the date in the signature and the date of the request, leading to AWS rejecting your requests.
Pre-requisites:
Add a Firewall rule in your Compute Gateway which allows NTP traffic to 169.254.169.123
Services: NTP (UDP:123)
Steps to add Firewall Rule:
Click Security Groups under Security on the navigation pane after opening the Amazon VPC Console. Click Security Groups under Security on the navigation pane after opening the Amazon VPC Console
A list of security groups will be displayed. Select the security group you want to modify.
Click Actions
Configure Inbound Rules:
In the Inbound rules tab, click on Edit inbound rules.
Click on Add rule.
Set the Type to Custom UDP Rule.
Set the Port Range to 123.
Set the Source to a specific IP range (e.g., 0.0.0.0/0 for all IP addresses, but consider narrowing it down for better security).
Click Save rules.
Configure Outbound Rules:
In the Outbound rules tab, click on Edit outbound rules.
Click on Add rule.
Set the Type to Custom UDP Rule.
Set the Port Range to 123.
Set the Destination to a specific IP range (e.g., 0.0.0.0/0).
Click Save rules.
Traceback
Phoenix.log
[2024-03-12 03:15:42,459] [ERROR] [139823652820800] Failed to run command sudo ntpdate 169.254.169.123 with error : b'sudo: ntpdate: command not found\n'
[2024-03-12 03:15:42,459] [DEBUG] [139823652820800] Logging infra details of the proxy
[2024-03-12 03:15:42,466] [ERROR] [139823652820800] EC2Client : Failed to get EC2 instance details : error = An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials
[2024-03-12 03:15:42,466] [ERROR] [139823652820800] Error <class 'botocore.exceptions.ClientError'>:An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials. Traceback -Traceback (most recent call last):
File "/code/src/phoenix_client_lib/boto3/ec2client.py", line 2283, in get_ec2_instance_details
File "/code/src/phoenix_client_lib/boto3/ec2client.py", line 2236, in __get_bulk_ec2_instance_details
File "/usr/local/pyenv/versions/3.9.1/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
File "/usr/local/pyenv/versions/3.9.1/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call
botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials
[2024-03-12 03:15:42,466] [INFO] [139823652820800] AwsDRLib<retry_module>: not retrying any more since error is not retryable for Fn: get_ec2_instance_details
[2024-03-12 03:15:42,467] [ERROR] [139823652820800] Error <class 'inSyncLib.inSyncError.SyncError'>:AWS was not able to validate the provided access credentials (#100080080) (Error Code : AWS_DR128). Traceback -Traceback (most recent call last):
Resolution:
Connect to AWS DR proxy instance and use apt to install the chrony package. If you have already installed chrony, skip the install.
sudo apt install chronyOpen the
/etc/chrony/chrony.conffile using a text editor. (update the permission to this file using chmod command)Add the following line before any other server or pool statements that are already present in the file and save your changes.(attached is the screenshot for reference)
server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4Restart the chrony service.
sudo /etc/init.d/chrony restart
Reboot AWS DR Proxy for the changes to affect.
