[singholt]

Summary

This PR fixes the error that a container sees when it requests credentials over the TMDS credentials endpoint (169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI), when the credentials are unavailable.

The agent does not save task IAM role and task execution IAM role credentials to its state. So when an agent restart occurs, the agent needs ACS to resend the credentials. Until ACS does so, the agent must return a meaningful error code and reason in the response. Currently, customer would see a error code 400 (Invalid credentials ID), whereas they should see a 503 (credentials uninitialized).

The TMDS handler does have logic in place to throw 503 under such conditions but it is broken. This PR fixes it.

Implementation details

• Export and add a JSON tag to the CredentialsID in the task object. When the agent saves the task object in its state file, this change helps store the credentials ID field as well. This makes it consistent with how agent saves the execution role credentials ID to its state.
• Add a new map to the credentials manager to track known credentials IDs.
• When agent restarts and loads all the tasks from its state file, we now add known credential IDs to the credentials manager. This is done early in the agent initialization (before TMDS is up), so that the credentials manager knows when to return 503 vs 400.
• Updated the credentials manager interface with new helper methods that the TMDS handler and engine could use to (1) add known credentials ID and (2) determine whether a credentials ID is known but credentials have not yet arrived from ACS.

Testing

New tests cover the changes: yes

Added unit tests and also manually tested the scenario as described below.

Steps to reproduce the issue:

• Launch an ECS container instance and run an ECS task that has a task IAM role.
• Exec into the task container and query the TMDS credentials endpoint. You should get valid credentials in the response.
• Next, add the following iptables rules to block outbound traffic to ACS, and restart the ECS agent.

sudo iptables -A OUTPUT -d 0.0.0.0/0 -p tcp --dport 443 -m string --string "ecs-a" --algo bm -j DROP
sudo iptables -A OUTPUT -d 0.0.0.0/0 -p tcp --dport 80 -m string --string "ecs-a" --algo bm -j DROP

• When the agent restarts, it won't be able to establish a connection with ACS, hence the creds will never arrive. Querying the TMDS creds endpoint would yield:

{"code":"InvalidIdInRequest",
"message":"CredentialsV2Request: Credentials not found","HTTPErrorCode":400}

• Build a test agent using the changes in this PR and repeat the above steps. It now yields

{"code":"CredentialsUninitialized",
"message":"CredentialsV2Request: Credentials uninitialized for ID","HTTPErrorCode":503}

Description for the changelog

Bugfix: Fix tmds response for unavailable credentials upon agent restart.

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions? No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.