Skip to content

engine/serviceconnect: read image digest from metadata json - #5006

Merged
singholt merged 1 commit into
aws:devfrom
nade007:sc-sidecar-image-digest
Jun 16, 2026
Merged

singholt merged 1 commit into
aws:devfrom
nade007:sc-sidecar-image-digest

Conversation

@nade007

@nade007 nade007 commented Jun 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Service Connect sidecar container is loaded from a tarball at /managed-agents/serviceconnect/ecs-service-connect-agent.interface-v1.tar. Because it never gets pulled from a registry, Docker has no RepoDigests for it, so the agent reports an empty ImageDigest in SubmitContainerStateChange. This means consumers downstream to the ECS controlplane cannot consume them.

This change makes the agent read a sidecar JSON file at /managed-agents/serviceconnect/ecs-service-connect-agent.interface-v1.metadata.json containing {"schemaVersion":1,"digest":"sha256:..."}, cache the digest on the manager struct at LoadImage time, and stamp it on the SC container in augmentAgentContainer. The existing state change pipeline carries it the rest of the way to control plane.

The metadata.json is shipped by the SC RPM (the file is produced upstream in the SC release pipeline that pushes the image to ECR, captured via docker inspect --format '{{index .RepoDigests 0}}' right after pull, and bundled into the RPM next to the tarball).

Implementation details

  • manager.go: one new method on the interface, GetLoadedImageDigest() string.
  • manager_linux.go: a constant for the metadata path, a struct field appnetImageDigest, a helper readLoadedImageDigest with graceful failure, the call site in LoadImage, the public getter, and the SetImageDigest call inside augmentAgentContainer.
  • manager_other.go: a no-op stub returning "" for non-Linux. SC is Linux only anyway.
  • mock/manager.go: gomock entries for the new method (manually edited to match what mockgen would produce, the existing go:generate directive in generate_mocks.go regenerates it identically).
  • manager_linux_test.go: two new tests covering the read fn (4 subcases including missing file and malformed JSON) and the augment plumbing (digest set when present, untouched when empty).

Backward compatibility

If the file is absent (older RPM that predates this work), malformed, or has an empty digest field,readLoadedImageDigest logs a warning and returns "". The if d != "" guard in augmentAgentContainer then skips the SetImageDigest call entirely. SC sidecar still launches normally, just with empty digest, which is the existing behavior on those AMIs.

The metadata file is consumer-facing but optional. Agent works the same as today on hosts that don't have it.

Testing

Unit tests:

  • TestReadLoadedImageDigest covers the four shapes of metadata.json (valid, missing, malformed, empty digest field).
  • TestAugmentTaskContainerSetsImageDigest verifies the plumbing both when the manager has a cached digest and when it doesn't.
  • Full SC manager test suite passes (go test -tags "linux unit" ./engine/serviceconnect/...).
  • go build ./... clean across the whole agent module.
  • gofmt, go vet clean.

End to end on a real EC2:

1. Install the SC RPM that ships metadata.json alongside the tarball on a fresh amazon-ecs-optimized-amazon-linux-2 AMI host. After install, /var/lib/ecs/deps/serviceconnect/ contains the tar, the metadata.json ({"schemaVersion":1,"digest":"sha256:..."}), and the ecs-service-connect-agent.tar symlink.

2. Build the patched agent with the linker flags the release pipeline uses. Without these the binary panics at startup because DefaultPauseContainerImageName and DefaultPauseContainerTag are blank:

go build -ldflags '\
  -X github.com/aws/amazon-ecs-agent/agent/config.DefaultPauseContainerTag=0.1.0 \
  -X github.com/aws/amazon-ecs-agent/agent/config.DefaultPauseContainerImageName=amazon/amazon-ecs-pause' \
  -o out/amazon-ecs-agent ./

3. Swap the agent binary on EC2.

4. Agent log on first load confirms the new appnetImageDigest field is populated, matching what docker inspect --format '{{index .RepoDigests 0}}' returns for the same tag in ECR:

level=info msg="Successfully loaded Appnet agent container tarball: /managed-agents/serviceconnect/ecs-service-connect-agent.interface-v1.tar"
  appnetImageDigest="sha256:8bdb42f6a48b7422f5eee379a6522a446850269001e9432cea58296f5fbb1169"
  image="ecs-service-connect-agent:interface-v1"

5. Launch a real SC-enabled task. Once the task transitions to RUNNING:

aws ecs describe-tasks --cluster default --tasks <task-id> \
  --query 'tasks[0].containers[].[name,imageDigest,lastStatus]'
web                            sha256:c9001ce8...   RUNNING
ecs-service-connect-XXXXXX     sha256:8bdb42f6...   RUNNING   ← was empty before this change

The SC sidecar reports the registry manifest digest from metadata.json end to end through the agent into the ECS API.

With metadata.json present (sidecar shipped by current RPM):

    level=info msg="Successfully loaded Appnet agent container tarball: ..."
      image="ecs-service-connect-agent:interface-v1"
appnetImageDigest="sha256:8bdb42f6a48b7422f5eee379a6522a446850269001e9432cea58296f5fbb1169"

Without metadata.json (older RPM):

    level=warn msg="AppNet metadata sidecar unavailable"
      error="open /managed-agents/serviceconnect/ecs-service-connect-agent.interface-v1.metadata.json: no such file or directory"
    level=info msg="Successfully loaded Appnet agent container tarball: ..."
      image="ecs-service-connect-agent:interface-v1"
      appnetImageDigest=""

New tests cover the changes: yes

Description for the changelog

Enhancement - Populate ImageDigest for Service Connect sidecar containers.

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions?
No

Does this PR include the addition of new environment variables in the README?
No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@nade007
nade007 force-pushed the sc-sidecar-image-digest branch from a46f409 to bc9dacd Compare June 12, 2026 01:46
@nade007
nade007 marked this pull request as ready for review June 12, 2026 02:58
@nade007
nade007 requested a review from a team as a code owner June 12, 2026 02:58
singholt
singholt previously approved these changes Jun 15, 2026

@singholt singholt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few nits

@mye956 mye956 changed the title engine/serviceconnect: read image digest from metadat json engine/serviceconnect: read image digest from metadata json Jun 15, 2026
@singholt
singholt merged commit 024663e into aws:dev Jun 16, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants