Skip to content

Feature - Add support for ECS S3 Files integration - #4962

Merged
rsu206 merged 1 commit into
aws:devfrom
rsu206:s3files
May 28, 2026
Merged

rsu206 merged 1 commit into
aws:devfrom
rsu206:s3files

Conversation

@rsu206

@rsu206 rsu206 commented May 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This commit adds S3 Files volume support to the ECS Agent. It enables tasks to mount S3-backed file systems as NFS volumes via the ECS volume plugin, following the same architectural pattern used for EFS volumes. The agent advertises the storage.s3-files capability, receives S3 Files volume attachments from ACS, converts them into volume definitions using the ECS volume plugin, and manages their lifecycle with proper task-network dependencies.

Implementation details

New Attachment Type & Constants (ecs-agent/api/attachment/resource/)

  • Defines S3FilesTaskAttach = "s3files" as a new attachment type.
  • Adds property key constants: S3FilesVolumeIdKey, S3FilesVolumeNameKey, S3FilesRootDirectoryKey, S3FilesTransitEncryptionPortKey,
    S3FilesAccessPointIdKey.

S3 Files Volume Configuration (agent/taskresource/volume/dockervolume_s3files.go)

  • New S3FilesVolumeConfig struct holding volume ID, name, root directory, TLS port, access point ID, and internal Docker volume name.
  • ParseS3FilesTaskVolumeAttachment() — parses ACS attachment properties into the config, validates required fields (volumeId, volumeName).
  • GetVolumePluginDriverOptions() — builds the driver options map (type, device, o) for the ECS volume plugin. Mount options include tls, iam,
    awscredsuri, and optionally tlsport and accesspoint.

ACS Payload Handling (agent/acs/session/payload_responder.go)

  • getS3FilesAttachmentVolumeNames() extracts volume names from S3 Files attachments.
  • Before TaskFromACS deserialization, marks matching volumes as "attachment" type so UnmarshalJSON doesn't reject them.

Task Attachment Handler (agent/api/task/task_attachment_handler.go)

  • Routes s3files attachments to a dedicated handler that parses each into S3FilesVolumeConfig, appends it as a TaskVolume, and removes any
    pre-existing placeholder volume with the same name.

Volume Initialization (agent/api/task/task.go)

  • initializeS3FilesVolumes() iterates task volumes of type s3files, calls addS3FilesVolumes() which creates a VolumeResource via the ECS volume
    plugin with the driver options, registers it as a task resource, and sets up container volume dependencies.

Task Volume Serialization (agent/api/task/taskvolume.go)

  • Adds s3files case to UnmarshalJSON/MarshalJSON using the s3filesVolumeConfiguration JSON key.

Docker Volume Resource (agent/taskresource/volume/dockervolume.go)

  • S3FilesVolumeType = "s3files" constant.
  • S3 Files volumes depend on task network (like EFS) — DependOnTaskNetwork(), BuildContainerDependency(), and GetContainerDependencies() now include
    the S3FilesVolumeType.

Agent Capability (agent/app/agent_capability*.go)

  • Registers ecs.capability.storage.s3-files on Linux. Windows and unspecified platforms return a no-op.

ECS Volume Plugin (ecs-init/volumes/ecs_volume_plugin.go)

  • Registers "s3files" as a volume driver type using the same ECSVolumeDriver implementation as EFS.

Testing

This was tested manually on an EC2 instance. See following logs

level=info time=2026-05-08T01:15:09Z msg="Creating new volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:09Z msg="Creating mount target for new volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:09Z msg="Saving state of new volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:13Z msg="Returning volume information for ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:13Z msg="Returning volume information for ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:13Z msg="Received mount request &{Name:ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00 ID:6800b3e816b634cc7b6cda3ef77d09f9ccdbdb545981fb74b0d38efd14153901}"
level=info time=2026-05-08T01:15:13Z msg="Mounting volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00 as there are no existing mounts for it"
level=info time=2026-05-08T01:15:13Z msg="Validating create options for volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:13Z msg="Mounting volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00 of type s3files at path /var/lib/ecs/volumes/ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:14Z msg="Volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00 mounted successfully"
level=info time=2026-05-08T01:15:14Z msg="Adding mount 6800b3e816b634cc7b6cda3ef77d09f9ccdbdb545981fb74b0d38efd14153901 to volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:25Z msg="Returning volume information for ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:25Z msg="Received unmount request &{Name:ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00 ID:6800b3e816b634cc7b6cda3ef77d09f9ccdbdb545981fb74b0d38efd14153901}"
level=info time=2026-05-08T01:15:25Z msg="Removing mount 6800b3e816b634cc7b6cda3ef77d09f9ccdbdb545981fb74b0d38efd14153901 from volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00"
level=info time=2026-05-08T01:15:25Z msg="No active mounts left on volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00, unmounting it"
level=info time=2026-05-08T01:15:25Z msg="Unmounted volume ecs-S3Files-attach-1-s3files-aea4ebc5e189ece77e00 successfully."

New tests cover the changes:

New tests passed in preprod:

--- PASS: TestS3Files (1.16s)
    --- PASS: TestS3Files/rw-host (11.00s)
    --- PASS: TestS3Files/rw-br (11.06s)
    --- PASS: TestS3Files/rw-none (11.17s)
    --- PASS: TestS3Files/rw-avpc (21.65s)

Description for the changelog

Feature - Add support for ECS S3 Files integration

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions? No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@rsu206
rsu206 requested a review from a team as a code owner May 14, 2026 21:52
@rsu206
rsu206 force-pushed the s3files branch 2 times, most recently from 6004445 to 98dbb8c Compare May 19, 2026 22:33
amogh09
amogh09 previously approved these changes May 20, 2026
@ShelbyZ

ShelbyZ commented May 20, 2026

Copy link
Copy Markdown
Contributor
  1. Should we add the new capability to the existing test/check - https://github.com/aws/amazon-ecs-agent/blob/master/agent/app/agent_capability_unix_test.go#L913-L918

@rsu206

rsu206 commented May 27, 2026 •

Copy link
Copy Markdown
Contributor Author

New MACIS tests passed in preprod:

--- PASS: TestS3Files (1.16s)
    --- PASS: TestS3Files/rw-host (11.00s)
    --- PASS: TestS3Files/rw-br (11.06s)
    --- PASS: TestS3Files/rw-none (11.17s)
    --- PASS: TestS3Files/rw-avpc (21.65s)

@rsu206
rsu206 merged commit bcfd693 into aws:dev May 28, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants