⚽ Live Match: aws/amazon ecs agent/pull/

harishxr · 2026-07-24T05:32:23Z

Summary

ecs-init discovers per-GPU usable memory (NVML v2 Total - Reserved) during Setup and persists it as a UUID->MiB map in the GPU info file. The agent loads the map and populates PlatformDevice.GpuInfo.MemoryInMiB when building
the RegisterContainerInstance payload, so the control plane can size GPUs for MPS memory-based placement.

Implementation details

ecs-init (ecs-init/gpu/nvidia_gpu_manager.go): new DetectGPUMemory() iterates over each device and computes usable memory with NVML v2 (DeviceGetMemoryInfo_v2), usable = (Total - Reserved) / MiB. Results are stored as GPUMemoryMiB map[string]uint64 (UUID -> MiB) and serialized to the GPU info file. Called from Setup().
agent (agent/gpu/nvidia_gpu_manager_unix.go): reads the matching GPUMemoryMiB field from the info file in Initialize(), and SetDevices() sets PlatformDevice.GpuInfo = &GpuPlatformDeviceInfo{MemoryInMiB: ...} when a UUID has a known value. Left nil for GPUs without a value, so legacy hosts are unaffected.

Backward compatible and additive: the field is omitempty on both sides, so older agents/hosts that do not report it send nothing, and the control plane treats it as optional.

Testing

On host nvidia-gpu-info.json GPUMemoryMiB = 22563, matching nvidia-smi (23034 total - 471 reserved).

# sudo cat /var/lib/ecs/gpu/nvidia-gpu-info.json
{"DriverVersion":"580.159.03","GPUIDs":["GPU-cc922717-b2bb-b68f-d7da-7e2ac056c6fb"],"GPUMemoryMiB":{"GPU-cc922717-b2bb-b68f-d7da-7e2ac056c6fb":22563},"MpsControlBinaryPresent":true,"MpsServiceEnabled":true,"HasVGPU":false}[root@ip-172-31-37-203 bin]#

CloudTrail RegisterContainerInstance requestParameters.platformDevices[0] shows gpuInfo.memoryInMiB = 22563 from this agent build, confirming the field reaches the control plane on the wire.


"platformDevices": [
--
{
"id": "GPU-cc922717-b2bb-b68f-d7da-7e2ac056c6fb",
"type": "GPU",
"gpuInfo": {
"memoryInMiB": 22563
}
}
]

New tests cover the changes: yes

Description for the changelog

Enhancement - Report per-GPU memory at container instance registration for GPU sharing

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions? No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

ecs-init discovers per-GPU usable memory via NVML and the agent reports it in the PlatformDevice.gpuInfo field so the control plane can place GPU instances by available memory.

amogh09 · 2026-07-24T17:30:07Z

 			Type: types.PlatformDeviceTypeGpu,
-		})
+		}
+		if memMiB, ok := n.GPUMemoryMiB[gpuID]; ok {


What is the case when memory could be missing?

Potential cases:

NVML does not return a response due to some reason when we query for memory, memory will be missing in such a case.

Agent version is newer than init version, ecs-init will not be able to write the gpuMemory to the gpu info file where agent reads from. In that case, memory will be empty as well.

amogh09 · 2026-07-24T19:07:29Z

+			seelog.Errorf("Failed to get UUID for device at index %d during memory detection: %v", i, nvml.ErrorString(ret))
+			continue


We should think about how we can better surface such errors to customers. Silently registering less capacity is not ideal.

Can this mismatch be caught in DescribeContainerInstances response vs EC2 DescribeInstances or something like that?

Discussed offline

mye956 · 2026-07-27T16:33:15Z

+			seelog.Errorf("Unexpected v2 memory for device %s: total %d < reserved %d; skipping", uuid, mem.Total, mem.Reserved)
+			continue
+		}
+		memory[uuid] = (mem.Total - mem.Reserved) / bytesPerMiB


Q: In the case of a zero value here, what would the agent's behavior be here? (e.g. would we still advertise ecs.capability.gpu-sharing-mps?). Also got a similar question with Amogh on how users can discover this on their own.

Agent reporting zero value, would not gate the capability from being advertised. For the second part, discussed offline

prateekchaudhry · 2026-07-27T18:02:43Z

+// a workload can actually allocate under MPS.
+func (n *NvidiaGPUManager) DetectGPUMemory() map[string]uint64 {
+	memory := make(map[string]uint64)
+	count, err := NvmlGetDeviceCount()


We should not need this, as we have already done a GetGPUDeviceIDs which populates n.GPUIDs. We can just iterate over those UUIDs as source of truth and get the needed device handles. Minor simplification.

Or we can combine both methods too

Good suggestion, will look into this as a follow up

harishxr changed the title ~~Mps ec2 rci memory~~ MPS GPU sharing: report per-GPU memory at container instance registration Jul 24, 2026

harishxr added the bot/test label Jul 24, 2026

amazon-ecs-bot removed the bot/test label Jul 24, 2026

harishxr marked this pull request as ready for review July 24, 2026 05:45

harishxr requested a review from a team as a code owner July 24, 2026 05:45

harishxr force-pushed the mps-ec2-rci-memory branch from c6c8dc3 to 5fb5ba5 Compare July 24, 2026 16:55

harishxr added the bot/test label Jul 24, 2026

amazon-ecs-bot removed the bot/test label Jul 24, 2026

harishxr enabled auto-merge (rebase) July 24, 2026 17:19

Report per-GPU memory at container instance registration

cb88dff

ecs-init discovers per-GPU usable memory via NVML and the agent reports it in the PlatformDevice.gpuInfo field so the control plane can place GPU instances by available memory.

harishxr force-pushed the mps-ec2-rci-memory branch from 5fb5ba5 to cb88dff Compare July 24, 2026 17:23

harishxr changed the title ~~MPS GPU sharing: report per-GPU memory at container instance registration~~ Report per-GPU memory at container instance registration Jul 24, 2026

amogh09 approved these changes Jul 24, 2026

View reviewed changes

harishxr added the bot/test label Jul 24, 2026

amazon-ecs-bot removed the bot/test label Jul 24, 2026

mye956 reviewed Jul 27, 2026

View reviewed changes

prateekchaudhry reviewed Jul 27, 2026

View reviewed changes

mye956 approved these changes Jul 27, 2026

View reviewed changes

harishxr merged commit 6f361e9 into aws:dev Jul 27, 2026
45 checks passed

This was referenced Jul 29, 2026

fail agent bootstrap when per-GPU memory cannot be read #5069

Closed

Gate gpu-sharing-mps on complete per-GPU memory #5075

Merged

harishxr deleted the mps-ec2-rci-memory branch August 4, 2026 00:36

Navigation Menu

Repository navigation

Report per-GPU memory at container instance registration - #5061

Report per-GPU memory at container instance registration#5061
harishxr merged 1 commit into
aws:devfrom
harishxr:mps-ec2-rci-memory

harishxr commented Jul 24, 2026 •

edited

Loading

Uh oh!

amogh09 Jul 24, 2026

Uh oh!

harishxr Jul 24, 2026

Uh oh!

amogh09 Jul 24, 2026

Uh oh!

amogh09 Jul 24, 2026

Uh oh!

harishxr Jul 24, 2026

Uh oh!

mye956 Jul 27, 2026

Uh oh!

harishxr Jul 27, 2026

Uh oh!

prateekchaudhry Jul 27, 2026

Uh oh!

prateekchaudhry Jul 27, 2026

Uh oh!

harishxr Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		seelog.Errorf("Failed to get UUID for device at index %d during memory detection: %v", i, nvml.ErrorString(ret))
		continue

Conversation

harishxr commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation details

Testing

Description for the changelog

Additional Information

Licensing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

harishxr commented Jul 24, 2026 •

edited

Loading