Skip to content

MPS GPU sharing: advertise ecs.capability.gpu-sharing-mps - #5053

Merged
harishxr merged 1 commit into
aws:devfrom
harishxr:mps-gpu-sharing
Jul 23, 2026
Merged

harishxr merged 1 commit into
aws:devfrom
harishxr:mps-gpu-sharing

Conversation

@harishxr

@harishxr harishxr commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

TODO:

  • The capability advertisement to control plane has been commented out and will be added in once the rest of the MPS implementation is done.

Summary

Adds a new container instance capability, ecs.capability.gpu-sharing-mps, which the agent advertises when the instance is able to run GPU workloads sharing a single physical GPU through the NVIDIA Multi-Process Service (MPS) control daemon.

The agent advertises the capability only when all of the following hold:

  • an NVIDIA GPU is present,
  • the MPS control binary (/usr/bin/nvidia-cuda-mps-control) is installed,
  • the nvidia-mps.service systemd unit is enabled, and
  • the GPU is not an NVIDIA vGPU slice (vGPU does not support MPS).

If any condition is unmet the capability is withheld and the agent logs the specific reason. This pairs with the AL2023 GPU AMI change (aws/amazon-ecs-ami#744) that ships and enables nvidia-mps.service.

Implementation details

The facts are gathered on the host by ecs-init and consumed by the agent, mirroring how the existing NVIDIA driver version already flows:

  • ecs-init: ecs-init/gpu/nvidia_gpu_manager.go gathers three new facts during GPU setup and persists them in /var/lib/ecs/gpu/nvidia-gpu-info.json:
    • MPSControlBinaryPresent - stat of the MPS control binary,
    • MPSServiceEnabled - systemctl is-enabled nvidia-mps.service (keys off the printed state, since is-enabled exits non-zero for a known-but-disabled unit),
    • HasVGPU - NVML DeviceGetVirtualizationMode, folded into the existing device loop; fails open (only a definite VGPU result counts, so a probe error never strips MPS from a real GPU).
  • Shared decision (ecs-agent/utils/gpu/gpu.go) - a new package that takes the four boolean inputs and returns whether to advertise, plus the unmet conditions with failure reasons. Keeping the decision in the shared library lets agent variants reach the same verdict from the same inputs.
  • Agent - agent/gpu/nvidia_gpu_manager_unix.go reads the three new fields from the JSON and exposes them via getters; agent/app/agent_capability_unix.go builds the inputs (GPU-present derived from discovered devices), calls the shared decider, and appends capability.GPUSharingMps on success or logs each unmet reason otherwise. No-op stubs are added for the windows/unspecified builds.
  • The capability name constant lives in ecs-agent/capability/capability.go.

Testing

New tests cover the changes: yes

Verified end-to-end on g6.xlarge, covering all four gate conditions via DescribeContainerInstances and the agent log:

  • All conditions met (daemon enabled + active) - ecs.capability.gpu-sharing-mps advertised via DescribeContainerInstances
  • nvidia-mps.service not enabled - withheld - nvidia-mps systemd service is not enabled on host
  • MPS control binary absent - withheld - MPS control binary not present on host
  • vGPU instance -withheld - instance has an NVIDIA vGPU slice, where we do not support MPS

Description for the changelog

Feature - Add ecs.capability.gpu-sharing-mps capability for NVIDIA MPS GPU sharing

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions? No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@harishxr
harishxr marked this pull request as ready for review July 20, 2026 21:34
@harishxr
harishxr requested a review from a team as a code owner July 20, 2026 21:34
Comment thread ecs-init/gpu/nvidia_gpu_manager.go Resolved
Comment thread ecs-agent/utils/gpu/gpu.go Resolved
Comment thread ecs-init/gpu/nvidia_gpu_manager.go Resolved
Comment thread agent/app/agent_capability_unix.go Resolved
@harishxr
harishxr enabled auto-merge (rebase) July 22, 2026 18:13
@harishxr
harishxr merged commit d611de9 into aws:dev Jul 23, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants