Skip to content

Inject NVIDIA MPS env vars and pipe mount into MPS containers - #5082

Merged
harishxr merged 1 commit into
aws:devfrom
harishxr:mps-ec2-runtime-injection
Aug 3, 2026
Merged

harishxr merged 1 commit into
aws:devfrom
harishxr:mps-ec2-runtime-injection

Conversation

@harishxr

@harishxr harishxr commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Injects the NVIDIA MPS per-client environment variables and the MPS control-pipe mount into MPS containers, so the limits parsed from a container's GPU sharingStrategy are actually applied at container launch.

A prior change parses a container's ACS GPU sharingStrategy into container.MPSConfig, but nothing yet acts on it. This wires that config through to the container: the MPS server can now enforce each container's memory and compute ceiling.

Implementation details

For any container with MPSConfig set container.UsesMPS():

  • Env injection populateGPUEnvironmentVariables alongside the existing NVIDIA_VISIBLE_DEVICES, the container receives:
    • CUDA_MPS_PIPE_DIRECTORY (always)
    • CUDA_MPS_PINNED_DEVICE_MEM_LIMIT, formatted 0=M from MPSConfig.Memory
    • CUDA_MPS_ACTIVE_THREAD_PERCENTAGE from MPSConfig.MaxComputePercent, only when the customer declared it. When unset, the var is omitted entirely so the MPS daemon default (100%) applies.
  • Pipe mount dockerHostConfig: a bind mount of the control-pipe directory /tmp/nvidia-mps is added. The NVIDIA container runtime's CDI spec injects only the MPS binaries, not the pipe directory, so without this mount the injected memory limit is silently unenforced (the runtime finds no pipe and runs without MPS).

Both hooks are gated on UsesMPS(), so whole-GPU and non-GPU containers are untouched.

The env-var names, the 0=M formatting, and the omit-when-undeclared rule live in a new shared package ecs-agent/utils/mps

Testing

Built a custom agent from this change and installed it on a AL2023 GPU AMI and ran some tasks carrying MPS limits (the payload was driven via a test-only mock, since the backend sharingStrategy path is not yet shipped). docker inspect confirmed the expected CUDA_MPS_* env vars (with compute omitted when undeclared) and the pipe-directory bind mount on MPS containers, and their absence on non-MPS containers. A workload with the memory limit set below the physical GPU memory was capped at the injected ceiling rather than the physical total, confirming the injected value is enforced.

New tests cover the changes: yes

Description for the changelog

Enhancement - Inject NVIDIA MPS env vars and pipe mount into MPS containers

Additional Information

**Does this PR include breaking model changes? If so, Have you added transformation functions?**No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@harishxr
harishxr marked this pull request as ready for review August 3, 2026 21:41
@harishxr
harishxr requested a review from a team as a code owner August 3, 2026 21:41
@harishxr
harishxr enabled auto-merge (rebase) August 3, 2026 22:24
@harishxr
harishxr merged commit be67b75 into aws:dev Aug 3, 2026
45 of 46 checks passed
@harishxr
harishxr deleted the mps-ec2-runtime-injection branch August 4, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants