[harishxr]

Summary

Wires the MPS control daemon health gate onto MPS tasks. The gate resource landed in #5107, but nothing created it for a task. This PR attaches one MPSDaemonResource per MPS task and makes every MPS container wait for it before it can be created.

Implementation details

initializeMPSDaemonResource (agent/api/task/task.go) runs in PostUnmarshalTask, right after addGPUResource. If any container in the task uses MPS, it creates the resource via NewMPSDaemonResource, adds it to the task, and makes each MPS container's ContainerCreated transition depend on the resource reaching MPSDaemonCreated. So a daemon that is not serving blocks the task at container creation instead of letting a container start with a CUDA_MPS_PINNED_DEVICE_MEM_LIMIT that would be silently ignored. Tasks with no MPS container get no resource , and non-MPS containers in a mixed task are not gated.

Testing

New tests cover the changes: yes

• End to end on a gamma GPU instance:
  • daemon healthy: MPS task RUNNING, injected memory cap enforced.
  • daemon down: task STOPPED with reason "MPS control daemon is not responding", no container ever created.
  • daemon restarted within the retry window: task proceeds to RUNNING.
  • daemon down past the window: "no success in 3 attempts", STOPPED.
  • new MPS task after the daemon is healthy again: RUNNING.
  • whole-GPU and non-GPU tasks with the daemon down: unaffected, no gate.

Description for the changelog

Enhancement - Attach the MPS control daemon health gate to MPS tasks

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions? No

Does this PR include the addition of new environment variables in the README? No

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.