🔥 Jadwal & Hasil: NVIDIA NeMo/Gym/pull/

iguehring-nvidia · 2026-09-22T11:36:27Z

What

gym eval submit gains an otel: block (on by default) and, for every benchmark job, runs an OpenTelemetry Collector beside the services:

The submit renders a collector configuration into <job dir>/otel/collector.yaml and starts a host-process otelcol-contrib as the first srun --overlap step of the sbatch script (a container is optional: the upstream collector image is distroless and enroot cannot start it). Services start only after the collector's health endpoint answers.
The collector scrapes every model service's Prometheus /metrics on localhost, plus the node's dcgm-exporter (:9400) and node_exporter (:9100) when present (ports configurable or null), and accepts OTLP on :4317/:4318 from anything in the job.
It stamps the resource attributes dashboards filter on (user, run_id = the gym job id, slurm_job_id as a string, benchmark, cluster, model), renames vllm:<metric> to vllm_<metric> after parsing so types survive, and exports to the configured OTLP/HTTP endpoint and to <job dir>/otel/*.jsonl.
After the driver exits it waits one scrape interval, signals the collector process itself (not its srun, which Slurm would treat as a step kill), waits for a clean shutdown, and exits with the driver's return code. Preserving that code matters: with the cleanup appended after the driver (and ending in || true), a job would otherwise exit 0 whenever the collector stopped cleanly, and a non-zero exit from gym eval run would no longer fail the Slurm job. This is the defect the legacy launcher fixed in fix(slurm): capture the evaluation exit code before server teardown Evaluator#1161 (exit status captured after a teardown ending in || true); DRIVER_RC=$? is the first statement after the driver's srun so it cannot recur here.
The ingest token is read from the submitting shell (otel.token_env), exported into the remote shell only, and referenced as ${env:...} from the collector config. It is never written to the job directory. A missing token or an unset endpoint/service_name fails the submit before anything is staged, with a message naming the key to set or how to disable.

Nothing site-specific is in code: endpoint, routing service name, token variable name and binary path come from the submit config (in practice a cluster fragment).

Why

Every gym-native evaluation should be observable without per-run setup: engine throughput, latency and KV-cache on the shared boards, filterable by user and run. Doing it in the orchestrator means one collector per job, no daemon on the nodes, and a per-run identity stamped where the data is produced. The same collector is the on-ramp for Gym's own Lens telemetry (follow-up PR) and for sandbox metrics.

No issue filed; the design is documented in the PR and in the follow-up docs page.

Decision to review

Failure semantics. A collector that does not become healthy within health_check_timeout_seconds (300 s) aborts the job. Loud by design; an otel.required: false mode that logs and continues is a possible follow-up.

Validation

pytest tests/unit_tests/test_orchestration_otel.py tests/unit_tests/test_slurm_script.py tests/unit_tests/test_slurm_executor.py tests/unit_tests/test_orchestration_api.py: 275 passed.
pre-commit run --files ... on the changed files: pass.
--resolve-only output (sbatch + collector YAML) reviewed for the token path: the token appears only as ${env:<token_env>}.
Tested e2e with grafana dashboard and actual benchmark runs.

Compatibility

otel.enableddefaults totrue, so a config with a model service and no otel:block, on a deployment whose fragments do not yet provideendpointandservice_name, fails at submit with a validation error that names the keys (nothing is staged, no allocation is used). This is intentional: the orchestrator has no production users yet, and the collector should be the default from the first one. Opt out with otel.enabled: false`.

Jobs without a model service get no collector in this PR, because the scraped /metrics pages are its only producers. The follow-up that routes Gym's own telemetry through the collector removes that condition, since Gym's servers produce data whether or not a local model service exists.

copy-pr-bot · 2026-09-22T11:36:30Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-09-22T11:38:51Z

🌿 Preview your docs: https://nvidia-preview-iguehring-orchestrator-otel-collector.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

iguehring-nvidia · 2026-09-22T13:39:01Z

Heads-up on the default: this makes observability.enabled: true the default and fails the submit when endpoint/service_name are not configured. The gym-native cluster fragments in EFB do not carry an observability: block yet, so once this merges and the runner pins a Gym commit that includes it, gym-native submits from EFB fail until the fragments provide endpoint, service_name, token_env and binary.

@wprazuch, that is the EFB change I'd like to land together with the runner pin bump; I have the fragment values ready for oci-hsg and will open the MR against your branch or main, whichever the runner lives on by then. Until then, observability.enabled: false in a config is the escape hatch.

oyilmaz-nvidia

Hey @iguehring-nvidia , qq, I have this PR #2935 that basically collects OTel data and stream it to tools like Graphana, etc using NeMo Lens. I wonder if this PR is overlapping with what we started?

oyilmaz-nvidia · 2026-09-23T03:34:52Z

@@ -0,0 +1,207 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.


I'd recommend changing the filename to otel.py or something similar to this. We use observability for non-OTel metrics such as reward, etc. Otherwise, it might confuse the users.

Done in 4ff7c4d: the module is now nemo_gym/orchestration/executors/otel.py (imports and the test module updated). Open question for you: the config block is still observability: (observability.enabled, observability.endpoint, ...). Should that become otel: for the same reason, or is the user-facing key fine as is? Happy to rename it in this PR if you prefer.

Update: I went ahead and renamed the config block too, in 2d173e5, so no back-and-forth is needed. observability: is now otel: (OtelConfig, otel_active(), default token variable OTEL_TOKEN, error messages name otel.endpoint etc., test module test_orchestration_otel.py). Your argument applies more to the user-facing key than to the module, otel matches the NEMO_GYM_OTEL_* variables users already know, and the orchestrator has no production users yet so there is nothing to migrate. The fern page notes how otel:, observability_enabled and telemetry.enabled relate. Shout if you would rather keep the old key.

oyilmaz-nvidia · 2026-09-23T03:38:14Z

+        {
+            "job_name": f"vllm-{name}",
+            "scrape_interval": interval,
+            "static_configs": [{"targets": [f"localhost:{port}"], "labels": {"gym_service": name}}],


Would this static_config work with the multi node vLLM? If the vLLM service head is also on the first node this works; if it isn't, localhost:{port} might miss it entirely. Can you check and make sure please?

Checked. Two parts:

Where the vLLM head lands. script_templates.py builds the Ray prelude with head_node_hostname=${nodes_array[0]}, i.e. the first node of the allocation, which is also the batch host (where sbatch runs the script). The collector step is an srun --overlap launched from that same script, and the driver already pins itself there with --nodes=1 --ntasks=1. So the vLLM head, the driver and the collector share one node and localhost:<port> reaches both the vLLM metrics endpoint and the OTLP receiver.

A real bug your question surfaced. The collector step had no node flags. In a multi-node allocation srun --overlap without --nodes/--ntasks fans the step out to every node, so an N-node job would have started N collectors (N-1 of them scraping a port with nothing behind it, all under the same run id). Fixed in 4ff7c4d: the collector step now gets --nodes=1 --ntasks=1 when the job is multi-node, same as the driver; test_script_pins_the_collector_to_one_node_of_a_multi_node_job covers it.

Caveat: the scrape only covers the head's vLLM process. Worker-node ranks in a TP/PP deployment do not serve /metrics (the API server on the head aggregates), so nothing is lost there; a data-parallel multi-node layout with several API servers would need per-node scraping, which this PR does not attempt.

oyilmaz-nvidia · 2026-09-23T03:39:38Z

+            "SLURM_JOB_ID": f"{RUNTIME_ENV_PREFIX}SLURM_JOB_ID",
+        },
+        mounts=[f"{remote_bench_dir}:{remote_bench_dir}"],
+        workdir=str(remote_bench_dir),


nit: when obs.container is None, both are passed to _render_service_command but container_flags is "" so they never appear in the script. This is correct (the job dir is simply accessible on the node) and the _render_service_command comment documents it. But passing arguments that are ignored is confusing — considering skipping them in the call when obs.container is None would make the intent clearer.

Agreed, fixed in 4ff7c4d: _render_collector_service passes mounts/workdir only when obs.container is set.

…hmark job `gym eval submit` now adds an `otel_collector` service step to each benchmark's sbatch script, on by default. The collector scrapes every model service's Prometheus `/metrics` on localhost, accepts OTLP from the job's own processes on :4317/:4318, stamps `user`, `run_id` (the gym job id), `slurm_job_id`, `benchmark`, `cluster` and `model` as resource attributes, and exports to the configured OTLP/HTTP endpoint while keeping `<job dir>/otel/*.jsonl` as a local copy. The destination (`observability.endpoint`, `observability.service_name`) has no default and is validated at submit time, as is the ingest token, which is read from the submitting shell (`observability.token_env`) and forwarded through sbatch's environment rather than written into the job directory. Both are checked before anything is staged, on dry runs too. The collector starts before the model services so their startup is captured, is health-checked on :13133 before the model health checks run, and after the driver exits gets one more scrape interval and a SIGTERM for a graceful flush, preserving the driver's exit code. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com> (cherry picked from commit cead17c)

…kdir The upstream collector image is distroless and has no /root. With no workdir declared by the image, enroot's switchroot cds into $HOME and the step dies with "failed to change directory: /root" before the collector starts. The mounted job directory always exists, so use it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com> (cherry picked from commit 2caaa18)

…r job, no ignored args Three points from review: - The collector module is nemo_gym/orchestration/executors/otel.py. "Observability" already names Gym's rollout-evidence feature (reward, trajectories); the module is about the OpenTelemetry collector only. - The collector step is pinned to one node with --nodes=1 --ntasks=1 in a multi-node allocation. Without that, srun fans the step out to every node, giving N collectors under one run id. It runs on the batch host, which is the first node of the allocation, where the Ray prelude puts the head of a multi-node vLLM service and where the driver runs, so scraping and receiving on localhost reaches both. Test added. - The mounts and workdir arguments are passed only when the collector runs in a container; on the node they were ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

Follows the module rename: the block configures the OpenTelemetry Collector and nothing else, while "observability" already names Gym's rollout-evidence feature (`observability_enabled`) and `telemetry.enabled` the in-process SDK. Renamed while the orchestrator has no production users, so no migration is needed. - `observability:` -> `otel:` in SubmitConfig; `ObservabilityConfig` -> `OtelConfig` - `observability_active()` -> `otel_active()` - default token variable `OBSERVABILITY_TOKEN` -> `OTEL_TOKEN` - validation messages name `otel.endpoint`, `otel.service_name`, `otel.enabled` - test module `test_orchestration_observability.py` -> `test_orchestration_otel.py` - fern page updated, with a note on how the three settings relate Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

oyilmaz-nvidia

LGTM!

github-actions · 2026-09-23T19:00:37Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35906308951

…NeMo#3716) ## What does this PR do? Fail a submitted benchmark job when fewer samples finish than expected, even if the collector returns normally. Submission enables `require_complete` by default; set `driver.benchmarks.<name>.run.require_complete: false` to opt out. Standalone runs retain their current default behavior. Uses the collector's existing expected count and persisted results. Zero-reward results count as finished; duplicates and failure records do not. Checks run after partial artifacts and metrics are saved. GDPVal uses its stage counts, including the original plans for cached stages. Interrupted runs also exit nonzero when `require_complete` is enabled. No separate issue: this fixes false-success job status. Regression checks reuse existing tests. There are no new files or dependencies. ## Validation At `673e9a46`, 808 affected tests passed, including collector, orchestration, CLI, and GDPVal tests. GDPVal's 42 tests were rerun after the final cached-plan edit. Exact suite command using the shared development venv and this checkout's server namespace: ```bash ../nemo-gym-venv-setup-race/.venv/bin/python -c 'from pathlib import Path; import sys, resources_servers; resources_servers.__path__ = [str(Path.cwd() / "resources_servers")]; import pytest; sys.exit(pytest.main(sys.argv[1:]))' \ tests/unit_tests/test_rollout_collection.py \ tests/unit_tests/test_slurm_script.py \ tests/unit_tests/test_orchestration_otel.py \ tests/unit_tests/test_orchestration_api.py \ tests/unit_tests/test_slurm_executor.py \ tests/unit_tests/test_cli_eval_submit.py \ tests/unit_tests/test_cli_eval.py \ tests/unit_tests/test_cli_main.py \ resources_servers/gdpval/tests/test_multistage_orchestrator.py \ --override-ini addopts='' -q ../nemo-gym-venv-setup-race/.venv/bin/python -m pre_commit run --all-files git diff --cached --check ``` Local generated-shell smoke checks returned 0 for complete standard/GDPVal runs and 1 for incomplete/interrupted runs. Slurm and service startup were substituted locally; the collection entrypoints and completion checks were real. Disabling the check makes the partial-collection regression fail with `DID NOT RAISE`. At the current head (`51e3ffe0`), all five targeted submission/collection cases passed: the default, explicit enablement, explicit opt-out, and partial collection with the check enabled/disabled. The opt-out case failed before the change. Pre-commit checks on all three changed files and `git diff --check` passed. The earlier full suite was not rerun for this change. Full CI (core, sandbox, eight server shards) and live Slurm execution were not run locally. ## Rollouts N/A for model rollouts: this changes completion accounting and exit status, not agent or verifier behavior. ## Compatibility and benchmark impact Failure is per submitted benchmark job. Partial metrics remain available, and scoring is unchanged. Submitted drivers need an updated Gym installation. NEL launches bypass Gym submission and must explicitly pass `++require_complete=true` to opt in. Custom collection drivers must honor `require_complete` by calling `check_completion` with their full expected count; the built-in GDPVal driver does so. Collector cleanup and exit propagation from NVIDIA-NeMo#3610 remain unchanged. ## Checklist - [x] Contributing guidelines read; change focused. - [x] Relevant tests and pre-commit checks pass as detailed above. - [x] Config behavior documented in its field description and method docstring. - [x] No new source files or dependencies. - [x] All commits have DCO sign-off. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>

…ctor The driver installs Gym with the telemetry extra and receives the environment that switches nemo-lens on and points it at the job collector on localhost; every Gym server process then pushes spans, metrics and logs into the pipeline the collector PR (#3610) set up. The collector keeps each producer's own service.name as the display identity, derives latency and call-count series from spans with the span_metrics connector, and runs for every job whether or not it has a model service to scrape. Logs use the collector's gRPC port because nemo-lens builds its log exporter over gRPC regardless of the protocol setting. Config: otel.gym_span_groups (default "default,verify") and otel.gym_logs (default on). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

…ctor (NVIDIA-NeMo#3619) Standalone against `main` since NVIDIA-NeMo#3610 (the collector) merged as f2f446a. Revision of 2026-09-30: rebased onto main (28c6035), the feature squashed into one commit, plus the two review fixes as their own commits (display-name doc, submit-time check for the telemetry extra) and a comment trim. ## What Gym's own servers become producers for the collector that NVIDIA-NeMo#3610 runs beside every job: - The driver is installed with the `telemetry` extra and started with the environment that switches nemo-lens on and points it at the collector on localhost: `NEMO_GYM_OTEL_ENABLED`, `NEMO_GYM_OTEL_RUN_ID` (the gym job id), `NEMO_GYM_OTEL_SPAN_GROUPS`, `NEMO_GYM_OTEL_LOGS_ENABLED`, `OTEL_EXPORTER_OTLP_ENDPOINT`/`_PROTOCOL`. Every server the driver starts inherits it; an explicit `driver.env` value wins. - `otel.gym_telemetry` (default on) switches Gym's own nemo-lens telemetry; it needs `driver.gym_install`, where the `telemetry` extra is installed, and the submit fails with a message naming both ways out when that is missing. Off keeps the collector for engine and node metrics only. - `otel.gym_span_groups` (default `default,verify`) selects Gym's span groups for the job; `otel.gym_logs` (default on) ships Gym's Python logging as OTel logs. Because nemo-lens builds its log exporter over gRPC regardless of the protocol setting, logs get the collector's gRPC port through `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. - The collector keeps each producer's own `service.name` as `service.name.override` before stamping the routing identity (`transform/identity`), accepts OTLP metrics, and derives latency histograms and call counters from spans with the `span_metrics` connector, keyed by span name, producer and sandbox provider, so sandbox start/exec latency and error rates need no metric of their own. - The collector now runs for every enabled job, not only jobs with a model service: a `type: api` policy or a CPU-only judge run produces rollout and verify telemetry all the same. The Prometheus receiver is rendered only when there are scrape jobs. ## Why NVIDIA-NeMo#3610 gives engine and node metrics with a per-run identity. This PR adds the application layer: how long rollouts and verifications take, one trace per rollout across agent, model and resources servers, request latency per route, sandbox operations, and trace-correlated logs, all under the same `run_id`/`user` labels on the same boards. No issue filed; follow-up to NVIDIA-NeMo#3610 as planned there. ## Validation - `pytest tests/unit_tests/test_orchestration_otel.py tests/unit_tests/test_slurm_script.py tests/unit_tests/test_slurm_executor.py tests/unit_tests/test_orchestration_api.py tests/unit_tests/test_fern_docs_links.py`: 363 passed (including the docs-link tests). The executor bookkeeping tests, which build model-less configs, now set `otel.enabled: false`, since the collector no longer skips such jobs. - `pre-commit run --files ...` on the changed files: pass. ## Rollout evidence Slurm cluster, vLLM policy, `gym eval submit`, nemo-lens in the driver venv: - scicode (jobs 7329053, 7329528): `gym.rollout.duration_ms`, `gym.verify.*` (with the `verify` group) and FastAPI request histograms per Gym server in the metrics backend; one trace per job visible in the traces backend with `run_id` on every span. - Terminal-Bench 4 on OpenSandbox (7331831; full runs 7332698, 7335617, 7336762, 7337676): `traces_span_metrics_*` series for `gym.sandbox.start`/`gym.sandbox.exec` with the provider label, including error status on failed creates. - Logs (7335821, 7337676): Gym log records reach the collector trace-correlated once routed to the gRPC port; an earlier run with the HTTP port lost them all and hung the driver at exit in the exporter's retry loop, which is why the per-signal endpoint is set explicitly. ## Compatibility Adds fields with defaults; no change for jobs with a model service beyond the extra telemetry. Jobs without a model service now get a collector too (they got none in NVIDIA-NeMo#3610). `otel.gym_span_groups`/`gym_logs` tune or turn off Gym-side telemetry; `otel.enabled: false` turns the whole thing off. --------- Signed-off-by: Ingo Guehring <iguehring@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…VIDIA-NeMo#3911) ## What The job collector (NVIDIA-NeMo#3610) stamps the ingest token as an `Authorization` resource attribute on every batch, because the routing proxy in front of the managed backend resolves the tenant from it. The same pipelines also fed the file exporters, so the job-directory copies `otel/{metrics,traces,logs}.jsonl` carried the token in plaintext. The collector also logged at `debug`, which prints each batch's resource attributes, token included, into `logs/otel_collector.log`. Both files are created with the user's default umask on shared storage. Fix: two pipelines per signal over the same receivers. `<signal>` stamps the token (`resource/managed`) and ships to the managed backend; `<signal>/local` stamps everything but (`resource/local`) and writes the job-directory copy. The `span_metrics` connector hangs off the local traces pipeline and its series enter both metrics pipelines like any other metric. Collector log level is `info` (export failures still log at error). ## Why it matters Every run since NVIDIA-NeMo#3610 merged wrote a team write-credential into world-readable files in the job directory. Found while checking the EFB fragments MR end to end (jobs 7601272 and 7602043 on oci-hsg). Affected directories on that cluster have been chmod'ed; the token should be rotated once this lands. ## Validation - `pytest tests/unit_tests/test_orchestration_otel.py tests/unit_tests/test_slurm_script.py tests/unit_tests/test_slurm_executor.py tests/unit_tests/test_orchestration_api.py tests/unit_tests/test_fern_docs_links.py`: 386 passed. New tests: `test_the_token_never_reaches_the_job_directory_copies` (no file-writing pipeline carries the attribute; managed and local processors differ by exactly `Authorization`) and `test_collector_log_level_does_not_print_batches`. - Rendered config validated with `otelcol-contrib 0.152.1 validate` (the binary the clusters run). - **Real run on oci-hsg (job 7602618, 8 Terminal-Bench rollouts, collector config rendered by this branch):** zero occurrences of the token anywhere in the job directory; `otel/metrics.jsonl`, `traces.jsonl`, `logs.jsonl` carry 567/559/68 resources with `run_id`, `user` and `service.name.override` on every one and no `Authorization`; the collector log has 39 lines, none at debug, and no export errors. The same check on a pre-fix run finds the token in all four files. ## Compatibility Pipeline names change (`metrics/local` etc.); nothing outside the collector config refers to them. The managed export is unchanged: same attributes, same header. Dashboards unaffected. Signed-off-by: Ingo Guehring <iguehring@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

iguehring-nvidia self-assigned this Sep 22, 2026

iguehring-nvidia marked this pull request as ready for review September 22, 2026 14:18

iguehring-nvidia added area:orchestration Job submission, deployment, services, and lifecycle management area:docs Generic documentation without another dominant domain labels Sep 22, 2026

iguehring-nvidia requested review from agronskiy and wprazuch September 22, 2026 14:31

iguehring-nvidia enabled auto-merge September 22, 2026 14:45

iguehring-nvidia requested a review from lafrancef September 22, 2026 14:47

iguehring-nvidia mentioned this pull request Sep 22, 2026

feat(orchestration): route Gym's Lens telemetry through the job collector #3619

Merged

iguehring-nvidia requested a review from oyilmaz-nvidia September 22, 2026 15:38

oyilmaz-nvidia reviewed Sep 22, 2026

View reviewed changes

oyilmaz-nvidia requested changes Sep 23, 2026

View reviewed changes

iguehring-nvidia force-pushed the iguehring/orchestrator-otel-collector branch from e53d170 to de933b7 Compare September 23, 2026 08:13

nemo-automation-bot Bot requested review from a team September 23, 2026 08:13

copy-pr-bot Bot temporarily deployed to public September 23, 2026 08:13 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 08:15 Inactive

iguehring-nvidia and others added 2 commits September 23, 2026 12:15

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:17 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:18 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:20 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:21 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 12:28 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 12:29 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 12:30 Inactive

agronskiy approved these changes Sep 23, 2026

View reviewed changes

github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 23, 2026

github-actions Bot mentioned this pull request Sep 23, 2026

[Tracker] Pull request handoff SLA #3402

Open

oyilmaz-nvidia approved these changes Sep 23, 2026

View reviewed changes

iguehring-nvidia added this pull request to the merge queue Sep 23, 2026

Merged via the queue into main with commit f2f446a Sep 23, 2026
43 checks passed

iguehring-nvidia deleted the iguehring/orchestrator-otel-collector branch September 23, 2026 19:17

Glorf mentioned this pull request Sep 28, 2026

fix(orchestration): fail evaluations with incomplete samples #3716

Merged

5 tasks done

This was referenced Oct 1, 2026

feat(observability): collect benchmark-job engine and node telemetry #3864

Closed

feat(observability): export Gym traces metrics and logs through job collectors #3865

Closed

[epic] Rollout observability & reliability fast-follows #2831

Open

iguehring-nvidia mentioned this pull request Oct 1, 2026

fix(orchestration): keep the ingest token out of the job directory #3911

Merged

iguehring-nvidia mentioned this pull request Oct 19, 2026

feat(eval): add otel.vllm_traces to join vLLM request spans to rollout traces #4749

Merged

		@@ -0,0 +1,207 @@
		# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Conversation

iguehring-nvidia commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Decision to review

Validation

Compatibility

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

iguehring-nvidia commented Sep 22, 2026

Uh oh!

oyilmaz-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

iguehring-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

iguehring-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

iguehring-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

iguehring-nvidia Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iguehring-nvidia commented Sep 22, 2026 •

edited

Loading