Release Summary

NeMo Gym v0.7.0 introduces infrastructure for defining reusable agent-environment interaction patterns while improving the reliability of large-scale evaluation and training.

Highlights:

• Define reusable interaction patterns between agents and environments with Environment Servers that manage sessions, verification, cleanup, and failures
• Reuse compatible agent harnesses across tasks and environments through common sandbox and verification infrastructure
• Finish large evaluations reliably by respecting job time limits, keeping memory use predictable, and automatically resuming Slurm jobs after timeout or preemption
• Preserve token-level training trajectories with durable worker-side capture for vLLM and Megatron
• Launch multi-service OpenSandbox workloads from supported Compose files and pause or resume sandboxes without losing task state

First-Time Contributors

We welcomed 39 first-time contributors to NeMo Gym with this release:

• @pthombre added durable external-sink token capture with worker-owned staging and terminal attribution
• @lauradang added the Megatron worker capture backend and multimodal Megatron capture
• @iguehring-nvidia added per-job OpenTelemetry collection and routed NeMo Lens telemetry through the job collector
• @afourniernv added ATIF v1.7 export and reverification of supported external ATIF trajectories
• @waple0820, @deepbuilder, @lafrancef, @muskank7, @lilithgrigoryan, and @krishnpci expanded the environment catalog with Interactive Browser, ViBench, Terminal-Bench 4, Aegis v4, AssayBench, and Indian Banking

Thank you to all 91 NeMo Gym contributors this cycle, including 39 first-time contributors!

Build and Configure Environments

• Environment Servers define reusable interaction patterns that coordinate agent and resources sessions, verification, cleanup, and failures; v0.7 includes a built-in single-agent-turn implementation
• Prepare reusable environment datasets without custom conversion scripts and report metrics at task, dataset, and environment levels
• Reproduce benchmark setup through resolved manifests and catch configuration problems before servers start with strict validation
• Reduce startup and memory overhead for servers that do not use distributed execution

Configure Agent Harnesses

• Reuse compatible agent harnesses across tasks and environments while Environment Servers coordinate sandbox access, verification, and cleanup

Configure Models

• Scale multi-turn vLLM evaluation across nodes with Ray Serve and prefill/decode disaggregation while keeping each conversation on the backend holding its cached context

Configure Sandboxes

• Run and debug trusted workloads directly on your machine with the new Local provider; no container runtime required
• Run multi-service workloads on OpenSandbox from supported Compose YAML; pause or resume sandboxes to preserve long-lived task state
• Improve OpenSandbox reliability with configurable TLS verification, shared HTTP connections, and cleaner recovery from terminal and cancellation failures

Evaluation and Training

• Preserve complete token-level training data even when inference runs outside the NeMo Gym process by staging token records on vLLM or Megatron workers before returning responses
• Resume Slurm evaluations automatically after timeout or preemption while preserving the submission record and resolved configuration
• Finish within an allocation's walltime by stopping new work at a dispatch budget; unstarted rows remain available when the run resumes, and retry rules control which failures run again
• Keep driver memory predictable during large runs with bounded task admission and optional non-retaining collection
• Avoid inflated scores in incomplete runs by optionally counting missing rollouts as zero, and distinguish infrastructure failures from unsuccessful task outcomes
• Inspect and validate a fully resolved job configuration before submitting it
• Spread resource-heavy repeats across a run by scheduling each task's first rollout before its second

Inspect and Export Agent Runs

• Trust saved rollout evidence with health checks that catch failed model calls and missing turns across model and agent harness activity
• Trace slow or failing submitted evaluations end to end with a per-job OpenTelemetry collector and model-call capture
• Export NeMo Gym trajectories as ATIF v1.7 and re-verify supported ATIF trajectories from external systems

Environments and Benchmarks

• Software engineering and terminal tasks: repair production repositories with SWE-bench Pro; solve longer terminal tasks with Terminal-Bench 4; evaluate scientific code with SciCodePile; grade workspace changes with JobBench and Workspace-Bench; and build web apps from product requirements with ViBench
• Browser, search, and agentic tasks: interact with websites through local Playwright or remote CDP in Interactive Browser; evaluate deep and broad research with DeepSearchQA and WideSearch; compare chat quality through LMArena proxy evaluations; and train multi-step reasoning with Wordle
• Science, chemistry, and mathematics: rank genes from CRISPR screens with AssayBench, reason about organic reaction mechanisms with FukuyamaBench, edit molecules under toxicity constraints with MolDeTox, and evaluate mathematical proofs with expanded MathArena suites
• Safety and reward modeling: classify multimodal safety with Aegis v4, test sycophancy with False Statement Judge, and train models to act as Generative Reward Model judges
• Finance and professional tasks: evaluate multi-turn retail banking, search a local SEC filings corpus, reason over professional PDF documents with GDP.pdf, and run more robust multimodal and multi-stage GDPVal judging

See the Available Environments table for the full list.

Deprecation and Compatibility Notices

• NeMo Gym now routes each episode through an Environment Server, which coordinates the agent, resources, verification, cleanup, and failure handling. Existing agent-only configurations continue to work through an automatically generated legacy_agent compatibility server but emit a migration warning. Add an explicit Environment Server or run scripts/add_legacy_agent_environment_servers.py
• Ray now requires version 2.58.0 or newer
• The optional vLLM dependency moved from vLLM 0.25.1 and FlashInfer 0.6.13 to vLLM 0.29.0 and FlashInfer 0.6.18; resync local model-server environments

Bug Fixes

• Constrained tokenizers and datasets versions for Python 3.13 and PyArrow compatibility and updated dependencies for security fixes
• Preserved structured Responses fields, upstream execution outcomes, and per-rollout seeds during conversion and replay
• Stopped local and OpenCode sandbox processes when their rollouts are cancelled
• Improved reliability across Terminal-Bench, SWE-bench Pro, Spider2, SciCode, and GDPVal

Documentation

• Added an observability section covering rollout evidence, OpenTelemetry, metrics, and distributed tracing
• Added a guide for comparing agent harnesses
• Added sandbox setup and operations guides for Docker Compose, Daytona, and OpenSandbox
• Expanded the CLI reference for ATIF export, reverification, dispatch budgets, missing-rollout accounting, interleaved repeats, and materialized tasksets

Release Summary

NeMo Gym v0.6.0 lets you use external agent harnesses during RL training and adds Switchyard for evaluating model-routing strategies.

Highlights:

• Use supported external agent harnesses during RL training; NeMo Gym preserves each model call's token IDs and reconstructs the multi-step run as one training response
• Run routing-aware evaluations with Switchyard by comparing fixed and routed conditions on the same benchmark
• Trust and debug rollouts with automatic health checks, end-to-end traces, and per-agent token, turn, tool-call, and latency diagnostics
• Evaluate and compare multiple agents and datasets in one run, with task-level routing to the appropriate harness
• Submit eval jobs to a Slurm cluster directly from NeMo Gym, scaling vLLM across GPUs or nodes for higher rollout concurrency

First-Time Contributors

We welcomed 19 new contributors to NeMo Gym with this release:

• @giuliolovisotto added automatic rollout health checks and documented how to diagnose rollout quality
• @mlazuka added per-rollout and per-run aggregate observability metrics
• @imxj added the LiteLLM model server
• @max-sudolabs added the E2B sandbox provider
• @aroshanghias-nvd added a simple agent with configurable context-compaction policies
• @ehosseiniasl added video input support to the vLLM model server
• @adv-andrew added the synthetic OpenAir 5G congestion-control environment
• @knayaka added the Citation IF instruction-following benchmark
• @nickson-quak added a social-bias evaluation environment that separately scores answer correctness and whether explanations rely on stereotypes
• @gnalbandyan added the multimodal HLE vision benchmark
• @michal2409 improved OpenHands reliability and generation-limit handling for SWE agents

Thank you to all 64 NeMo Gym contributors this cycle, including 19 first-time contributors!

Configure Tasks and Data

• Resources servers can declare their datasets, and gym dataset collate records each row's task_source so NeMo Gym can select the correct agent at run time
• Use agent_map to reroute selected tasks or fan-out to run the same tasks with multiple agents; invalid agent mappings fail before execution
• Each resources server defines a typed task_data schema. Collation validates rows before a run, and gym env schema --resources-server <name> shows the expected data shape

Configure Agent Harnesses

• Use --agent-type to swap a benchmark or environment's configured harness, with compatibility checks for unsupported pairings
• OpenCode Sandboxed Agent runs OpenCode inside task-specific NeMo Gym sandboxes, with configurations for SWE-Bench Verified and Multilingual, DeepSWE, and Terminal-Bench 2.1
• Cline Agent runs the Cline CLI headlessly and converts its tool-use trace into a NeMo Gym trajectory; this initial integration is evaluation-only
• Conversational Tool Use workflow adds three agents that generate customer-service domains, policies, tools, and scenarios, plus a policy-loop agent and resources server that simulate and score multi-turn customer and tool interactions
• OSWorld Agent runs OSWorld desktop tasks end to end, executes model-generated GUI actions, and returns NeMo Gym trajectories with OSWorld rewards
• Prime Agent runs Prime Intellect's agent headlessly through NeMo Gym, with included math and Reasoning Gym environments
• Simple Strands Agent wraps the Simple Strands Resolver agent with configurable local tools; included environments cover math and Reasoning Gym
• Terminus-2 Agent runs Harbor's terminal-command agent through NeMo Gym, with AnyTerminal integration and support for other terminal-compatible resources servers
• Simple Agent with Compaction applies configurable history policies before each model call, including policies that omit older image observations or reasoning blocks
• Image Tools Agent lets models iteratively transform and inspect images, scores their image-tool calls, and delegates final-answer verification to the mapped environment
• Verified Code QA (VCQA) Agent gives models read-only repository snapshots or Git histories to investigate with file, search, and shell tools, then scores answers against a rubric with an LLM judge
• OpenClaw with AnyTerminal runs the evaluation-only OpenClaw harness inside Terminal Bench task containers and scores each run with the task's tests

Configure Models

• LiteLLM Model connects NeMo Gym to providers supported by LiteLLM
• Switchyard Model compares model-routing strategies on the same benchmark and records the route and configuration used for reproducible results
• The vLLM model server now supports video inputs
• Use sampling_overrides to enforce generation settings when external agent harnesses omit them, preserving on-policy training behavior
• Use endpoint_file to update a vLLM backend address without restarting NeMo Gym, with bounded connection retries during endpoint rotation

Rollout Observability and Export

• Export configs, metrics, and rollouts through a shared exporter framework. MLflow is new; W&B uses the same path
• Automatic rollout health checks flag missing turns, inconsistent model calls, token-count mismatches, and runaway generation, with structured reports from evaluation and aggregation
• NeMo Lens telemetry traces each rollout across NeMo Gym's agent, model, and resources servers, making latency and failures easier to diagnose
• Rollout diagnostics report token usage, turns, tool calls, and latency, with agent-level traces for OpenClaw, Hermes, Pi, OpenCode, and SWE agents
• Repeated-rollout metrics now include variability and 95% confidence intervals

Evaluation and Training

• Compatible external agent harnesses can produce on-policy training data: NeMo Gym preserves exact token IDs across model calls and rebuilds each multi-step episode as one training response
• Opt in to route_failures_to_sidecar to keep an evaluation running when agent calls fail; coverage reporting identifies excluded rollouts, and selected failure classes can instead count as zero
• Trust eval artifacts: automatic health checks catch missing turns, broken model calls, and runaway generation; rollouts record tokens, turns, tool calls, and latency
• Verifiers can return a human-readable failure_reason to explain unsuccessful rollouts

Sandboxing and Orchestration

• E2B joins the built-in sandbox providers with template building, command execution, file operations, and lifecycle management
• The OpenSandbox provider adds asynchronous interactive PTY sessions and detached PTY execution for long-running commands
• OpenSandbox adds configurable network policy, run-scoped cleanup, bounded background-status polling, and an OpenSandbox backend for OSWorld
• The Apptainer provider accepts a custom binary path
• Experimental gym eval submit runs one or more vLLM instances on a Slurm node or distributes whole replicas across nodes, with configurable environment variables, container mounts, and outputs

Environments and Benchmarks

• Software engineering and agentic workflows: Sandboxed OpenCode configurations for SWE-Bench Verified and Multilingual repository repair, DeepSWE software-engineering tasks, and Terminal-Bench 2.1 terminal tasks; Verified Code QA for repository investigation; OSWorld for desktop automation; and synthetic conversational tool use
• Finance: Vals Finance Agent v2 for multi-step financial analysis and local full-text search over SEC filings
• Long context, knowledge, and reasoning: LongMemEval for conversational memory, RULER v2 for long-context reasoning, SpartQA for spatial reasoning, and new math and Reasoning Gym pairings across agent harnesses
• Safety and instruction following: AgentIF for agentic instruction-following scenarios, Citation IF for citation compliance, and synthetic social-bias questions that separately score answer correctness and whether explanations rely on stereotypes
• Science and multimodal: Expanded Litmus ADME drug-discovery profiles, HLE vision for multimodal expert-level questions, image-tool PivotRL verification, and a synthetic OpenAir 5G congestion-control environment
• Aviary environments: Standalone entries for scientific data analysis with BixBench-Hypothesis and BixBench, grade-school math with GSM8K, and multi-hop question answering with HotpotQA

See the Available Environments table for the full list.

Deprecation Notices

• upload_rollouts_to_wandb has been renamed to upload_rollouts because it now controls rollout uploads to every configured exporter. The old name still works but emits a deprecation warning
• NeMo Gym now requires openai==2.44.0 instead of openai<=2.7.2. A parent environment that forces another version fails configuration by default. Set allow_openai_version_skew: true only when intentionally allowing the parent and server environments to use different SDK versions
• +agent_name=<name> now sends every row to that agent. Previously, it only applied to rows that did not specify an agent. Use +agent_map=... to reroute only selected rows
• Datasets collated before v0.6.0 may contain agent_ref without task_source. They still run, but this routing format is deprecated; run gym dataset collate again or set agent_map explicitly
• Because collation now validates task data, malformed or mismatched rows that were previously accepted may fail during preparation. Use gym env schema --resources-server <name> to see the expected shape
• Agentic math environments now use harness-first names. Previous environment names, config paths, preparation scripts, and agent references remain supported as deprecated aliases and emit migration warnings:
  • `reasoning_gym_c...

Release Summary

The NeMo Gym 0.5.0 release expands the sandbox ecosystem to seven providers, adds four new general-purpose agent harnesses (Codex, KiloCode, RemoteAgent, and Any-SWE) bringing the total to 20, adds 21 new benchmarks and environments, and wires rollout observability end-to-end from the model server boundary through agent transcripts.

Highlights:

• Seven sandbox providers: Docker, Daytona, ECS Fargate, Enroot, and OpenShell join OpenSandbox and Apptainer; large-scale OpenSandbox reliability significantly improved
• Four new agent harnesses: Codex CLI, KiloCode, RemoteAgent, and anyswe_agent
• Recompute rewards from stored rollouts without re-running inference with gym eval reverify
• Rollout observability joined end-to-end: model-call capture, agent observations, and a standardized ng_trajectory schema
• 21 new environments across six domains: Agentic, Knowledge and instruction following, Long context, Science and coding, Translation and multilingual, and Reasoning

First-Time Contributors

We welcomed 22 new contributors to NeMo Gym with this release:

• @mpatel31415 added gym eval reverify command and --judge-failed-only flag for recovering failed judge rows
• @Glorf added per-rollout model-call capture, Docker and ECS Fargate sandbox providers, rollout observation contract, Claude Code rollout observations, and standardized ng_trajectory schema
• @nblintao added per-request policy endpoint override for the SWE agents, enabling RL training frameworks to route each episode through a per-episode recording proxy
• @JeffPengCoder brought OSWorld — a stateful desktop GUI benchmark — into the benchmark catalog
• @rystewart-nvidia added the Legal Agent Bench integration, exposing Harvey's 1,749-task LAB benchmark through standard Gym eval commands
• @fallintoplace fixed the long-standing disagreement between runtime aggregate metrics and the persisted output.jsonl
• @jonathanlli added RULER pretrain evaluation, enabling text-completion scoring for base and midtraining checkpoints
• @thompsonb overhauled WMT24++ and FLORES translation evaluation (55 locales, 219 language pairs, chrF/spBLEU scoring); expanded MMLU-ProX to 29 languages
• @hkumar92 added the PinchBench agentic benchmark
• @pachmu added ToolSandbox, IHEval, RoleMRC, and RAGTruth benchmarks

Thank you to all 57 NeMo Gym contributors this cycle, including 22 first-time contributors!

Command Line Interface

• gym eval reverify — re-run only the verifier on stored rollouts; --judge-failed-only recovers rows that failed due to a flaky judge without re-verifying successful rollouts
• gym list and gym search extended to cover models, resources-servers, and agents; gym list <type> <name> drills into a single artifact
• External plugins discoverable via --search-dir and environment variables; -v/--verbose now accepted before any subcommand

Sandboxing

Five new built-in providers: Docker, Daytona, ECS Fargate, Enroot, and OpenShell join OpenSandbox and Apptainer. Provider choice is a one-line config swap — any agent built on nemo_gym.sandbox works with any provider unchanged.

OpenSandbox reliability at scale is significantly improved: keepalive-bounded transport eliminates silent rollout zeroing at concurrency 300–1500; image registry auth supports private container images; sandbox resources are automatically labeled with team, user, and workload identifiers.

See Available Sandbox Providers for the full list.

Configure Agent Harnesses

New harnesses join the existing set (Claude Code, Hermes, mini-SWE-Agent, OpenClaw, Pi, and more):

• Codex and KiloCode integrate codex exec and kilo run respectively, routing model calls through Gym for per-rollout capture
• anyswe_agent runs any Gym harness inside a SWE task container
• swe_agents adds OpenCode as a supported agent framework alongside OpenHands, with DeepSWE and DeNovoSWE dataset support and message replay for trajectory branching
• RemoteAgent drives any external service that implements POST /v1/responses, with Gym owning the tool loop and verification

Configure Models

• vLLM can now drive /v1/completions for base and pretrain checkpoint evaluation via opt-in use_completions_api
• All Gym model servers now accept stream: true on /v1/chat/completions via synthesized SSE, unblocking streaming-first clients such as OpenClaw and Codex
• Add expose_tools_over_mcp: true to any resources server config to serve its tools over MCP with no handler code changes

Rollout Observability

• Per-rollout model-call capture records requests, responses, token usage, and latency at the model server boundary
• Claude Code transcripts populate ng_agent_observations; agent observations and model-call capture are joined through a standardized ng_trajectory schema
• Judge failures are routed to a _failures.jsonl sidecar, keeping aggregate metrics over successfully-judged rows only

New Benchmarks and Environments

21 new environments across six domains:

• Agentic: PinchBench (147 real-world tasks), OSWorld (desktop GUI with VM-backed evaluation), Legal Agent Bench (1,749 Harvey LAB tasks), ToolSandbox (Apple multi-turn tool-use), BrowseComp (web research), BioMNIBench DA, Tau3 banking (BM25+grep offline eval path)
• Knowledge and instruction following: SECQUE, FinanceBench, Finance SEC Search, IHEval (instruction hierarchy, rule-based), Litmus-Bench v0.1, RoleMRC (role-play MRC), RAGTruth (hallucination detection)
• Long context: NIAH (retrieval with overlap penalty)
• Science and coding: CVDP Agentic (expanded to support the agentic subset, harness-agnostic)
• Translation and multilingual: WMT24++ (expanded from 5 to 55 locales), FLORES (expanded from 30 to 219 language pairs, chrF/spBLEU scoring), MMLU-ProX (expanded to 29 languages); RULER now supports pretrain text-completion evaluation
• Reasoning: ReasoningGym environments — six agentic variants: Claude Code, Hermes, and four LangGraph-based variants (orchestrator, reflection, parallel thinking, and ReWOO)

See the Available Environments table for the full list.

Deprecation Notices

• WMT24++ and FLORES scores from prior versions are not comparable with this version's chrF/spBLEU output
• Python 3.13.14 is now required (previously 3.12); users running Gym in Python 3.12 environments must upgrade

Bug Fixes

• SciCode realigned to the AA 65-problem test set with per-rollout subtask accuracy reporting
• Fixed silent rollout zeros at high concurrency against OpenSandbox (keepalive-bounded transport)
• Fixed frozen rollouts in long-running benchmarks (TCP keepalive on global aiohttp connector)
• Fixed gym eval run failing with FileNotFoundError when the output directory did not exist
• Fixed tool_choice sent to vLLM without tools, causing request rejection
• Fixed Claude Code max_turns hardcoded to 30; max_turns: null now removes the cap
• Fixed Apptainer sandbox env vars injected into subprocess argv instead of environment
• Fixed MCQA answer parsing for wrapped formats ($D$, (D), \boxed{\text{Answer: G}})
• Fixed aggregate metrics including non-persisted rollouts, causing disagreement with output.jsonl

Documentation

• Rewrote the key terminology glossary with a Gym overview, component map, and links to how-to pages
• New page documenting the Anthropic Messages dialect (POST /v1/messages) and wiring Claude Code through a Gym model server
• Updated NeMo RL v0.7.0 compatibility guidance
• Migrated all remaining docs examples from legacy ng_run/ng_collect_rollouts to the unified gym CLI
• Documented gym list and gym search extensions, external plugin discovery, and MCP auto-exposure
• Added gym eval reverify and multi-reward verification contract documentation

Release Assets

GitHub Release v0.5.0

Release Summary

NeMo Gym v0.4.0 expands evaluation tooling and agent integrations. It establishes a new monthly release cadence; we will continue to provide day-zero support for Nemotron models, datasets, and environments.

Highlights:

• Unified gym CLI: find agents and benchmarks by name with gym list, and catch config mistakes early with gym env validate
• Diagnose evaluations with BLADE, an analysis skill for agents that reads your evaluation results and produces an evidence-backed report of which tasks failed, why, and the highest-impact fix (e.g. to the agent harness, training, verifier, or prompt)
• Measure the impact of agent skills: run the same tasks with different skill sets and compare how each changes agent performance
• Run agents in isolated sandboxes through a new pluggable provider framework
• More agent harnesses out of the box, including OpenClaw, Pi, and OpenCode
• Connect to hosted inference providers: Fireworks, Together.ai, OpenRouter, and more
• New benchmarks across science, long-context, and interactive tasks

First-Time Contributors

We welcomed 20+ new contributors to this release! A few highlights:

• @marta-sd and @wprazuch led the CLI refactor and clearer config errors
• @hemildesai added the pluggable sandbox provider infrastructure and OpenSandbox as the first built-in
• @adil-a laid the groundwork for Gym-owned MCP resources servers, letting a server expose its tools over MCP
• @eric-tramel added the BunsenChem chemistry benchmark
• @jeffwillette added the long machine translation datasets and servers

Thank you to all the new contributors for helping make NeMo Gym better!

Command Line Interface

• One gym command for the full workflow, with gym env, gym eval, gym list, and gym dataset subcommands
• Reference agents, benchmarks, and environments by name: use gym list to see what is available
• gym env validate checks your config for missing, malformed, or empty values before a run and reports actionable errors

Evaluation & Diagnostics

• Skill evaluation: measure how agent skills affect performance by running the same tasks with different skill sets. Skills apply at rollout time as a run-level knob, so one dataset works across all skill variants and every rollout is tagged for comparison
• BLADE (Benchmark Level Analysis and Diagnostics Engine): a built-in analysis skill that reads an agent run's rollouts, metrics, and configs and produces an evidence-backed report of which tasks failed, why, and the highest-impact fix (e.g. harness, training, verifier, or prompt)

Sandboxing

• Run tool-using and coding agents in isolated sandboxes through a pluggable provider framework
• Built-in OpenSandbox and Apptainer providers, with third-party providers discoverable via entry points

Configure Agent Harnesses

New harnesses join the existing built-in set (Claude Code, Hermes, OpenHands, and more):

• Added OpenCode, OpenClaw, and Pi agents for evaluation
• Claude Code runtime capabilities (tool access, MCP servers, and bare vs. native auto-discovery mode) are now easily set via the server config

Configure Models

• New inference_provider model server connects to any OpenAI-compatible hosted provider (Fireworks, Together.ai, OpenRouter, DeepInfra, Gemini, and more) with ready-made configs
• Every Gym model server now speaks the Anthropic Messages API, so Anthropic-native harnesses like the Claude Code CLI can run against any model you serve with Gym

New Benchmarks

• Science: CritPt (research-level physics), SciCode (scientific coding), BunsenChem (chemistry multiple-choice), and FrontierScience Research (rubric-scored science)
• Long context: Graphwalks (long-context graph reasoning) and Long Machine Translation (PG19, WMT24++)
• Interactive: TALES, a text-adventure game suite

See the Available Environments table for the full list.

Deprecation Notices

• The legacy ng_* and nemo_gym_* CLI commands (such as ng_run and ng_collect_rollouts) are deprecated in favor of the unified gym CLI. They still work for now but will be removed in a future release.

Bug Fixes

• Fixed intermittent connection errors during high-concurrency rollout collection
• Clear error messages instead of crashes when a config file contains invalid YAML

Documentation

• New Build Verifiers section with verification patterns and multi-reward verification
• New Evaluate section covering benchmarks, evaluation metrics, and a guide to agent-native results diagnostics
• New page for configuring and evaluating agent skills

Release Summary

NeMo Gym v0.3.0 ships alongside the NVIDIA Nemotron 3 Ultra model release, open sourcing the environments and corresponding datasets used during training.

Highlights:

• 70+ new environments, including benchmarks such as Tau2 and Nemotron RL training environments
• Popular harness available out-of-the-box such as Claude Code and Hermes
• Integrations with OpenEnv and Harbor - use environments from these libraries directly with NeMo Gym
• Integration with VeRL - train with VeRL and scale rollout collection with NeMo Gym

First-Time Contributors

We welcomed 30+ new contributors to this release! Here are a few highlights:

• @grace-lam added the integration to run Harbor environments with NeMo Gym
• @aleksficek — added Competitive Coding Challenges environment
• @jthomson04 improved rollout resilience when models emit malformed tool-call arguments or missing message content

Thank you to all the new contributors for helping make NeMo Gym better!

New Environments & Benchmarks

Added 70+ new environments including novel datasets and integrations of popular benchmarks. New coverage spans:

• Coding — competitive programming, code infilling, SQL generation, and software-engineering benchmarks with execution-based verification
• Math & proofs — olympiad-style problems, proof grading and validation, and formal verification (including Lean)
• Knowledge & science — graduate-level QA, chemistry and physics tasks, and lab-style reasoning (including multimodal figure, table, and protocol tasks)
• Agentic — multi-turn tool use, search, sandboxed execution, finance workflows, and tau-bench-style conversational agents
• Instruction following — format constraints, citation compliance, and IFBench-style rule verification
• Safety & RLHF — jailbreak detection, abstention calibration, prompt-injection resistance, and generative reward modeling
• Multimodal, speech & translation — VLM benchmarks, visual grounding, ASR evaluation, and machine-translation quality metrics
• Chat & broad knowledge — arena-style preference evaluation and MMLU-family benchmarks
• Interactive RL — Gymnasium-style multi-step environments for spatial and game-based training

See the Available Environments table for the full list.

Configure Agent Harnesses

• Claude Code — available out of the box in NeMo Gym
• Hermes — available out of the box in NeMo Gym
• LangGraph agent — an adapter that lets you build custom agents using LangGraph patterns (reflection, subagent orchestration, parallel thinking, rewoo)
• Gymnasium agent — generic multi-turn harness for use with OpenAI Gym-style environments

Configure Models

• Optional max_concurrent_requests on the OpenAI model server to cap in-flight API calls — useful for rate-limited external endpoints when rollout concurrency is high

Rollout Collection & Profiling

• New ng_aggregate_rollouts command to merge rollout shards collected independently across multiple nodes, enabling distributed eval without requiring a single coordinated collection job

Environment Library Integrations

• OpenEnv — combine OpenEnv environments with NeMo Gym environments
• Harbor — combine Harbor environments with NeMo Gym environments

Deprecation Notices

• Documentation has moved from Sphinx to Fern. Old Sphinx URLs redirect to the new site at docs.nvidia.com/nemo/gym. The docs/ directory is no longer used for publishing.

Bug Fixes

• Fixed aiohttp connection limit exhaustion under FastAPI/Uvicorn with multiple workers
• Fixed session cookie propagation for Starlette >= 1.0.0
• Fixed duplicated usage counting and errors on empty usage in subsequent model calls
• Improved rollout resilience when models emit malformed tool-call arguments or missing message content
• Fixed prompt-key hashing when inputs contain Pydantic BaseModel objects

Documentation

• New concepts pages for environments, evaluation, and training
• Improved Architecture page to clarify how environments map to NeMo Gym components
• Consolidated detailed setup and quickstart into a single improved quickstart with clearer descriptions
• Expanded Ecosystem page with environment library, training framework, and agent harness integrations

• pypi fixes for 0.2.1 patch release by @cmunley1 @kajalj22 :: PR: #1081

Release Summary

NeMo Gym v0.2.0 ships alongside the NVIDIA Nemotron 3 Super model release, open sourcing the RL environments and corresponding datasets used during training. Highlights:

• 17 new training environments across coding, math, science, reasoning, agentic tasks, and safety.
• Integrations with Future House Aviary, Open-Thought Reasoning Gym, and Prime Intellect Verifiers let you use environments from these libraries directly within NeMo Gym
• End-to-end rollout collection with a locally managed vLLM server
• Install directly from PyPI with pip install nemo-gym

First-Time Contributors

We welcomed 15 new contributors to this release! Here are a few highlights:

• @sidnarayanan added the Aviary integration to enable training on any Aviary environment, a library of interactive RL environments spanning math, science, biology, and more
• @3mei added the text-to-SQL environment to generate SQL queries from natural language across multiple SQL dialects
• @Kelvin0110 added the NewtonBench environment to discover scientific laws through interactive experimentation

Thank you to all the new contributors for helping make NeMo Gym better!

Major Features & Improvements

New Environments

• Added 17 new resources servers spanning:
  • Coding: Text to SQL (#648), SWE RL Gen (#561), SWE RL LLM Judge (#561)
  • Math: Lean4 Mathematical Proofs (#563)
  • Science: Aviary (#55), NewtonBench (#650)
  • Reasoning: MultiChallenge (#654), ARC-AGI (#105), Reasoning Gym (#113)
  • Agent tasks: xLAM Function Calling (#262), Tavily Search (#825), Single Step Tool Use with Argument Comparison (#825), Terminus Judge (#594), NeMo Skills Tools (#571)
  • Safety: Jailbreak Detection (#825), Over Refusal Detection (#825)
  • RLHF: Generative Reward Model Compare (#674)
• Added 5 new agent servers: Aviary agent (#55), proof refinement agent (#563), SWE agents (#343), tool simulation agent (#826), and verifiers agent (#573)

Environment Library Integrations
Combine environments from other libraries with NeMo Gym environments

• Future House Aviary (#55, #590)
• Open-Thought Reasoning Gym (#113)
• Prime Intellect Verifiers (#573)

Model Serving

• Local vLLM model server with end-to-end rollout collection without an external API (#558, #762)
• vLLM 0.16+ support for the reasoning field in responses (#816)
• VLLMModel chat template kwargs support (#538, #636)
• Per-task chat template and extra body args, enabling per-task control of reasoning mode and thinking budget (#672)

Rollout Collection & Profiling

• New ng_reward_profile command to compute per-task pass rates and aggregate metrics (#83, #621)
• CPU profiling for rollout performance analysis (#763)
• Add option for seeding on num_repeats for rollouts (#740)

Infrastructure & Developer Experience

• PyPI compatibility: install via pip install nemo-gym (#649)
• Dry run mode: ng_run +dryrun=true to validate configs and install environments without starting servers (#743)
• ng_status command to list running servers and their health (#290)
• Server stdout/stderr redirection with server name prefixes (#703)
• FastAPI worker support for higher throughput across multiple workers (#566)

Model Recipes

• Nemotron 3 Nano training recipe (#699)
• Nemotron 3 Super training recipe (#863)

Deprecation Notices

• Deprecated ng_viewer due to a Gradio security vulnerability. We plan to revisit rollout viewing with a more robust solution in a future release.

Bug Fixes

• Fixed 0.1.1 environments to work correctly with RL training pipelines (#768)
• Fixed crash when server receives malformed JSON during rollout collection (#770)
• Fixed dry run mode failing (#746)
• Fixed nested responses_create_params overrides not merging correctly from CLI (#827)
• Fixed ng_prepare_data failing when multiple environments define overlapping metrics (#738)
• Fixed reward profiling failing when model response doesn't include usage stats (#824)
• Fixed NeMo-Skills python tool to use HTTP calls instead of subprocess execution (#606)
• Bumped Pillow and other packages to address security vulnerabilities (#667, #739)
• ng_dump_config now redacts API key values from output (#567)

Documentation

• New training tutorials: Unsloth training with NeMo Gym, multi-environment training
• New environment tutorials: creating a training environment, custom data preparation, integrating external environment libraries, environment best practices
• Model recipes: reproduce the training for Nemotron 3 Nano and Nemotron 3 Super
• Concepts & architecture overhaul: rewrote concepts docs, added architecture diagrams, added agent server and resources server docs
• Training approaches: added training approaches docs page covering SFT, RL (GRPO), and RLVR
• Ecosystem page: revamped ecosystem page with training framework integrations and environment library integrations
• Infrastructure: added SWE RL infrastructure case study, deployment topology docs
• Quality pass: redirect sweep, style guide sweep, consistent naming, FAQ additions, broken link fixes

Looking Ahead

• VLM support: add support for VLM models and environments with images, e.g. browser environments and computer use agent (CUA) environments
• Benchmark environments: add popular OSS environments such as OSWorld, Tau Bench, BrowseComp
• Integrate existing agents: integrate popular existing agents, e.g. coding harnesses, as well as agents developed via popular agent frameworks, e.g. LangGraph
• Environment tutorials: incorporate more complex agentic loops during training such as multi-turn conversation and user modeling

Release Assets

GitHub Release: https://github.com/NVIDIA-NeMo/Gym/releases/tag/v0.2.0
Container: nvcr.io/nvidia/nemo-rl:v0.5.0.nemotron_3_super

What's Changed

• Bump to v0.2.0 by @bxyu-nvidia in #510
• reasoning-gym resource server by @cmunley1 in #113
• docs: redirect setup by @lbliii in #513
• docs: Miscellaneous GRPO tutorial fixes by @bxyu-nvidia in #512
• docs settings update by @lbliii in #525
• Debug server package versions by @fsiino-nvidia in #406
• List running server health and status by @fsiino-nvidia in #290
• VLLMModel supports chat template kwargs by @pjin-nvidia in #538
• Salesforce xlam-function-calling-60k resources server by @cmunley1 in #262
• python flag for colab venv installation by @cmunley1 in #526
• add unsloth and trl to docs by @cmunley1 in #536
• docs: remove trl docs by @cmunley1 in #543
• Remove PlainTextResponse response_class by @fsiino-nvidia in https://github.com/NVIDIA-N...

What's Changed

• Bump package info for v0.2.0 by @bxyu-nvidia in #337
• fix: Update incorrect path in docs: library_judge_math -> math_with_j… by @shashank3959 in #355
• Update secret detector to work with forks by @chtruong814 in #358
• Removed reference to gitlab master by @hwolff99 in #377
• Mark experimental tutorials by @bxyu-nvidia in #386
• docs: experimental label by @lbliii in #391
• Fixed typos by @hwolff99 in #400
• Readme dataset discoverability cont by @fsiino-nvidia in #344
• Add absolute ip for multi node by @sdevare-nv in #286
• docs: removed "How to Navigate" section from concepts by @ahmadki in #414
• docs: Fixed image embedding in core abstractions page by @ahmadki in #410
• docs: Fixed Licensing information in structured outputs by @ahmadki in #412
• docs: Added hyperlinks to github repo in docs by @ahmadki in #413
• docs: Add software / hardware requirements to README and docs. by @ffrujeri in #401
• docs: Cleaned the "Quick Start" section in the README by @ahmadki in #411
• Display system and version info by @fsiino-nvidia in #347
• docs: Improve language around resources servers. by @ffrujeri in #408
• docs: Add Create Resource Server Tutorial by @ffrujeri in #407
• miniswe w/ offline uv by @sdevare-nv in #357
• update vllm model comments by @cmunley1 in #423
• docs: linked several terms to their defenition in glossary by @ahmadki in #424
• docs: Explain why GPT-4 is used and clarify support for other models by @ahmadki in #425
• Removed internal section by @hwolff99 in #430
• docs: various improvements and fixes by @ahmadki in #415
• docs: Relate sections Get Started and Rollout Collection by @fsiino-nvidia in #426
• Guide user on next steps after finishing get started by @cwing-nvidia in #435
• Add placeholder author by @jkyi-nvidia in #440
• Clarify training environment framing and align docs messaging by @cwing-nvidia in #438
• docs: Added CLI documentation by @ahmadki in #444
• Change NeMo Gym from framework to library by @cwing-nvidia in #456
• Add Data Designer and links to ecosystem page by @cwing-nvidia in #462
• docs: Moved configuration system under about by @ahmadki in #420
• Add benefits to About page aligned with README by @cwing-nvidia in #452
• Explain where the name Gym comes from; Gym Key Terminology doc is missing some of the old material by @bxyu-nvidia in #470
• add calendar env for multi-turn IF by @sanjaykariyappa in #297
• docs(readme): fix Example Resource Servers table - correct Multi Step… by @lbliii in #464
• Remove penguin references by @ahmadki in #469
• docs: Training framework integration by @bxyu-nvidia in #439
• Bug: inconsistent documentation around servers running by @bxyu-nvidia in #472
• docs: Improve server reference info by @bxyu-nvidia in #474
• pyproject typos and grammar fixes by @ahmadki in #473
• Miscellaneous infra improvements/fixes by @pjin-nvidia in #317
• Expose server host and port in dataset viewer CLI by @ahmadki in #476
• Rename examples simple_weather and stateful_counter by @fsiino-nvidia in #479
• More single tool call filename updates by @fsiino-nvidia in #480
• docs: Fix wrong count vs actual by @fsiino-nvidia in #482
• Fix duplicate reference sections by @bxyu-nvidia in #483
• docs: home pg, quickstart move, gh icon by @lbliii in #463
• More single tool call filename updates cont by @fsiino-nvidia in #484
• Fix NeMo Gym Pyproject links by @bxyu-nvidia in #486
• docs: move FAQ by @lbliii in #489
• docs: contribute section by @lbliii in #490
• Misc rollout fixes by @pjin-nvidia in #447
• improve framing of training framework integration guide for contributing by @cwing-nvidia in #493
• Docs: Contribution Home & Dev Setup by @cwing-nvidia in #494
• Add environment contribution docs by @cwing-nvidia in #498
• FAQ cleanup by @cwing-nvidia in #499
• Simplify contributing.md by @cwing-nvidia in #500
• Reorder README structure by @cwing-nvidia in #501
• docs: End-to-end GRPO Training with NeMo RL tutorial [master branch] by @bxyu-nvidia in #481
• Update dataset configs with HuggingFace links by @bxyu-nvidia in #508
• Change to v0.1.1 release version by @bxyu-nvidia in #509

New Contributors

• @shashank3959 made their first contribution in #355
• @hwolff99 made their first contribution in #377
• @ahmadki made their first contribution in #414
• @ffrujeri made their first contribution in #401
• @sanjaykariyappa made their first contribution in #297

Full Changelog: v0.1.0...v0.1.1

What's Changed

• Add copy-pr-bot by @chtruong814 in #1
• Add initial repo template by @chtruong814 in #2
• Update GitHub with Gitlab main by @bxyu-nvidia in #3
• Alias as Penguin by @bxyu-nvidia in #4
• Add Copyright docs README FAQ by @bxyu-nvidia in #7
• Dapo17k by @bxyu-nvidia in #6
• Fix docs build failures by @bxyu-nvidia in #8
• Fix docs by @bxyu-nvidia in #10
• Improve Github SSH Key setup docs by @bxyu-nvidia in #12
• Comp-Coding Verifier by @kbhardwaj-nvidia in #5
• Dataset viewer simple aggregations by @fsiino-nvidia in #9
• VLLMModel docs in main Readme by @bxyu-nvidia in #13
• Fix agent name in docs by @bxyu-nvidia in #15
• VLLMModel propogates token IDs by @bxyu-nvidia in #11
• VLLMModel tokenize params cleanup by @bxyu-nvidia in #21
• Update Comp-Coding README.md by @kbhardwaj-nvidia in #26
• Docs improvements - remove Why NeMo Gym section and add CI/CD tests info by @bxyu-nvidia in #27
• update server logging format to be more consistent by @cmunley1 in #22
• update readmes from ng_collect_traj to ng_collect_rollouts by @cmunley1 in #25
• Simple agent stop criteria requires no tool calls AND output message item to be present by @bxyu-nvidia in #19
• Server spinup polling by @bxyu-nvidia in #31
• Rename top-level config key 'openai_model' => 'policy_model' by @pjin-nvidia in #33
• Simple agent allows non-json tool responses by @bxyu-nvidia in #35
• Multi-verifier docs by @bxyu-nvidia in #36
• Servers have easy hooks into individual instances via session by @bxyu-nvidia in #24
• Add Math Stack Overflow dataset by @damon-mosk-aoyama-nvidia in #42
• Add Workbench validation dataset by @bxyu-nvidia in #46
• Docs update by @bxyu-nvidia in #47
• Implements LLM-as-Judge for Response Equivalence by @soares-f in #16
• Configure global httpx client by @pjin-nvidia in #50
• Fix OpenAI ResponseReasoningItem.status property by @bxyu-nvidia in #54
• VLLMModel data parallel; explicit RunHelper shutdown handle by @bxyu-nvidia in #52
• removed simple_agent_stateful, uses fastapi to keep track of session by @RahulSChand in #44
• Migrate text_based_game: sudoku and game agent features by @RahulSChand in #30
• Revert "Migrate text_based_game: sudoku and game agent features" by @bxyu-nvidia in #65
• Add data aggregations to data preparation by @fsiino-nvidia in #49
• Instantiate one httpx async client per unique connection / base url by @bxyu-nvidia in #75
• Swap async http backend from httpx to aiohttp; various server infra improvements by @bxyu-nvidia in #77
• Remove unnecessary GHA CI and add uv config to enable dependency scanning by @chtruong814 in #66
• VLLMModel fix whitespace stripping and unwarranted spaces by @bxyu-nvidia in #70
• Fix aggregation rounding in ng_prepare_data by @fsiino-nvidia in #76
• Add profiling; improve rollout collection usability and efficiency; add uvicorn logging filtering by @bxyu-nvidia in #79
• Delete .github/ISSUE_TEMPLATE directory by @pablo-garay in #87
• Add support for num_repeats by @mahanfathi in #99
• Comp coding fixes; lots of misc infra items by @bxyu-nvidia in #90
• chore: Update cherry-pick workflow to use v0.63.0 by @pablo-garay in #108
• Make Workbench stateful and sign commits by @abhibha-nvidia in #110
• Clean deprecated Comp coding by @bxyu-nvidia in #106
• Bxyu/misc infra 20251001 by @bxyu-nvidia in #116
• Resource Server Organization by @fsiino-nvidia in #80
• Add metrics conflict error FAQ to Readme by @fsiino-nvidia in #93
• Azure OpenAI model support by @bxyu-nvidia in #112
• Use python env for precommit hook; alter files trigger by @fsiino-nvidia in #125
• Update issue templates by @bxyu-nvidia in #152
• Add back Nemo Framework templates by @bxyu-nvidia in #153
• Fix Workbench invalid function name by @bxyu-nvidia in #167
• VLLMModel enable reasoning parsing by @bxyu-nvidia in #129
• Add Attributions for Third Party Softwares by @banghuaz-nvidia in #154
• Fix infinite OpenAI endpoint query; misc improvements by @bxyu-nvidia in #171
• docs: Add Tutorial 00 - Key Terminology by @cwing-nvidia in #180
• docs: Add tutorial README with learning path structure by @cwing-nvidia in #177
• Redirect main Gym readme to Tutorials by @bxyu-nvidia in #201
• docs: Add Tutorial 01 - Understanding Core Concepts by @cwing-nvidia in #181
• docs: Add Tutorial 09 - Configuration Management by @cwing-nvidia in #183
• Add CODE_OF_CONDUCT.md for community guidelines by @cwing-nvidia in #148
• Add SECURITY.md with NVIDIA security policy by @cwing-nvidia in #149
• Make metrics conflict criteria less strict by @fsiino-nvidia in #150
• Move tutorials to docs by @bxyu-nvidia in #205
• docs: Replace README with improved version by @cwing-nvidia in #192
• Large docs improvement PR from @cwing-nvidia by @bxyu-nvidia in #208
• Add back How-To's and FAQs by @bxyu-nvidia in #209
• Docs fixes by @bxyu-nvidia in #210
• Improve CONTRIBUTING.md by @cwing-nvidia in #151
• feat (OpenQA): Add OpenQA support with per-record regex and rescue features by @psgundecha-nv in #155
• feat(mcqa): Add custom answer extraction via template_metadata to support STEM MCQA dataset by @psgundecha-nv in #128
• Add README to docs folder by @bxyu-nvidia in #216
• Ray comp coding infra by @sdevare-nv in #195
• Misc docs fixes by @bxyu-nvidia in #218
• CLI help and command help; misc improvements by @bxyu-nvidia in #229
• Misc infra 20251024 by @bxyu-nvidia in #234
• Fix ray version mismatch by @sdevare-nv in #231
• Misc fixes 20251027 by @bxyu-nvidia in #243
• Validate server port selection by @fsiino-nvidia in #233
• bxyu/misc-infra-20251027-001 by @bxyu-nvidia in #247
• Fix input assistant messages by @bxyu-nvidia in #248
• Misc infra 20251028 002 by @bxyu-nvidia in #253
• Structured Outputs JSON Environment by @jkyi-nvidia in #251
• Bump OpenAI version to 2.6.1; improve dependency constrain resolution by @bxyu-nvidia in #255
• Update missing header and attributions by @banghuaz-nvidia in #237
• Misc infra 20251031 by @bxyu-nvidia in #263
• Update math dataset examples and metrics by @damon-mosk-aoyama-nvidia in #265
• Misc infra 20251101 by @bxyu-nvidia in #267
• Almost-server detection and reporting by @fsiino-nvidia in #249
• Miniswe env by @sdevare-nv in #241
• Differentiate Example-only and Training Resource Servers...