epic: reduce configuration friction for environment authors and users #1205

Use cases, pain points, and background

NeMo Gym's configuration system is built on Hydra + OmegaConf, which provides powerful hierarchical composition for internal infrastructure. However, as the primary interface that both environment authors and evaluation users interact with daily, the current config experience introduces significant friction at every step — from authoring a new environment, to running an existing one, to composing environments with different agents or models.

Current UX

workplace_assistant:
  resources_servers:
    workplace_assistant:
      entrypoint: app.py
      domain: agent
      verified: false
      description: Workplace assistant multi-step tool-using environment
      value: Improve multi-step tool use capability
workplace_assistant_simple_agent:
  responses_api_agents:
    simple_agent:
      entrypoint: app.py
      max_steps: 6
      resources_server:
        type: resources_servers
        name: workplace_assistant
      model_server:
        type: responses_api_models
        name: policy_model
      datasets:
        - name: train
          type: train
          jsonl_fpath: resources_servers/workplace_assistant/data/train.jsonl
          huggingface_identifier:
            repo_id: nvidia/Nemotron-RL-agent-workplace_assistant
            artifact_fpath: train.jsonl
          license: Apache 2.0
        ...

Target UX

YAML Config

# Hypothetical simplified version
data: workplace_assistant
environment: workplace_assistant
model: gpt-4.1
agent: simple_agent
max_steps: 6

CLI
Ref: #1434

gym run --data workplace_assistant --env workplace_assistant --model gpt-4.1 --agent simple_agent --max-steps 6

Friction points

This is a dependency graph expressed through stringly-typed cross-references in flat YAML. The user must mentally stitch together how these objects relate, and there's no tooling to visualize or validate the graph.

3. Cross-references are unvalidated strings

Server-to-server references use a type + name lookup:

resources_server:
  type: resources_servers
  name: workplace_assistant
model_server:
  type: responses_api_models
  name: policy_model

These are resolved at runtime with no validation at authoring time. A typo in name (e.g., workplace_assitant) produces a runtime crash, not an actionable error message. There's no schema, no autocomplete in editors, and no "did you mean?" suggestion.

5. Hard-coded relative paths assume repo-root execution

Dataset paths are specified as relative paths from the repository root:
jsonl_fpath: resources_servers/workplace_assistant/data/train.jsonl

This breaks when Gym is used as a dependency in another project (#625). Users who install Gym into their own codebase cannot resolve these paths without symlinking into the Gym repo directory.

6. Every agent-environment combination requires a separate config

Running the same benchmark with a different agent (e.g., SWE-bench with OpenHands vs. SWE Agent vs. a custom agent) requires duplicating the entire YAML config and changing one line. There is no way to compose agent + environment + model at the CLI level without creating a new file (#1192).

7. No inline documentation in generated configs

Configs generated by ng_init_resources_server contain zero comments (#191). New users cannot determine which fields are required, what valid values are, or what a field does without consulting external documentation. Example from a generated config:

simple_weather_simple_agent:
  responses_api_agents:
    simple_agent:
      entrypoint: app.py
      resources_server:
        type: resources_servers
        name: simple_weather
      model_server:
        type: responses_api_models
        name: policy_model

None of this is self-explanatory to someone unfamiliar with the codebase.

8. Config paths require knowledge of internal directory structure

To run any environment, users must specify full internal paths:

ng_run "+config_paths=[resources_servers/workplace_assistant/configs/workplace_assistant.yaml,responses_api_models/openai_model/configs/openai_model.yaml]"

This requires knowing that model configs live in responses_api_models/, environment configs in resources_servers/, and that you need to pass both. It also requires shell-quoting the brackets. Compare to a hypothetical:

gym run --env workplace_assistant --model gpt-5.5

9. Naming confusion in config values

• local_vllm_model vs vllm_model — similar names, different responsibilities, distinction unclear from name alone (local_vllm_model vs. vllm_model Naming Confusion #1107)
• domain: agent — overly broad category alongside specific ones like math and coding, with no guidance on when to use which (#600)
• the name "Resources Server" is nebulous (#395)

10. Hydra ??? placeholders produce opaque errors

Some configs use Hydra's mandatory override syntax:

container_formatter: ???
dataset_path: ???

If a user forgets to override these, the error is a Hydra MissingMandatoryValue exception rather than a helpful message explaining what value is expected and how to provide it.

11. Inconsistent dataset identifier schemas

Some configs use huggingface_identifier, others use gitlab_identifier, with different nested field names:

# HuggingFace style
huggingface_identifier:
  repo_id: nvidia/Nemotron-RL-agent-workplace_assistant
  artifact_fpath: train.jsonl
# GitLab style
gitlab_identifier:
  dataset_name: swegym_for_sweagent_and_openhands
  version: 0.0.2
  artifact_fpath: swegym-converted.jsonl

Users see both patterns across configs and don't know which to use or whether both are required.

12. No config validation tooling

There is no gym config lint or gym config validate command. Users discover config errors at server startup time, often after waiting for Ray initialization and server bootstrapping. A fast-fail validation step would catch most issues instantly.

Related Issues

#191 — Add inline comments to auto-generated YAML configs
#600 — Unclear taxonomy on domain values
#625 — Cannot run ng_run outside of the Gym repository
#1029 — Init environment CLI with composable scaffolding
#1032 — Support POSIX/GNU argument syntax for CLI
#1034 — Change CLI to tool subcommand pattern
#1107 — local_vllm_model vs vllm_model naming confusion
#1192 — Improve evaluation UX for agent developers

Design:
What files should be touched? What logic should be written?

Out of scope:
What are some items that this issue could be mistaken to cover that this issue should explicitly NOT cover?

Acceptance Criteria:

• Environment authors can define a working environment without consulting external documentation
• Users can run an environment by name without specifying internal file paths
• Agent/environment/model combinations are composable without creating new config files
• Typos and missing fields are caught before server startup
• Gym can be used as a dependency in external projects without path resolution errors