[jkyi-nvidia]

Summary

Adds a Gym-native pipeline for generating and evaluating long-horizon synthetic conversational tool-use tasks.

The pipeline is split into five composable components:

• conversational_tool_use_domain_generation: generates domain candidates.
• conversational_tool_use_policy_tool_generation: generates, refines, validates, and judges domain policies and tool schemas.
• conversational_tool_use_scenario_generation: generates customer scenarios for accepted policy/tool artifacts.
• conversational_tool_use_agent: runs the policy-model conversation loop.
• conversational_tool_use_simulation: simulates users and tool results, maintains session state, and verifies completed trajectories.

Each component uses standard Gym agents, resource servers, model-server references, rollout JSONL, and explicit materialization steps. This PR does not change nemo_gym core infrastructure.

Design

Generation pipeline

The generation stages are independent Gym agents connected through typed rollout artifacts:

domain rollouts
  -> policy/tool inputs and rollouts
  -> scenario inputs and rollouts
  -> conversational tool-use inputs
  -> policy agent + rollout simulation

Each rollout preserves the generated artifact, source lineage, model requests and responses, parsing outcomes, and validation or judgment results. Materializers produce stable IDs and the exact row contract required by the next stage.

The pipeline supports general and proactive profiles. Prompt assets, schemas, golden policy/tool references, checked-in examples, and package-specific configuration live with the component that owns them.

Conversational rollout environment

The policy agent supports multi-turn assistant messages, sequential and parallel function calls, prefilled histories, transfers, and bounded policy-model steps. A normal assistant message does not terminate the rollout: the agent asks the simulator for the next user turn and continues until a stop condition, transfer, incomplete response, verification failure, or step limit.

The simulation resource server owns:

• per-rollout policy, tools, scenario, and conversation state;
• user and tool-result simulation;
• generated tool routing and JSON Schema validation;
• message-level and final-conversation verification;
• retryable provider-failure handling and failure labels;
• cleanup of completed or abandoned sessions.

Policy, user-simulator, tool-simulator, and judge roles are separate model-server references and can use different models or providers.

Transcript and data contracts

The returned Gym row has two aligned views:

• responses_create_params and response contain the policy-visible Responses API transcript;
• result.trajectory contains simulator-native user, agent, tool, terminal, and verification state.

The agent validates that the materialized system prompt and model-visible tools match the top-level policy and simulator tool definitions before starting a rollout.

Provider output order is preserved when a response contains assistant content and function calls. Parallel tool results are appended only after all selected response items. Each decomposed trajectory message records its source response_id and response_output_index; the complete raw response is stored once rather than duplicated on every message.

Transient seeding, generation, and verification failures use Gym's failure-sidecar contract and are excluded from scored rollout JSONL. Semantic failures remain verified rollout results with synchronized reward, invalid-reason, failure-label, and judge-diagnostic fields.

Dataset preparation

The resource-server package includes builders for checked-in examples, full general and proactive datasets, combined datasets, parallel-tool-call variants, and deterministic balanced sampling.

Builders reject malformed or incomplete tools, invalid JSON Schemas, malformed scenarios, duplicate tool names, and model-template leakage. Full generated datasets, reports, and rollout outputs are intentionally not checked in.

Validation

Automated validation

• 68 passed: resources_servers/conversational_tool_use_simulation test suite.
• 34 passed: responses_api_agents/conversational_tool_use/simulation test suite.
• Ruff and repository pre-commit hooks pass.

Live rollout validation

• A hosted-inference rollout completed through Gym's inference_provider using DeepSeek V4 Pro as the policy and Nemotron 3 Ultra as simulator and judge.
• The reward-1 trajectory exercised parallel function calls and confirmed converted output ordering as reasoning -> message -> function_call(s).

End-to-end NeMo RL validation

The materialized conversational tool-use dataset was used to train NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 through Step 30 with multi-turn GRPO. Steps 10, 20, and 30 were converted from DCP to Hugging Face format and evaluated alongside the base checkpoint on TAU2 and TAU3 BM25 artificial analysis.

Checkpoint	Airline	Retail	Telecom	TAU2 average	TAU3 BM25-AA
Base	33.50%	62.94%	44.08%	46.84%	4.95%
Step 10	36.00%	62.39%	43.53%	47.31%	4.33%
Step 20	36.00%	63.93%	44.08%	48.00%	5.36%
Step 30	35.50%	62.94%	46.49%	48.31%	4.54%

TAU2 average is the arithmetic mean across airline, retail, and telecom. This run validates the complete downstream path from dataset materialization through rollout collection, verification, training, checkpoint conversion, and Gym evaluation. TAU2 average increased from 46.84% at base to 48.31% at Step 30.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.