Repository navigation
cp: fix(hermes): report native token usage (4139) into r0.7.0 - #4185
Merged
Merged
Conversation
Hermes returned zero input and output tokens for every rollout, even when the pinned native harness supplied nonzero usage. Export its cache-inclusive `prompt_tokens` and `completion_tokens` through the shared host/sandbox response converter, with cache and reasoning details. Missing native totals now remain unknown instead of becoming zero. This preserves the provider-failure and partial-result behavior established in #3720 and #3554. No linked issue: this fixes a token-accounting failure found during conformance validation. Validation: - Seven regression cases failed against the original converter and pass with this change; they cover cache-inclusive totals, partial results and missing usage/details. - `gym env test +entrypoint=responses_api_agents/hermes_agent +should_validate_data=true +skip_venv_if_present=true`: 222 passed, 4 skipped. - `pre-commit run --all-files`: passed. - Final-head CI passed, including core tests, all eight server-test shards, lint and DCO. - Real ReasoningGym `manipulate_matrix`, seed 42, one task/repeat with the same model and native verifier: reward 1.0, health **healthy**, 1,235 input / 376 output tokens. Original run was unhealthy because its response reported zero tokens. Capture and observability were enabled; the live agent source hash matches this commit. - Seven fresh controlled scenarios through the native adapter: five healthy, one correctly unhealthy for intentionally omitted provider usage, one unobserved terminal-error sidecar. Before the fix: zero healthy, six unhealthy, one unobserved. P0 conformance remains incomplete for missing retry/terminal-error evidence. - Real native SWE-Pro NodeBB sandbox rollout: 91 model calls, **healthy**, no findings or unobserved rollout checks. Independent capture reconciliation matched all 1,874,133 input / 4,611 output / 1,830,400 cached input tokens. The native verifier completed with reward 0: the model supplied a plan and only created a backup file; the required `getOrCreate()` implementation remained missing. This validates accounting, not task correctness. - Sandbox run used the pinned native Hermes revision `26bb847a88493342ca1b194e0455b479073ae21d` and task image `sha256:605b6da16030d20129e0b9f9376fd494d60ec5f68f5e05f608742133bd39ae98`. A Linux-prepared runtime was mounted to meet the documented host/sandbox architecture requirement, and the existing `chat_template_kwargs_enabled=false` option was used for hosted GPT-4.1. Agent/resource/verifier code and task contents were unmodified. The all-P0 artifact assessment still lacks sandbox lifecycle evidence; healthy does not imply full conformance. This exposes the aggregates produced by pinned Hermes. It does not repair native omissions from auxiliary or delegated calls, nor infer missing usage from the health check's capture data. Both live runs exercised ordinary root calls; the SWE-Pro run completed before its 100-turn limit, so limit-summary and delegated-call accounting remain unvalidated. Review coordination: a maintainer requested consolidation with #3981, which derives usage from shared lifecycle capture and leaves direct `/v1/responses` usage null. This narrower implementation exports native Hermes aggregates without requiring capture. The approaches overlap and should be consolidated deliberately; the live results above validate this PR's implementation, not #3981. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
svcnvidia-nemo-ci
added
area:agent
Contributor
Author
|
/ok to test cd478d7 |
ananthsub
approved these changes
Oct 7, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beep boop [🤖]: Hi @giuliolovisotto 👋,