Repository navigation
fix(bird_sql): distinguish result mismatches and timeouts from execution failures (#3355) - #3468
Merged
yaoyu-33 merged 1 commit intoSep 18, 2026
Conversation
…ion failures (NVIDIA-NeMo#3355) Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
nemo-automation-bot
Bot
added
the
community-request
yaoyu-33
added
area:environment
yaoyu-33
enabled auto-merge
September 17, 2026 16:33
yaoyu-33
approved these changes
Sep 17, 2026
yaoyu-33
left a comment
yaoyu-33
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Result mismatches, prediction timeouts, and gold-query timeouts are now distinguished while preserving the SQL matching logic, binary rewards, and helper default behavior. All 46 bird_sql tests passed on this head; pre-commit run --all-files also passed. Live model rollouts were not rerun for this diagnostic-only change. Full CI remains pending.
yaoyu-33
added
ready-to-merge
Contributor
|
/ok to test a1e8f65 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35306561422 |
5 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3355.
resources_servers/bird_sqlpreviously reportedFailureCode.EXECUTION_ERRORwhenever the predicted query's results did not match the ground truth, conflating answer correctness failures with database/execution crashes and timeouts.Changes
FailureCode.RESULT_MISMATCH: When both gold and predicted queries execute successfully but their result sets differ,failure_reasonis set toFailureCode.RESULT_MISMATCH.FailureCode.EXECUTION_TIMEOUTandFailureCode.GOLD_EXECUTION_TIMEOUT. Updatedexecute_sqlite_asyncandexecute_and_compareineval_utils.pyto distinguishasyncio.TimeoutErrorfrom SQLite syntax/execution errors.1.0for matching execution,0.0otherwise).test_verify_mismatch_with_codeblockand added tests for both timeout outcomes inexecute_and_compareandverify.Validation
pytest resources_servers/bird_sql/tests/test_app.py(46 passed)ruff check resources_servers/bird_sql/(all checks passed)