Skip to content

Retry D585S service-mode switch to handle post-enumeration FW readiness race - #15329

Merged
Nir-Az merged 7 commits into
realsenseai:developmentfrom
Nir-Az:d585s-safety-mode-wait
Jul 1, 2026
Merged

Nir-Az merged 7 commits into
realsenseai:developmentfrom
Nir-Az:d585s-safety-mode-wait

Conversation

@Nir-Az

@Nir-Az Nir-Az commented Jun 30, 2026 •

Copy link
Copy Markdown
Collaborator

Overview: On the D585S, the test wrappers switched safety mode immediately, but the FW can transiently reject a safety_mode change (RuntimeError: Failed to set the option to value N) — especially the first attempt after enumeration / disruptive ops under CI load. This flaked the d500 safety suite and tests entering service mode via test_device_wrapped (e.g. test_advanced_mode_support, test_depth_laser_on), bench-to-bench on identical FW.

Root cause: Transient FW readiness. A first set_option attempt often fails (blocks ~3s, throws); a retry succeeds. Across 330 logged transitions the success envelope is tight — 1 attempt (<2s) or 2 attempts (~3.6s); no transition ever succeeded beyond ~3.6s. get_safety_interface_config() isn't a reliable readiness signal, so retrying the actual set_option is what works.

Fix: tests_wrapper.set_safety_mode(safety_sensor, mode, timeout=8, interval=0.5) retry helper (uses rspy.stopwatch.Stopwatch, logs safety_mode set to <mode> after N attempt(s), Xs), routed through every D585S safety-mode transition:

  • rspy/tests_wrapper.py → start_wrapper / stop_wrapper
  • conftest.py → test_device_wrapped (enter + teardown)
  • pytest-operational-mode.py, pytest-operational-mode-stress.py, pytest-interface-config-get-set.py (post-reboot)

Timeout is 8s (~2× the 3.6s worst observed success); larger values were tried but never recovered anything a small retry didn't — the rare truly-stuck transition recovers only via a full test rerun, tracked separately in RSDEV-12512 (module-scoped test_device_wrapped can't re-setup on pytest-retry → hard error). D585S-only; other devices untouched.

CI verification (vtglnx163 + D585S 333622320169):

Jira: RSDEV-12509 (fix), RSDEV-12512 (fixture-retry fragility), RSDEV-12232 (FW-side, closed)

🤖 Generated by AI

@Nir-Az
Nir-Az requested a review from remibettan June 30, 2026 15:56

@sysrsbuild-gh-agentic sysrsbuild-gh-agentic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[rs-agentic bot] Automated review of PR #15329 — synchronize event. No Copilot reviewer detected, proceeding with full inline review.

Comment thread unit-tests/py/rspy/tests_wrapper.py Resolved
Comment thread unit-tests/py/rspy/tests_wrapper.py Resolved
Comment thread unit-tests/py/rspy/tests_wrapper.py Resolved
Comment thread unit-tests/py/rspy/tests_wrapper.py Resolved
Comment thread unit-tests/conftest.py Resolved
@Nir-Az
Nir-Az requested a review from kojokaro July 1, 2026 06:23
@Nir-Az

Nir-Az commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review (rs-agentic bot). Addressed in 1e2e5f902, with a few respectful push-backs:

Fixed (1e2e5f902):

  • Trailing newline added at EOF.
  • Stale last_exc: it's now reset each iteration, with a log.w(...) on a set-accepted-but-read-back-mismatch, so the error surfaced at timeout is never stale/irrelevant.
  • Redundant teardown import in conftest.py removed (it's already bound from the setup block across the yield).

Not changing — with rationale:

  • supports() guard before get_info(name) (start/stop_wrapper): this call predates the PR — the wrappers already used it; the PR only inserts the retry call. They run solely for live D585S device tests, where camera_info.name is always present, so it's never been an issue. Can be a separate cleanup if the team wants parity with conftest.
  • PEP-8 spacing: the spaced style matches this file's existing convention — def start_wrapper( dev = None ): / def stop_wrapper( dev = None ): directly below use the same spacing, so tightening only the new function would make it inconsistent with its neighbors. The repo is mixed here (e.g. rspy/devices.py has both time.sleep( 1 ) and time.sleep(1)). Keeping local consistency; a file-wide PEP-8 pass can be separate.
  • Stress-test retry masking regressions: two notes — (1) pytest-operational-mode-stress.py is currently @pytest.mark.skip ("disabled until HKR FW stable"), so it isn't running; (2) CI timing shows mid-session SERVICE transitions transiently need a 2nd attempt (~3.6s) even warm, so the suggested timeout=2–3s sits below the normal transient and would cause false failures. If we want fail-fast sensitivity there, ~5s is the right floor (above 3.6s, below the 8s default). Left on the 8s default for now since it's disabled; can set timeout=5 on the in-loop calls when it's re-enabled.

Already resolved: the 60s-timeout thread (reduced to 8s in c49adc8, which the bot approved) and the stale PR-description note (description updated to 8s + full-suite verification).

🤖 Generated by AI

@Nir-Az
Nir-Az merged commit d0a2ebb into realsenseai:development Jul 1, 2026
31 checks passed
@Nir-Az
Nir-Az deleted the d585s-safety-mode-wait branch July 1, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants