Closed Bug 2002517 Opened 11 months ago Closed 11 months ago

Extension native messaging got significantly slower in Firefox 140 ESR

Categories

(WebExtensions :: General, defect, P2)

Firefox 140
defect

Tracking

(firefox-esr115 unaffected, firefox-esr140147+ verified, firefox145 wontfix, firefox146 verified, firefox147 verified)

VERIFIED FIXED
147 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox-esr140 147+ verified
firefox145 --- wontfix
firefox146 --- verified
firefox147 --- verified

People

(Reporter: d.rakousky, Assigned: aaiunusov)

References

(Regression)

Details

(Keywords: enterprise, perf, regression)

Attachments

(1 file, 3 external files, 1 obsolete file)

Steps to reproduce:

We have observed a significant performance regression in native messaging functionality between 128esr and 140esr. Latency for native messaging communication has increased by approximately 20-25x.

For reproduction there is FirefoxNativeMessagingBugRepro.zip attached. The README.md describes in detail how to set it up. Here is a quick summary:

  1. Build native-host via dotnet publish -c Release -r win-x64 --self-contained
  2. Register native messaging manifest in the registry
  3. Load the extension in Firefox
  4. Open the extension popup and press "Run Performance Test"
  5. After it is done it will show a summary. There is also a firefox-native-messaging-perf.log next to the executable

Compare between 140esr and 128esr. We did not test the non esr versions. So it could be introduced before version 140.

Actual results:

From the test extension:

128.14.0esr (64-bit)

Messages: 100
Message Size: 1000 bytes
Total Duration: 200.00ms
Average Latency: 1.93ms
Min Latency: 1ms
Max Latency: 19ms

Messages: 1000
Message Size: 1000 bytes
Total Duration: 1988.00ms
Average Latency: 1.897ms
Min Latency: 0ms
Max Latency: 29ms


140.5.0esr (64-bit)

Messages: 100
Message Size: 1000 bytes
Total Duration: 4670.00ms
Average Latency: 46.65ms
Min Latency: 32ms
Max Latency: 57ms

Messages: 1000
Message Size: 1000 bytes
Total Duration: 46848.00ms
Average Latency: 46.771ms
Min Latency: 32ms
Max Latency: 53ms

Expected results:

Version 140esr should not be 20-25x slower. It should still take only a few milliseconds. This makes it incredibly difficult for our customers to use Firefox since we need a lot of communication. This is not an issue in any Chromium browser we tested.

The Bugbug bot thinks this bug should belong to the 'Firefox::Messaging System' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Messaging System
Component: Messaging System → Untriaged
Product: Firefox → WebExtensions

Hey Alex, could you please try to verify this, and if possible try to bisect which version/bug between 128 and 140 introduced this?

Flags: needinfo?(acornestean)

The nativeMessaging implementation was significantly refactored in bug 1979546, so that would be the primary suspect. That landed in ESR 140.4. If it did indeed cause the issue, then the observed performance regression should be absent in ESR 140.3 and earlier. Could you check whether that is the case?

Flags: needinfo?(d.rakousky)

(In reply to Rob Wu [:robwu] from comment #3)

The nativeMessaging implementation was significantly refactored in bug 1979546, so that would be the primary suspect. That landed in ESR 140.4. If it did indeed cause the issue, then the observed performance regression should be absent in ESR 140.3 and earlier. Could you check whether that is the case?

Tested 140.3.0esr (64-bit) and 140.3.1esr (64-bit) with the reproduction I have setup, the slow behavior is still present:

140.3.0esr:
Messages: 100
Message Size: 1000 bytes
Total Duration: 4680.00ms
Average Latency: 46.73ms
Min Latency: 41ms
Max Latency: 53ms

140.3.1esr:
Messages: 100
Message Size: 1000 bytes
Total Duration: 4692.00ms
Average Latency: 46.81ms
Min Latency: 44ms
Max Latency: 53ms

Flags: needinfo?(d.rakousky)

I reproduced the issue on the latest Release (145.0.2/20251124145406), Beta (146.0b8/20251126090434), Nightly (147.0a1/20251126203239) and ESR 140.5.0 under Windows 11.

On the above versions - total duration is close to 4900ms, average latency is close to 50ms, min latency is above 30ms and max latency is close to 50ms. These are approximate average values as I observed them after running the performance test several times on each version.

By comparison, on ESR 128.14.0 which appears to not be affected โ€“ total duration is close to 150ms, average latency is around 1.5ms, min latency 0-1ms and max latency around 12-15ms.

So, a performance decrease by a factor of around 30 based on what I observed.

The bisection spanning FX 128 to present day, revealed that the regression was introduced sometimes in FX 138; FX 137 not being affected and registering performance similar to ESR 128. For FX 138 and above performance dropped by a factor of around 10 and kept getting worse.

Bisection results:

2025-11-27T14:01:06.894000: DEBUG : Found commit message:
Bug 1945470 - Fix data races and enable dom.workers.timeoutmanager for Nightly, r=smaug

  • data races fixed
  • Workers: TimeoutBudgetManager is not static anymore
  • Workers: do not update gRunningTimeoutDepth
  • added UpdateCCFlag() calls
  • do not update PopupState in worker case
  • increased MAX_ROUND_TRIP_TIME_MS

Differential Revision: https://phabricator.services.mozilla.com/D237064

2025-11-27T14:01:06.894000: DEBUG : Did not find a branch, checking all integration branches
2025-11-27T14:01:06.894000: INFO : The bisection is done.
2025-11-27T14:01:06.897000: INFO : Stopped

Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=64d3da1d46e609750004ae82bb5013b3f258eb1a&tochange=31d8a35a81df1e12fc85c77b35c0f447610a9d4d

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(acornestean)
Regressed by: 1945470

:aiunusov, since you are the author of the regressor, bug 1945470, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Flags: needinfo?(aiunusov)

Alex: Is this a Windows-only issue or also affecting macOS and/or Linux?

The regressor seems to relate to timers; we use setTimeout to yield to the event loop before polling again (relevant source code for windows and Linux/macOS). If this is too long, we could consider a microtask instead (well, ideally a task schedule that runs asap but processes pending postMessage messages to the worker), but first I'd like to understand why we have this regression.

Note: the regressor changed two unit tests that measured the performance of natIveMessaging. The change was not reviewed by extension peers: https://hg-edge.mozilla.org/mozilla-central/rev/31d8a35a81df#l12.2

This test is failing intermittently as explained at https://bugzilla.mozilla.org/show_bug.cgi?id=1951522#c31 . In retrospect we should have been more concerned in bumping the expected maximum duration in the performance test.

Flags: needinfo?(acornestean)

I am checking currently and provide a fix asap

Flags: needinfo?(aiunusov)
Priority: -- → P2

Here is the wip patch with all of my tweaks
After applying it test_ext_native_messaging_perf.js is passing with 30ms round trip

I'll clean up the patch. Remove debugging changes and push it soon

Assignee: nobody → aiunusov
Attachment #9529652 - Attachment description: WIP: Bug 2002517 - FIX Extension native messaging got significantly slower in Firefox 140 ESR, r=smaug → Bug 2002517 - FIX Extension native messaging got significantly slower in Firefox 140 ESR, r=smaug
Status: NEW → ASSIGNED

Messages: 1000
Message Size: 1000 bytes
Total Duration: 1482.00ms
Average Latency: 1.428ms
Min Latency: 0ms
Max Latency: 22ms
Success Rate: 100.00%

Messages: 100
Message Size: 1000 bytes
Total Duration: 157.00ms
Average Latency: 1.54ms
Min Latency: 0ms
Max Latency: 8ms
Success Rate: 100.00%

(patch is ready for the review)

Thanks for reporting!

While patch is in review, I'd like to suggest a temporary workaround without updating the browser.

about:config --> navigate dom.min_timeout_value --> change it to 0

Flags: needinfo?(d.rakousky)
Attachment #9529652 - Attachment description: Bug 2002517 - FIX Extension native messaging got significantly slower in Firefox 140 ESR, r=smaug → Bug 2002517 - Lower minimum timeout for off-thread timers, r=smaug
Attachment #9529652 - Attachment description: Bug 2002517 - Lower minimum timeout for off-thread timers, r=smaug → Bug 2002517 - FIX Extension native messaging performance, r=smaug

(In reply to Artur Iunusov from comment #13)

Thanks for reporting!

While patch is in review, I'd like to suggest a temporary workaround without updating the browser.

about:config --> navigate dom.min_timeout_value --> change it to 0

We ran our performance tests with the suggested workaround, and can report that the tests are now passing again. Thank you the quick response here.

Regarding the workaround, I would like to know if there are any potential side effects we should be aware of. Additionally, would it be appropriate to recommend this flag to our customers as an temporary solution until a permanent fix is released?

Flags: needinfo?(d.rakousky) → needinfo?(aiunusov)
Attachment #9529652 - Attachment description: Bug 2002517 - FIX Extension native messaging performance, r=smaug → WIP: Bug 2002517 - FIX Extension native messaging performance, r=smaug
Attachment #9529652 - Attachment description: WIP: Bug 2002517 - FIX Extension native messaging performance, r=smaug → Bug 2002517 - FIX Extension native messaging performance, r=smaug

Hi Dominic!

setting dom.min_timeout_value to zero will give a performance slightly better then usual, in cases like yours (at the cost of CPU and battery).
It should not crash the browser or another tabs

My patch is ready. I think it should pass the reviews and uplift process till Monday.

Flags: needinfo?(aiunusov) → needinfo?(d.rakousky)
Severity: -- → S3

(Looks like patch is landed successfully. I am going to create uplift requests)

firefox-release Uplift Approval Request

  • User impact if declined: This patch increases an extension native messaging performance
  • Code covered by automated testing: yes
  • Fix verified in Nightly: yes
  • Needs manual QE test: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: patch is trivial
  • String changes made/needed: no
  • Is Android affected?: no
Attachment #9530084 - Flags: approval-mozilla-release?
  • added a chrome worker exception

Original Revision: https://phabricator.services.mozilla.com/D274356

Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → 147 Branch
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #9530084 - Attachment description: Bug 2002517 - FIX Extension native messaging performance, → Bug 2002517 - FIX Extension native messaging performance
Attachment #9530184 - Attachment description: WIP: Bug 2002517 - disable dom.workers.throttling in esr140, r=smaug → Bug 2002517 - disable dom.workers.throttling in esr140, r=smaug
Attachment #9530184 - Attachment description: Bug 2002517 - disable dom.workers.throttling in esr140, r=smaug → Bug 2002517 - disable dom.workers.timeoutmanager in esr140, r=smaug

firefox-esr140 Uplift Approval Request

  • User impact if declined: Extension native messaging performance
  • Code covered by automated testing: yes
  • Fix verified in Nightly: yes
  • Needs manual QE test: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: Pref flip (no code change)
  • String changes made/needed: no
  • Is Android affected?: no

https://phabricator.services.mozilla.com/D274614 - uplift request for Release (waiting for the approval)
https://phabricator.services.mozilla.com/D274675 - uplift request for ESR140 (pref flip. waiting for the approval)

@ Rob Wu @ Comment 7

Please excuse the delay with which I answered, but I was on PTO.

I only managed to check on Windows as I mentioned in Comment 5, as the STR only provided instructions for Windows.

With regards to the fix:

The issue is fixed. Tested on the latest Nightly (147.0a1/20251201213546) under Windows 11.

The performance test now shows values as before the regression, confirming the fix.

Total Duration: 126.00ms
Average Latency: 1.24ms
Min Latency: 1ms
Max Latency: 8ms

Total Duration: 121.00ms
Average Latency: 1.19ms
Min Latency: 0ms
Max Latency: 9ms

Total Duration: 119.00ms
Average Latency: 1.14ms
Min Latency: 0ms
Max Latency: 6ms

I see the report was set to REOPENED. I will mark it as Verified once itโ€™s been set to RESOLVED.

Flags: needinfo?(acornestean)

Basically, the solution for esr140 is: disable dom.workers.timeoutmanager pref.
Means, set dom.workers.timeoutmanager = false

The release and the next esr's will use dom.workers.timeoutmanager out of the box.

The patch landed in nightly and beta is affected.
:aiunusov, is this bug important enough to require an uplift?

For more information, please visit BugBot documentation.

Flags: needinfo?(aiunusov)

firefox-beta Uplift Approval Request

  • User impact if declined: extension native messaging performance will be reduced
  • Code covered by automated testing: yes
  • Fix verified in Nightly: yes
  • Needs manual QE test: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: patch is trivial
  • String changes made/needed: no
  • Is Android affected?: no
Attachment #9530580 - Flags: approval-mozilla-beta?
  • added a chrome worker exception

Original Revision: https://phabricator.services.mozilla.com/D274356

Flags: needinfo?(aiunusov)
Attachment #9530580 - Attachment is obsolete: true
Attachment #9530580 - Flags: approval-mozilla-beta?
Attachment #9530184 - Flags: approval-mozilla-esr140?

(Marking as RESOLVED because the fix landed in Nightly; whether the issue got fixed on ESR is tracked by the ESR tracking flags).

Status: REOPENED → RESOLVED
Closed: 11 months ago → 11 months ago
Component: Untriaged → General
Flags: needinfo?(d.rakousky)
Resolution: --- → FIXED

Based on Comment 29 and Comment 24, closing the report as Verified Fixed and marking Nightly as Verified.

I will verify the fix for the other versions when it lands/when builds are available.

Status: RESOLVED → VERIFIED

I think uplift for ESR is still waiting for approval: https://phabricator.services.mozilla.com/D274675 (simple pref flip)
And for release: https://phabricator.services.mozilla.com/D274614

(still waiting for approvals for esr140 and Release. In any case, just ping me in case of questions)

Attachment #9530184 - Flags: approval-mozilla-esr140? → approval-mozilla-esr140+
Regressions: 2005728
Attachment #9530084 - Flags: approval-mozilla-release? → approval-mozilla-release+

Verified as Fixed. Tested on the latest Firefox Release (146.0.1/20251216142450 from https://treeherder.mozilla.org/jobs?repo=mozilla-release&revision=457232b02da232ac37ab03ca7ac912b59b0ce52b) under Windows 11.

The performance test now shows values as before the regression, confirming the fix.

Hello,

Verified as Fixed. Tested on the latest Firefox Release (140.7.0esr/20251210174349 from https://treeherder.mozilla.org/jobs?repo=mozilla-esr140&revision=3e2bc3a0c0713653c32a17d521258fbde35d4a24) under Windows 11.

The performance test now shows values as before the regression, confirming the fix.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: