Extension native messaging got significantly slower in Firefox 140 ESR
Categories
(WebExtensions :: General, defect, P2)
Tracking
(firefox-esr115 unaffected, firefox-esr140147+ verified, firefox145 wontfix, firefox146 verified, firefox147 verified)
People
(Reporter: d.rakousky, Assigned: aaiunusov)
References
(Regression)
Details
(Keywords: enterprise, perf, regression)
Attachments
(1 file, 3 external files, 1 obsolete file)
|
8.45 KB,
application/x-zip-compressed
|
Details |
Steps to reproduce:
We have observed a significant performance regression in native messaging functionality between 128esr and 140esr. Latency for native messaging communication has increased by approximately 20-25x.
For reproduction there is FirefoxNativeMessagingBugRepro.zip attached. The README.md describes in detail how to set it up. Here is a quick summary:
- Build native-host via
dotnet publish -c Release -r win-x64 --self-contained - Register native messaging manifest in the registry
- Load the extension in Firefox
- Open the extension popup and press "Run Performance Test"
- After it is done it will show a summary. There is also a
firefox-native-messaging-perf.lognext to the executable
Compare between 140esr and 128esr. We did not test the non esr versions. So it could be introduced before version 140.
Actual results:
From the test extension:
128.14.0esr (64-bit)
Messages: 100
Message Size: 1000 bytes
Total Duration: 200.00ms
Average Latency: 1.93ms
Min Latency: 1ms
Max Latency: 19ms
Messages: 1000
Message Size: 1000 bytes
Total Duration: 1988.00ms
Average Latency: 1.897ms
Min Latency: 0ms
Max Latency: 29ms
140.5.0esr (64-bit)
Messages: 100
Message Size: 1000 bytes
Total Duration: 4670.00ms
Average Latency: 46.65ms
Min Latency: 32ms
Max Latency: 57ms
Messages: 1000
Message Size: 1000 bytes
Total Duration: 46848.00ms
Average Latency: 46.771ms
Min Latency: 32ms
Max Latency: 53ms
Expected results:
Version 140esr should not be 20-25x slower. It should still take only a few milliseconds. This makes it incredibly difficult for our customers to use Firefox since we need a lot of communication. This is not an issue in any Chromium browser we tested.
Comment 1•11 months ago
|
||
The Bugbug bot thinks this bug should belong to the 'Firefox::Messaging System' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.
Updated•11 months ago
|
Updated•11 months ago
|
Comment 2•11 months ago
|
||
Hey Alex, could you please try to verify this, and if possible try to bisect which version/bug between 128 and 140 introduced this?
Comment 3•11 months ago
|
||
The nativeMessaging implementation was significantly refactored in bug 1979546, so that would be the primary suspect. That landed in ESR 140.4. If it did indeed cause the issue, then the observed performance regression should be absent in ESR 140.3 and earlier. Could you check whether that is the case?
| Reporter | ||
Comment 4•11 months ago
|
||
(In reply to Rob Wu [:robwu] from comment #3)
The nativeMessaging implementation was significantly refactored in bug 1979546, so that would be the primary suspect. That landed in ESR 140.4. If it did indeed cause the issue, then the observed performance regression should be absent in ESR 140.3 and earlier. Could you check whether that is the case?
Tested 140.3.0esr (64-bit) and 140.3.1esr (64-bit) with the reproduction I have setup, the slow behavior is still present:
140.3.0esr:
Messages: 100
Message Size: 1000 bytes
Total Duration: 4680.00ms
Average Latency: 46.73ms
Min Latency: 41ms
Max Latency: 53ms
140.3.1esr:
Messages: 100
Message Size: 1000 bytes
Total Duration: 4692.00ms
Average Latency: 46.81ms
Min Latency: 44ms
Max Latency: 53ms
Comment 5•11 months ago
|
||
I reproduced the issue on the latest Release (145.0.2/20251124145406), Beta (146.0b8/20251126090434), Nightly (147.0a1/20251126203239) and ESR 140.5.0 under Windows 11.
On the above versions - total duration is close to 4900ms, average latency is close to 50ms, min latency is above 30ms and max latency is close to 50ms. These are approximate average values as I observed them after running the performance test several times on each version.
By comparison, on ESR 128.14.0 which appears to not be affected โ total duration is close to 150ms, average latency is around 1.5ms, min latency 0-1ms and max latency around 12-15ms.
So, a performance decrease by a factor of around 30 based on what I observed.
The bisection spanning FX 128 to present day, revealed that the regression was introduced sometimes in FX 138; FX 137 not being affected and registering performance similar to ESR 128. For FX 138 and above performance dropped by a factor of around 10 and kept getting worse.
Bisection results:
2025-11-27T14:01:06.894000: DEBUG : Found commit message:
Bug 1945470 - Fix data races and enable dom.workers.timeoutmanager for Nightly, r=smaug
- data races fixed
- Workers: TimeoutBudgetManager is not static anymore
- Workers: do not update gRunningTimeoutDepth
- added UpdateCCFlag() calls
- do not update PopupState in worker case
- increased MAX_ROUND_TRIP_TIME_MS
Differential Revision: https://phabricator.services.mozilla.com/D237064
2025-11-27T14:01:06.894000: DEBUG : Did not find a branch, checking all integration branches
2025-11-27T14:01:06.894000: INFO : The bisection is done.
2025-11-27T14:01:06.897000: INFO : Stopped
Comment 6•11 months ago
|
||
:aiunusov, since you are the author of the regressor, bug 1945470, could you take a look? Also, could you set the severity field?
For more information, please visit BugBot documentation.
Comment 7•11 months ago
|
||
Alex: Is this a Windows-only issue or also affecting macOS and/or Linux?
The regressor seems to relate to timers; we use setTimeout to yield to the event loop before polling again (relevant source code for windows and Linux/macOS). If this is too long, we could consider a microtask instead (well, ideally a task schedule that runs asap but processes pending postMessage messages to the worker), but first I'd like to understand why we have this regression.
Note: the regressor changed two unit tests that measured the performance of natIveMessaging. The change was not reviewed by extension peers: https://hg-edge.mozilla.org/mozilla-central/rev/31d8a35a81df#l12.2
This test is failing intermittently as explained at https://bugzilla.mozilla.org/show_bug.cgi?id=1951522#c31 . In retrospect we should have been more concerned in bumping the expected maximum duration in the performance test.
| Assignee | ||
Comment 8•11 months ago
|
||
I am checking currently and provide a fix asap
Updated•11 months ago
|
Updated•11 months ago
|
| Assignee | ||
Updated•11 months ago
|
| Assignee | ||
Comment 9•11 months ago
|
||
| Assignee | ||
Comment 10•11 months ago
|
||
Here is the wip patch with all of my tweaks
After applying it test_ext_native_messaging_perf.js is passing with 30ms round trip
I'll clean up the patch. Remove debugging changes and push it soon
Updated•11 months ago
|
| Assignee | ||
Comment 11•11 months ago
|
||
Messages: 1000
Message Size: 1000 bytes
Total Duration: 1482.00ms
Average Latency: 1.428ms
Min Latency: 0ms
Max Latency: 22ms
Success Rate: 100.00%
Messages: 100
Message Size: 1000 bytes
Total Duration: 157.00ms
Average Latency: 1.54ms
Min Latency: 0ms
Max Latency: 8ms
Success Rate: 100.00%
| Assignee | ||
Comment 12•11 months ago
|
||
(patch is ready for the review)
| Assignee | ||
Comment 13•11 months ago
|
||
Thanks for reporting!
While patch is in review, I'd like to suggest a temporary workaround without updating the browser.
about:config --> navigate dom.min_timeout_value --> change it to 0
Updated•11 months ago
|
Updated•11 months ago
|
| Reporter | ||
Comment 14•11 months ago
|
||
(In reply to Artur Iunusov from comment #13)
Thanks for reporting!
While patch is in review, I'd like to suggest a temporary workaround without updating the browser.
about:config --> navigate dom.min_timeout_value --> change it to 0
We ran our performance tests with the suggested workaround, and can report that the tests are now passing again. Thank you the quick response here.
Regarding the workaround, I would like to know if there are any potential side effects we should be aware of. Additionally, would it be appropriate to recommend this flag to our customers as an temporary solution until a permanent fix is released?
Updated•11 months ago
|
Updated•11 months ago
|
| Assignee | ||
Comment 15•11 months ago
|
||
Hi Dominic!
setting dom.min_timeout_value to zero will give a performance slightly better then usual, in cases like yours (at the cost of CPU and battery).
It should not crash the browser or another tabs
My patch is ready. I think it should pass the reviews and uplift process till Monday.
Comment 16•11 months ago
|
||
| Assignee | ||
Updated•11 months ago
|
| Assignee | ||
Comment 17•11 months ago
•
|
||
(Looks like patch is landed successfully. I am going to create uplift requests)
Comment 18•11 months ago
|
||
firefox-release Uplift Approval Request
- User impact if declined: This patch increases an extension native messaging performance
- Code covered by automated testing: yes
- Fix verified in Nightly: yes
- Needs manual QE test: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: low
- Explanation of risk level: patch is trivial
- String changes made/needed: no
- Is Android affected?: no
| Assignee | ||
Comment 19•11 months ago
|
||
- added a chrome worker exception
Original Revision: https://phabricator.services.mozilla.com/D274356
Comment 20•11 months ago
|
||
| bugherder | ||
| Assignee | ||
Updated•11 months ago
|
Updated•11 months ago
|
| Assignee | ||
Comment 21•11 months ago
|
||
Updated•11 months ago
|
Updated•11 months ago
|
Comment 22•11 months ago
|
||
firefox-esr140 Uplift Approval Request
- User impact if declined: Extension native messaging performance
- Code covered by automated testing: yes
- Fix verified in Nightly: yes
- Needs manual QE test: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: low
- Explanation of risk level: Pref flip (no code change)
- String changes made/needed: no
- Is Android affected?: no
| Assignee | ||
Comment 23•11 months ago
|
||
https://phabricator.services.mozilla.com/D274614 - uplift request for Release (waiting for the approval)
https://phabricator.services.mozilla.com/D274675 - uplift request for ESR140 (pref flip. waiting for the approval)
Comment 24•11 months ago
|
||
@ Rob Wu @ Comment 7
Please excuse the delay with which I answered, but I was on PTO.
I only managed to check on Windows as I mentioned in Comment 5, as the STR only provided instructions for Windows.
With regards to the fix:
The issue is fixed. Tested on the latest Nightly (147.0a1/20251201213546) under Windows 11.
The performance test now shows values as before the regression, confirming the fix.
Total Duration: 126.00ms
Average Latency: 1.24ms
Min Latency: 1ms
Max Latency: 8ms
Total Duration: 121.00ms
Average Latency: 1.19ms
Min Latency: 0ms
Max Latency: 9ms
Total Duration: 119.00ms
Average Latency: 1.14ms
Min Latency: 0ms
Max Latency: 6ms
I see the report was set to REOPENED. I will mark it as Verified once itโs been set to RESOLVED.
| Assignee | ||
Comment 25•11 months ago
•
|
||
Basically, the solution for esr140 is: disable dom.workers.timeoutmanager pref.
Means, set dom.workers.timeoutmanager = false
The release and the next esr's will use dom.workers.timeoutmanager out of the box.
Comment 26•11 months ago
|
||
The patch landed in nightly and beta is affected.
:aiunusov, is this bug important enough to require an uplift?
- If yes, please nominate the patch for beta approval.
- See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
- If no, please set
status-firefox146towontfix.
For more information, please visit BugBot documentation.
Comment 27•11 months ago
|
||
firefox-beta Uplift Approval Request
- User impact if declined: extension native messaging performance will be reduced
- Code covered by automated testing: yes
- Fix verified in Nightly: yes
- Needs manual QE test: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: low
- Explanation of risk level: patch is trivial
- String changes made/needed: no
- Is Android affected?: no
| Assignee | ||
Comment 28•11 months ago
|
||
- added a chrome worker exception
Original Revision: https://phabricator.services.mozilla.com/D274356
| Assignee | ||
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Comment 29•11 months ago
|
||
(Marking as RESOLVED because the fix landed in Nightly; whether the issue got fixed on ESR is tracked by the ESR tracking flags).
Comment 30•11 months ago
|
||
Based on Comment 29 and Comment 24, closing the report as Verified Fixed and marking Nightly as Verified.
I will verify the fix for the other versions when it lands/when builds are available.
| Assignee | ||
Comment 31•11 months ago
|
||
I think uplift for ESR is still waiting for approval: https://phabricator.services.mozilla.com/D274675 (simple pref flip)
And for release: https://phabricator.services.mozilla.com/D274614
| Assignee | ||
Comment 32•11 months ago
|
||
(still waiting for approvals for esr140 and Release. In any case, just ping me in case of questions)
Updated•11 months ago
|
Updated•11 months ago
|
Comment 33•11 months ago
|
||
| uplift | ||
Updated•11 months ago
|
Updated•11 months ago
|
Comment 34•11 months ago
|
||
| uplift | ||
Comment 35•11 months ago
|
||
Verified as Fixed. Tested on the latest Firefox Release (146.0.1/20251216142450 from https://treeherder.mozilla.org/jobs?repo=mozilla-release&revision=457232b02da232ac37ab03ca7ac912b59b0ce52b) under Windows 11.
The performance test now shows values as before the regression, confirming the fix.
Comment 36•10 months ago
|
||
Hello,
Verified as Fixed. Tested on the latest Firefox Release (140.7.0esr/20251210174349 from https://treeherder.mozilla.org/jobs?repo=mozilla-esr140&revision=3e2bc3a0c0713653c32a17d521258fbde35d4a24) under Windows 11.
The performance test now shows values as before the regression, confirming the fix.
Description
•