Skip to content

Update Unit Tests to run in parallel #27423

New issue
New issue
Open
Open
Update Unit Tests to run in parallel#27423
Labels
StaleNo activity for over 90 dayscomponent:testTest requests (e2e or unit)enhancementNew feature or request

Description

@NachoxMacho
NachoxMacho
opened on Apr 18, 2026

Summary

Unit tests are currently sometimes ran in parallel and sometimes ran sequentially. This leads to a lot of tests running up front but still having a relatively long trail of tests which increases the amount of time needed to execute the unit tests. For example, here is CPU usage on my machine during the tests.
Image

DONE 7180 tests, 6 skipped in 106.685s

There's an initial spike of CPU usage, but the second half of the test execution is much lower CPU execution as more tests are ran sequentially.

Motivation

This burns CI time (though not as much as the e2e tests), and slows down developers utilizing the make test functionality. As someone getting into to the codebase I find myself running make test frequently to ensure my changes aren't breaking other parts of the codebase.

I did a brief test using some ugly regex on my machine to get a ballpark number, and saw test runs drop to about a minute (test failures are due to my poor regexing and some tests not functioning in parallel).

DONE 4672 tests, 3 skipped, 1795 failures, 3 errors in 57.099s

Even if we get half the results, that's still saving ~25% of the execution time which seems like a reasonable tradeoff to add parallelism where possible.

Proposal

Where possible, we should be adding t.Parallel() to test functions and test runs. I think to make the work manageable this can be handled similar to #10592 with multiple PRs for smaller chunks to be updated.

Activity

  1. NachoxMacho
    added
    enhancementNew feature or request
    triage/pendingThis issue needs further triage to be correctly classified
    on Apr 18, 2026
  2. crenshaw-dev
    added a commit that references this issue on Apr 18, 2026

    test: add t.Parallel() to util jwt, crypto, and password tests (#27423)…

    6c1fd67
  3. ranakan19
    added
    component:testTest requests (e2e or unit)
    and removed
    triage/pendingThis issue needs further triage to be correctly classified
    on Apr 20, 2026

  4. blakepettersson commented on Apr 20, 2026

    @blakepettersson
    blakepettersson
    on Apr 20, 2026
    Member

    I did some initial testing and managed to only cut the total exec time down by ~8 seconds locally

    # master

DONE 7178 tests, 6 skipped, 3 failures in 141.650s
make: *** [test-local] Error 1

# my local branch

DONE 7178 tests, 6 skipped, 3 failures in 133.375s
make: *** [test-local] Error 1

    I excluded ~2k tests which are not trivially parallelizable due to reasons. Perhaps the gains will be much larger if we could fix those ones as well

  5. ranakan19
    added a commit that references this issue on Apr 20, 2026

    test: add t.Parallel() to util jwt, crypto, and password tests (argop…

    344397c
  6. agaudreault
    added a commit that references this issue on Apr 22, 2026

    test: add t.Parallel() to util jwt, crypto, and password tests (argop…

    7b1d605

  7. GuruduGanesh commented on Apr 24, 2026

    @GuruduGanesh
    GuruduGanesh
    on Apr 24, 2026
    Contributor

    Hi @NachoxMacho and @blakepettersson , I’d be interested in taking a first pass on this if nobody else is already working on it.

    Given the failure rate from the broad regex-based experiment, I agree this seems best handled in small PRs rather than trying to parallelize everything at once.

    My approach would be to start with one isolated subset of unit tests, review them for shared state or other parallel-safety concerns, add t.Parallel() only where isolation is clear, and then measure the impact before expanding further.

    If that sounds reasonable, I can start with a small, low-risk batch first.

  8. aali309
    added a commit that references this issue on Apr 28, 2026

    test: add t.Parallel() to util jwt, crypto, and password tests (argop…

    168e2c5

  9. NachoxMacho commented on Apr 29, 2026

    @NachoxMacho
    NachoxMacho
    on Apr 29, 2026
    Author

    I think this is going to take multiple people and quite a few PRs in order to be done, so I don't think we need to worry too much about colliding on work.

    Per the potential benefits, I do think spending time on parallelizing the non-trivial is where more of the benefit is going to be, and it's hard to get specific numbers as each run will be different. Even at an individual package level though, I think we will see more benefits. For example in just the applicationset controller test suite, adding parallelization reduce that unit test from a consistent 6-7 seconds to ~3 seconds.

    Also from some brief analysis using vgt, I see numerous tails like this:
    Image
    Where maybe a test or 2 is able to run in parallel but there is a long line of sequential tests unlike the middle of this where a whole suite of package tests can execute. This isn't the worst one I think small gains over the course of the execution would provide benefit.

  10. blakepettersson commented on May 6, 2026

    @blakepettersson
    blakepettersson
    on May 6, 2026
    Member

    @NachoxMacho I did a bit more experimentation and I managed to cut the time down to ~80s. Great tip in regards to vgt!

  11. blakepettersson commented on May 7, 2026

    @blakepettersson
    blakepettersson
    on May 7, 2026
    Member

    Feel free to work on parallelizing the tests @NachoxMacho @GuruduGanesh

  12. GuruduGanesh
    added 3 commits that reference this issue on May 10, 2026

    test: add t.Parallel() to util collections, io, rand, security tests (a…

    ef5b7cb

    test: add t.Parallel() to util collections, io, rand, security tests (a…

    6bcf4e2

    test: add t.Parallel() to util collections, io, rand, security tests (a…

    0e8e968
  13. ppapapetrou76
    added a commit that references this issue on May 19, 2026

    test: add t.Parallel() to util collections, io, rand, security tests (#…

    7799082
  14. pravindahal
    added a commit that references this issue on May 19, 2026

    test: add t.Parallel() to util jwt, crypto, and password tests (argop…

    3bf207a
  15. lohitkolluri
    added 8 commits that reference this issue on Jun 8, 2026

    test: Add t.Parallel() to util/app tests

    105fe2a

    test: Add t.Parallel() to util/app tests

    c62cf84

    test: Add t.Parallel() to util/app tests

    a1d414c

    test: Add t.Parallel() to util/ tests

    1b22dca

    test: Add t.Parallel() to util/ tests

    3321279

    test: Bulk-add t.Parallel() across util/ and fix global state races

    310449c

    test: Add t.Parallel() to unit tests across the codebase

    6e6114b

    test: Add t.Parallel() to unit tests across the codebase

    5fc0b7a
  16. Bisman-Singh
    added a commit that references this issue on Jul 17, 2026

    test: add t.Parallel() to util collections, io, rand, security tests (a…

    9cfc0b9

  17. github-actions commented on Aug 6, 2026

    @github-actions
    github-actionsBot
    on Aug 6, 2026 – with GitHub Actions
    Contributor

    This issue has been marked as stale because it has had no activity for 90 days. Please comment if this is still relevant.

  18. github-actions
    added
    StaleNo activity for over 90 days
    on Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    StaleNo activity for over 90 dayscomponent:testTest requests (e2e or unit)enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions