Repository navigation
[BUG] PPC64 jobs executing incredibly slowly #126
Description
In the last few days, we've seen that jobs have started running incredibly slowly. In particular we're seeing steps that are network+disk bound (e.g., downloading containers) are getting much slower.
https://github.com/pyca/cryptography/actions/runs/30634773408/job/91169617019?pr=15372 is an example job, I can find more if it'd be helpful.
Reacted by Anup Kodlekere
Metadata
Metadata
Assignees
Labels
No labels
Activity
dale-fu commentedon Jul 31, 2026
Thank you for letting us know. I suspect this could be due to the recent maintenance updates done to the ppc64le runners. From my understanding, the update window should be done by now. But I will let @anup-kodlekere confirm whether there are still any unsuspected issues on ppc64le.
alex commentedon Jul 31, 2026
I'm surprised maintenance would cause pulling images to go from 10s-of-seconds to 5+ minutes.
dale-fu commentedon Jul 31, 2026
Yeah, obviously something quite unexpected must have happened. I only handle the s390x runners so I don't have visibility myself on these ppc64le runners. But I have forward this issue to them.
anup-kodlekere commentedon Jul 31, 2026
@alex we're taking a look currently. I skimmed through the last few workflows and it seems to have returned to normal? Our team will investigate but would like to know if you're still seeing this.
alex commentedon Jul 31, 2026
It's not 100% of jobs, so I can't be certain. https://github.com/pyca/cryptography/actions/runs/30638517575/job/91182310843 is the most recent job with extreme initializing image stage.
rahulssv-ibm commentedon Aug 3, 2026
Hi @alex, thanks for your patience.
During the recent maintenance, a queueing bug caused our system to temporarily lose track of jobs, leaving them stuck in the "initializing" stage. We deployed a fix last Friday that corrects this, reducing job recovery time to under two minutes.
Please let us know if you are still facing issues. Otherwise, we will keep this open for a week and then close it.
alex commentedon Aug 3, 2026
Do you have monitoring that catches if this recurs and pages someone?
rahulssv-ibm commentedon Aug 3, 2026
Yes, we have internal monitoring in place. Our alerts actually caught this, and we deployed the fix to production before this issue was opened. However, it took some time for the backlog of jobs across all communities to fully recover.
anup-kodlekere commentedon Aug 6, 2026
@alex can you confirm if your issue is resolved? We'll keep this issue open for another 7 business days, post which this will be closed.
alex commentedon Aug 6, 2026
This issue is not presently occurring.
rahulssv-ibm commentedon Aug 6, 2026
We are currently aware that some of you are experiencing timeout errors when executing jobs. This is due to a widespread GitHub Actions outage and is not an issue with our platform. We are monitoring the situation closely. You can check GitHub's system status here: https://www.githubstatus.com/