Fedora 45 is a go for next week

The big news is that this week we got a few release candidates, and then finally on thursday a 'go' for fedora 45 release on tuesday.

This release did take a bit, but I think it really came together in the end. The composes were completely FINISHED, so no missing deliverables at all, we managed to get a plasma update in and there wasn't any proposed blockers that we reluctantly didn't take.

So, I hope f45 will be a very solid release for everyone.

The rest of 2026

So, after Fedora 45 is out next week, we are in november and the year is running out. There's the US thanksgiving holiday week at the end of november and then holidays in december.

I'm going to work on some storage cleanup in the coming weeks and we will likely do another update/reboot cycle before thanksgiving.

Aside that hopefully we can get some backlog items done. If there's some particular item you really would like me to work on before the end of the year, do let me know. :)

As always, comment on the fediverse: https://fosstodon.org/@nirik/117537261270743528

s390x tuning

A few weeks ago we managed to get in touch with some IBM folks who were willing and eager to help us out with our s390x setup. The problem is basically that we don't have enough resources to handle times when there's a lot of larger builds happening at once.

We quickly divided up things: Short term, they would help with any tuning that they could do to get things in a better place, then medium term they would look for additional resources for us and finally of course long term we would want to be in a place where we have all the resources we need.

The last few weeks focused on the short term performance improvements. I got them a ton of data (sysstat, sosreports) and explained how we had setup our lpar and how things worked.

Out of all that:

• They tuned the "cpu entitlement distribution" on our lpar. My understanding is that this controlls how much of real cpus the virtual cpus in the lpar get by priority.

• I allocated more cpus to each guest. I had been conservative on cpus as I was unsure what the effect of oversubscribing would be but 1.25x oversubscription should apparently be fine. This means that builders have 3 cpus where they had 2 before.

• They added more memory. This allowed us to boot on new/latest/7.x kernels. With the memory we had before, the lpar would boot on newer kernels, but then OOM kill all the guests. This increased memory allowed everything to boot up normally on the 7.x kernels. It also allowed adding memory to all the guests.

I think this has had a real impact, but we will see over the coming weeks exactly how much. Medium and Longer term efforts are ongoing.

Fedora 45 upgrades at home

I decided to upgrade my home servers to Fedora 45 (as I often do it in the weeks before a release to catch any last minute things).

On one server this went completely smoothly. No problems at all. The 1G fiber link made short work of the downloads too, so it was quite fast and easy.

However, things did not go as well on my main server. It was cruising along and applying updates and... it got a MCE and killed the update in the middle. I'm not sure why it got the error. It's not had any before or since. I sure hope it's not becoming unstable.

Luckily I was able to get in, remove some duplicate packages, reinstall a few more, run some remove --duplicates and then distro-sync again. In the middle of that I messed up and remove both the old and new dnf versions and so I had to download packages and use rpm to install them back.

Finally everything in place and... for some reason it didn't activate the lv's in my secondary raid. I am still trying to see why it didn't do that, but in the mean time I activated them manually and everything was back up and on f45.

It amazes me how messed up something can be and you can still cleanly recover from it and go back to a good state.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117457857359345126

Mass update/reboot

The mass update reboot cycle went pretty well. We also applied firmware to all the machines that had updates, which always takes a while, but probibly is worth it.

Database server upgrades

Also, as part of the outage on thursday to do the outage causing hosts updates and reboots, we moved all our database servers over to postgresql 18, rhel10, and the latest timescaledb (in the case of db-datanommer).

That also went reasonably well, and we shouldn't have to do that again for a while.

Fedora 45 infrastructure freeze coming up

This coming tuesday is the start of the infrastructure freeze for fedora 45 final release.

Out next weekend

I'm out next friday through sunday, heading to seattle for a rust concert. I'm planning on taking the train, so that should be fun, but there will not really be a recap next weekend. :)

As always, comment on the fediverse: https://fosstodon.org/@nirik/117385261098076042

A interesting interrelated issue

A while back we moved our ansible control host to rhel10. After this move, I noticed that the nightly job we run to run a --check --diff over all our playbooks was taking a really long time to finish. I added it to my todo list to look at sometime but didn't think it was super high priority.

Last week we got some zabbix alerts about disk space on our central logging host (log01). Usually when this happens, it's caused by some openshift application crash looping or sending large tracebacks over and over. However, that was not the case this time, as we had added some checks to notify if that ever started happening so we could fix things more quickly.

Looking at the (large) logs, it seemed like it was mostly builders and mostly ssh-session messages.

Thinking these two issues could somehow be related, we started digging into the check-diff run first. It seemed that ansible was connecting to the target hosts, then just disconnecting again and again in a loop. But why?

Dumping the trace output into a clanker gave a quick answer: it's an ansible-core bug. Specifically: https://github.com/ansible/ansible/issues/82685

TLDR: If you specify multiple facts gathering modules (which we do because we have one that gathers some python information), it tries to do them in parallel, but there is a bug and it fails.

So, That was the base cause of both issues. check-diff was slow because it would get stuck trying to gather facts (and get killed after 24h by the script), and it was also causing tons of log messages for each ssh connection which was filling up log01.

A simple:

ansible_facts_parallel: false

Fixed both things.

RHEL10 Database servers

Staging is all done, all servers there are postgresql 18 and rhel10 now. The last one was db-datanommer01.stg, which I did monday. Its still a complex process, but it seems working and doesn't need a intermediate rhel9 instance (as long as I am willing to upgrade the existing one's timescaledb).

So, next thursday we have a outage and I am going to try and migrate all the production database servers. I'll be glad when they are all done.

I might try and do db-datnommer production a different day. Its a lot more complex than the rest and might be good to get out of the way seperately. Also, it's not super end user facing, so it can probibly be down for a little while without too much ill effect. Thats likely to be tuesday or wed.

Mass updates/reboots/outage next week

This is going to be a big one:

• We are going to apply firmware updates on servers. This always makes reboots take a while as they have to apply the firmware upgrades.

• We have some machines that aren't booting with secureboot enabled and we want to fix those. Sadly that means a boot to setup, enable, and another boot, which takes a while.

• I am going to migrate all the prod database servers over to rhel10. This is likely to take a while, but not too bad. There's no dump restore needed, just 'fast' upgrades and shuffling things around so the old rhel9 instances are retired and the new rhel10 ones take their place.

Of course eariler in the week we will do staging and any hosts we can take out of dns, etc... so thursday will mostly just be the bare metal / vmhosts and their guests that we can't take down without an outage.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117338602297421813

RHEL10 Database migrations

So, I had finished off all the staging databases except for one the week before, so I spent a fair bit of time this last week looking at the last one. It's a pretty fun case, our db-datanommer.

This is the database that stores all the messages that go accross our message bus and allows you to search them via datagrepper. Because of the nature of the data in it, it uses the timescaledb extention on top of postgres. It's currently (in both staging and production) a rhel9 vm with postgresql 13 and timescaledb 2.10.1.

Before getting into the fun set of constraints, I'll mention that I am a paranoid sysadmin. This means I always think about 'what would happen if this migration fails there, or there' So I strongly prefer to leave something in a state where I can backout to it if something goes wrong. So, in this case I strongly prefered to just leave the existing vm alone an dmigrate to a new one, and if something blows up, can just move back to the old one.

So, the constraints were pretty fun here:

• timescaledb has a handy chart about which versions support which postgres versions. 2.10.1 doesn't support 16 or 18. No versions support both 13 and 18. 1 version does support both 13 and 16 however.

• rhel9 actually has postgresql-16 available, but as a module. Because it's a module it makes it hard to build normal rpms against it.

• dump and restore of the db would take way way way too long as a pg_dumpall (sql) dump. The pg_dump formats would be faster, but not all that much because timescaledb doesn't work with -j (N jobs) on restore.

• rhel10 has postgresql16 (default) and 18 available as seperate packages.

• When you upgrade from 16 to 18, you need to either enable data page checksums (running a tool over the db's) or make 18 start without them. They sound like a really good idea though, so I wanted to enable them

So, in order to do this under all my constraints it would require 3 vm's: The current one (untouched), another rhel9 one to sync the data to, upgrade timescaledb on, then sync that to a rhel10 one to upgrade to 16 and then another timescaledb upgrade and upgrade from 16 to 18.

In the end though, it seems like timescaledb upgrades are pretty smooth and easy, so I decided (at least in staging) to just upgrade timescaledb to 2.13.1 (which supports both 13 and 16). That allows no intermediate rhel9 vm.

So, thats the plan, but there's more gotchas I hit along the way:

• When upgrading from 13 to 16, you have to also have timescaledb from the 13 install installed in the right place for the upgrader to use it.

• You also need to modify the template postgresql.conf file the upgrader users to enable timescaledb, or else when it upgrades it blows up at the end because there's a missing extension.

• Both those last 2 things apply for 16 to 18 too.

• The upgrader also helpfully tells you to look at some log when things fail, but it's the log from the data dir it JUST DELETED BECAUSE IT FAILED! I ended up reading those logs by running the upgrader under strace. :(

After all that, I think I have a process now. I was going to do the staging one friday, but I made a stupid mistake (putting the 13 timescaledb.so in the wrong dir) so I ran out of time. Will move it monday.

Mass firmware update/mass reboot/database migrations

So, with the last database server sorted in staging, time to get production done. We are tenatively planning an outage for 2026-10-01, the thursday before final freeze. Hopefully we can get all the databases done then.

https://forge.fedoraproject.org/infra/tickets/issues/13563

Scrapers?

Of course this will jinx things, but things have been reasonably quiet on the scraper front of late. There was some activity where they were hitting our wiki really hard, but luckily it was reasonably easy to block them in that case. It did cause problems for other services however as it caused our proxy network to hit max connections.

Otherwise it's been quite nice not to have to battle them day to day so much.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117299904300111203

RHEL10 database migrations

I managed to migrate all but one database server in staging. No particular problems with those and they are all on postgresql 18 now with data page checksums enabled (well, except the single mariadb instance, and it's on 10.11.18). Hopefully I can get the production ones done in the next outage we have (probibly the week before final freeze).

The last staging database server is more interesting: db-datanommer01.stg This is using timescaledb on top of postgres and is currently the rhel9 default version: 13.

After looking around, I think this one is doable, but needs another step:

• sync a data backup off the existing server

• upgrade it to postgresql 16

• enable data page checksums

• install a new rhel10 vm

• sync the data to it

• upgrade to 18

• install the correct timescaledb extension

• take down the old instance and swap in the new

I think this will all work, just an extra 13->16 jump. I will try it out in staging next week. I'm also a bit worried about how long it will take to generate all those checksums. The datanommer db's are really large. However, if we can do it I would love to as it's a very nice feature. Doing staging should at least give us an idea.

Fedora 45 Beta is go for release next tuesday ( 2026-09-15 )

Looks like we are go for fedora 45 beta next week. I hope everyone will test it out and help us iron out any problems before the final release.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117260119278106794

RHEL10 migrations: databases

Next up on the migration train: databases. It turns out we already had one database server in staging moved to rhel10, but it was using the default postgresql 16 and since I am going to the trouble to migrate things to a new os, might as well move them to newer postgresql too.

postgresql 18 makes data page checksums default. You can disable them at startup if you really want to, but they seem like a really good idea to enable. So, I took db-koji01.stg down and added checksums and it took around 45minutes. Not super great, but not nearly as bad as it could be.

Then, the upgrade to 18 was super fast and painless.

I did some work in our ansible repo to set up rhel10 hosts with postgresql18 to start with and created a db-fas02.stg to migrate that to.

The rest of these servers will follow basically this pattern:

• Setup a rhel10/postgresql18 02 vm for each server

• Sync data from rhel9 one.

• Stop rhel9 one (OUTAGE)

• Sync data again from rhel9 one.

• Enable data page checksums

• Migrate from 16 to 18

• Profit

I hope to get the staging ones all done next week and find any issues, then we can look at scheduling an outage after Beta freeze to do all the production ones.

Laptop Battery replacement

My lenovo slim7x battery was getting pretty pathetic. It was 2 years old and it's max full capacity was something like 35% of design full and sometimes it was now refusing to charge without a reboot.

So, I ordered a new battery and installed it this last week.

This laptop is anoying to work on because, while there are screws on the bottom panel, it's also held on by clips. So you have to pry it off and it sounds like you are breaking it while doing so. Anoying.

Also on this laptop they routed all the cables around the battery. The battery has cable guides all around it. So, to replace the battery you have to move all those cables carefully and slide the old battery out and new one in, along with unplugging the bluetooth and wifi antennas, and disconnecting/connecting the battery connector.

I did manage to do it, but it took more time that I thought it would.

While I had the machine open I swapped the windows drive I had back in and updated the firmware (which there's no way to do from linux). It was quite a pain. windows took quite a while to notice that it was not 2024 and that it should check what _current_ updates were available. Did manage it in the end. I don't know what effect the newer firmware has, everything still works the same as far as I can tell.

Fedora 45 beta rc's

We have started making rc's for beta. That sure reads weird, but if you are able, please do test and file bugs.

Authentication issues

There's one weird auth issue I am aware of. Thats where services with keytabs are sometimes getting permission denied. It seems to be somehow some differing config on two of our ipa servers. Thats being tracked in https://forge.fedoraproject.org/infra/tickets/issues/13539

I am not aware of any other issues remaining. So, if you hit something, please do file a ticket and include enough information for us to try and fix it. ie, time/date, exactly what you were trying to login to, exactly what error message you got, etc.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117219935269136120

RHEL10 migrations

Bunch more progress of various machines over the last week or two, and we are down to:

• torrent01 (didn't manage to get this before beta freeze, will after)

• mailman/lists ( https://forge.fedoraproject.org/infra/tickets/issues/13519 )

• database servers ( https://forge.fedoraproject.org/infra/tickets/issues/13517 )

• rabbitmq servers ( https://forge.fedoraproject.org/infra/tickets/issues/13383 )

Nice to finish this off soon.

Authentication woes... over?

I am hopefull that we have solved the last of our auth issues late this week. There's a lot of moving parts in our authentication stack: ipa servers on the backend, noggin on the frontend (accounts.fedoraproject.org), ipsilon for identity provider (id.fedoraproject.org). All of these have had various issues recently, but we have worked through them and I did some tuning of ipsilon on thursday that seems to have really helped.

If you are still seeing any issue, please add exactly what you were trying to login to/do and time/date to our tracking ticket. ( https://forge.fedoraproject.org/infra/tickets/issues/13482 )

Fedora 45 Beta infrastructure freeze

We are in beta freeze now, so hopefully that will keep things a bit quieter and we can catch up on work in staging and docs and other like things.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117180272072423365

Mass updates/reboots/reinstalls

Much of my week was handling updates/reboots/reinstalls. Got all our instances, including our openshift clusters updated to the latest and rebooted. Also I managed to move almost the last of our vmhosts over to rhel10.

We are down to just 27 rhel9 instances left.

• 12 are db servers

• 6 are rabbitmq-servers

• 1 logserver

• 2 mailman servers

• 1 fedorapeople

• 1 storinator

• 1 torrent server

• 1 straggler vmhost that needs a disk replaced before reinstall

• 2 zabbix servers

I'm hoping to finish up the logserver, storinator and torrent servers next week, then I will start on the db servers. Hopefully doing staging first to catch any problems and then doing the prod ones in the time after beta but before final freeze.

We have a plan for the rabbitmq servers and zabbix servers. For mailman we will need to look and see what is missing in epel10 for them.

Looking forward to finishing this up and moving on to other things.

pagure.io is now read-only

The last bits got sorted out and now pagure.io is read-only. You can pull git repos and look at other content, but no login/push/api access is possible.

I don't know that we have a specific timeline for keeping this, but I think it should be a very long time. It's static with no auth so the attack surface is much smaller than the pagure instance.

Fedora 45 Beta freeze starts next tuesday

It's already getting to be time to start the Fedora 45 Beta freeze. So, if you have anything you want to land in the Beta, you best do it asap.

Laptop fun

My trusty Lenovo slim 7x that I have been using day to day all the time for the last 2 years (!) is finally showing some signs of wear. The battery at full is only 70% of it's orig full capacity. Some of the keys have really anoyingly been sticking for me (especially the 1 and down arrow keys). Sometimes they send 2 or 3 keypresses. Pretty anoying.

So, I have been pondering what to do.

I could try and replace the keyboard/battery in this slim7x, but not sure how easy that will be. This is not a very 'repairable' laptop. Even swapping the nvme drive was a super pain. The case uses clips instead of screws and sounds like it's going to shatter when you are trying to pry it open.

I could go back to my framework ryzen laptop. Should be perfectly usable. Not good enough for local ai playground, and would mean going back to x86, but otherwise it's just a reinstall and a bit of moving things around.

The newer snapdragon X2 version of the slim7x all models seem to only have 1024x768 screens, and thats a hard no from me.

There's a really nice looking x2ee hp laptop, the HP EliteBook X G2q. It has upstream support, but... man, you can configure one thats $7500. That is crazy. I could buy like 3 of the following laptops for that.

The asus zenbook a16 looks nice and has upstream support already, but I am not sure a 16" laptop will be very easy to carry around. I suppose it could be nice day to day. I'm not even sure my laptop bags would fit it. It would mean staying on aarch64, which I kinda like.

I could go all out and get a new framework 13 pro ( Ryzen™ AI 9 HX 370 ). That would get a nicer screen than my old framework, much faster, and possibily to play with local llm model stuff. Fair pile of money and going back to x86.

No great hurry to decide, and this is indeed a horrible time to buy new computer hardware sadly.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117140830512414290

Fedora 45 branched

Fedora 45 has branched off of rawhide. rawhide is now marching toward Fedora 46. Overall the actual branching went pretty reasonably, a few minor issues. There was a lot of issues with the last minute dnf repos move change that landed hours before branching. This caused a lot of work to try and get a compose with it, without reverting. Perhaps we should make sure all changes that affect the compose process have to land a week or so before branching or will just be reverted.

Some minor things:

• The new rawhide release in bodhi had 'f46' as it's branch name, but it was supposed to be 'rawhide' because this is used to match against the git branch. Easily fixed, but hopefully not something that happens next time.

• The openh264 repo is a bit of a problem at branching time. We need things setup for the new rawhide before we can build it and sign it with the new key and send it out to cisco. The choice then becomes if we want it to just 404 (not be there) or redirect it to the fedora-45 one (which is signed by the fedora-45 key). I've done the latter for now and we are syncing the new rawhide build out. Hopefully updated early next week.

• noarch_arches wasn't set on the new rawhide build tag in koji. I submitted a pr to fix the branching script for this case and it's corrected for f46-build

• eln composes were not fully resigned by the new rawhide key, the docs around this process were not very clear, we should fix them for next time. However, eln is going to just move to their own seperate key, so we just don't have to worry about this next time.

• Somehow fedora 44 base repos got their permissions messed up. I can only assume it was a script failing somewhere, but I've not been able to track it down. This meant that some mirrors deleted their f44 trees and then had to sync it again. Lots of unwanted churn when there's already a bunch of mirror churn due to the new branched compose and newly resigned rawhide.

BBR on downloads

It was suggested to me by John 'Warthog9' Hawley that we might want to look at moving our download servers to BBR. Bottleneck Bandwidth and Round-trip propagation time (BBR), is a tcp congestion control algorithm developed by Google. It's used by them at youtube and other places.

So, I switched our download servers over and... it seems to have resulted in a nice performance increase for mirrors syncing from the master mirrors.

We will see how it goes moving forward.

RHEL10 migrations

Managed to get in a few migrations this last week. There are now just 33 hosts left. Of those:

• I am hoping to do the last 5 vmhosts next thursday (see below)

• We have a plan for rabbitmq clusters (6).

• zabbix is planned soon (2)

• The last bastion server and logs server I also plan to do thursday.

• The database servers ( 12 ) I plan to start on once we are in beta freeze for staging, then prod after we are out.

• A few oddball ones will be hard to do now due to resource constraints (fedorapeople and torrent), so might defer them for now.

Outage next week

We will be doing a mass update/reboot/reinstall fest next week. Monday I am out on PTO (it's my b-day!). Tuesday will be staging, Wed a bunch of non outage causing things, and thursday the main event. Everything will get updated/rebooted, then I will reinstall the last 5 vmhosts and our last bastion server. Friday will be openshift clusters (but those should just not cause much notice).

Beta Freeze coming up

The week after next we go into beta freeze. I have to say I have thought about doing away with them, but I find them a nice time to focus on other work and relax a bit. Faster is not always better.

Scrapers

So, of course I can't post one of these these days without talking about the scrapers. (Whats the collective noun for scrapers? :)

We were getting hit really hard last weekend and early this week, but... Ryan thought to fight ai with ai and had a LLM dig through a bunch of our logs for any patterns. It managed to come up with some patterns that we likely wouldn't have seen, but blocking those things has made a MASSIVE improvement. Basically it's like they aren't even there right now.

Load on the backend for src.fedoraproject.org that had started hovering at 180 or so is down under 1 pretty much all the time now. I know that this will not last and they will change their patterns, but it's nice to have some respite at least for a little while.

As always, comment on the fediverse: https://fosstodon.org/@nirik/117101412562880287