FreeBSD considering end of ppc64 support

FreeBSD is considering retiring powerpc64 prior to branching 16, which would make FreeBSD 15 the last stable version to support the architecture. (32-bit PowerPC is already dropped as of FreeBSD 14, though both OpenBSD and NetBSD generally serve this use case, and myself I have a Mac mini G4 running a custom NetBSD kernel with code from FreeBSD for automatic restart.) Although the message says "powerpc64 and powerpc64le" it later on only makes specific reference to the big-endian port, whereas both endiannesses appear on the FreeBSD platform page and on the download server.

I asked Warner Losh for clarification but he hasn't gotten back to me yet, so it's unclear if this is meant to apply to just the BE port or both. Unlike the overall industry mood generally, especially from you underannuated whippersnappers who deal with the doublethink of big-endian math in your head but little-endian code in your repos, the BSDs have historically been friendly to big-endian hardware as a means of ensuring code correctness. Indeed, his note acknowledges this. One sticking point seems to be Power ISA's own unique architectural features but I suspect it is due at least as much to the higher cost and poorer availability of the hardware, which has admittedly been thin on the ground lately.

My preference in server operating systems is a BSD-variant and FreeBSD was going to be my own migration plan when I retire my POWER6 running AIX (big-endian, natch), but this news has made me consider OpenBSD/powerpc64 instead. This port is a little suboptimal in that it states it does not presently run under a hypervisor, but it is currently well-supported and specifically tested on Raptor hardware, which is going to be the majority of the ecosystem. Of course, OpenBSD is also a famously opinionated operating system, so there may be a little bit more of a learning curve for me personally, whereas I already have experience with FreeBSD on shared x86_64 hosting and my other BSD systems all run NetBSD (on /mac68k, /cobalt, /macppc, /alpha, /hpcmips and /hpcsh). Another possibility is to try to maintain security patches and fixes on the remnant of FreeBSD 15, though I have such limited time these days I'd prefer not to pour it into another endeavour with an obvious future dead end. I'd dearly love a NetBSD option, but there isn't a NetBSD port that runs on POWER9.

I do think it's a little unfair to cite the user base as being less than "x86 or armv7" when these were extremely common architectures in their day, and a lot of people are probably still running FreeBSD on older examples to get the most money they can out of their hardware. I can't imagine their RISC-V numbers are very high either but no one is saying that should go away, and Power ISA is still very relevant in today's big-iron shops. If and until "low-end" Power11 desktop hardware is available, we'll want all the operating system support we can get too.

Debian 13 Trixie

Debian 13 is out, which like most other architectures is one of the big three distros on ppc64le (which Debian calls ppc64el in accordance with its normal naming scheme). The current release is based on the LTS kernel 6.12, glibc 2.41 and systemd 257, and includes GNOME 48, KDE Plasma 6.3, LXDE 13, LXQt 2.1.0, Xfce 4.20, gcc 14.2, LLVM-Clang 19, bind 9.20, Perl 5.40, and Python 3.13. ppc64el is one of the supported seven architectures and boots directly on all current Raptor hardware. As with all kernels with a 4K page size, if you try to use KVM acceleration in QEMU with a 64K page size guest (the default), it will fail. This doesn't occur on Fedora because Fedora ships a kernel with a 64K page size. Fortunately, see this note for workarounds by either setting a 4K property on the guest or using a 64K page size on the OS side. Live and installer images are available.

Fedora 41 mini-review on the Blackbird and Talos II

It's Fedora upgrade time again! And in the same way I preface all these mini-reviews (see our last for Fedora 40), Fedora was one of the first mainstream distributions to support POWER9 out of the box, it's still one of the top distributions OpenPOWER denizens use and its position closest to the bleeding, ragged edge is where we see problems emerge first and get fixed (hopefully) before they move further downstream. That's why it's worth caring about it even if you yourself don't run it. Also, as always, recall both my Talos II and Blackbird are configured to come up in a text boot instead of gdm and I start the GUI manually from there. I always recommend a non-graphical boot as a recovery mechanism in case your graphics card gets whacked by something or other, and on Fedora this is easily done by ensuring the symlink /etc/systemd/system/default.target points to /lib/systemd/system/multi-user.target.

I'll also give a quick shout out here to Dan Horák at Red Hat, who responds to our user issues and maintains some additional Fedora packages in copr. Read his newly updated blog for information.

Irritatingly, dnf continues to fail to update grub2's config (bug 1921479, showing messages like 0ed84c0-p94177c1: integer expression expected during the process), so the process remains largely unchanged from F39 and F40:

dnf upgrade --refresh # upgrade prior system and DNF
grub2-mkconfig -o /boot/grub2/grub.cfg # force grub to update
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=41 # download F41 packages
dnf system-upgrade reboot # reboot into upgrader

Again there was no installation screen on either the Blackbird or T2 this time around; for both systems I needed to log in as root on an alternate VTY (Ctrl-Alt-F2 or as appropriate) and either dnf system-upgrade log --number=-1 intermittently to watch the updates, or just do journalctl -f if you want a live feed and don't mind the scrollspam. You can probably also still monitor it on the virtual TTY in the BMC web interface. Both systems then rebooted (fast reboot is disabled on both) and came up clean, again with no XFS burp on the T2! I think we can conclude the updated Petitboot firmware has resolved that issue. As usual, one more grub2-mkconfig -o /boot/grub2/grub.cfg was needed to get Petitboot's menu looking right and the install was complete.

I should also note that two people on the Raptor forums reported that their systems were unbootable after this update. However, both the Blackbird and the T2 upgraded with no hitches except for a really old F33 package on the T2 I had to blow away, so I can't replicate that here and neither can Dan.

Let's proceed with testing out the Blackbird, which is my proving ground system before upgrading my regular T2 workstation. The biggest update is that in the most recent kernel versions, especially if you haven't done any updates between the F40 upgrade and now, we finally have 1920x1200 support on the built-in BMC HDMI video!

Yes, much as it pains me to say it, Wayland has finally progressed to the point where it's useable with the ASPEED BMC framebuffer. This is a crucial point to make if you're running a GPU-less configuration, especially if you're trying to achieve a fully libre machine with no firmware blobs: you can now use Wayland with it with little obvious compromise, if you really really want.

All resolutions up to 1920x1200 are supported, including 1920x1080. This works in both GNOME and KDE Plasma 6.2, though I'll have something bad to say about GNOME in a moment.

Now, there's still reasons not to use Wayland with BMC graphics, and the big one is that the fans just ran continuously in Wayland because everything is getting crammed through llvmpipe. This is a little 4-core Blackbird and it could keep up with it, but it certainly earned its money doing so. When I switched back to Xorg using plasma6-x11-unsupported, all was quiet again.

X11 modelines never had any problems supporting BMC video, but here it is for the record.

I also commented on the Chromium build now available for ppc64le in the F40 mini-review, so it's only fair to see how far it's come. And, well, it's come a long way. Many more things work and the previous graphical glitches in Xorg seem gone now. It's still somewhat rickety and crash reports pop up intermittently as its subprocesses fart and die, and really big Wasm apps like (oh the irony) Google Earth will bring it to its knees, but casual browsing largely works fine and it can run MAME from the Internet Archive's online arcade now with reasonable performance. I still find my personal builds of Firefox with Baseline JIT to be faster with browsing and I retain my philosophical objections to Chrome and Chromium, but I can't deny the progress, so this will make others of you happy.

Something that won't make you happy was GNOME. Whether it was Wayland or Xorg, there were tons of graphical artifacts on the blob-free Blackbird (these are photographs of the screen since the framebuffer didn't always reflect the issue):

The Plasma 6.2 update doesn't seem to break anything major, so if you're blob-free, I'd just use KDE with the unsupported X11 support. It's easier on the cores and it generally works. Xfce would probably be okay too.

On the T2, other than that weir-dass F33 remainder package, the upgrade was similarly smooth. Nothing so far has seemed to regress. However, the Talos II has a Raptor-BTO AMD WX7100 card, unlike the Blackbird.

Regular readers know very well I'll complain when it's time to complain, but this time around the F41 upgrade worked well and the system seems solid. I'm sorry to hear about the problems some others have had, but I couldn't replicate them here. So far this has been one of the smoother updates, which was a profound relief.

CHRP support gone from Linux kernel

UPDATE: The patch has been retracted, see comments.

There weren't that many Common Hardware Reference Platform PowerPC systems anyway, but if you have one of them (notably systems like IBM's RS/6000 7046 Model B50, the Total Impact briQ, Motorola PowerStacks or the PegasosPPC or Pegasos II from Genesi), your days are numbered as CHRP support will be removed from the next Linux kernel release. CHRP never got much market traction; Apple, then the largest seller of PowerPC machines, only partially supported it (Old World Macs are sui generis and New World Macs are a combination of CHRP and PReP), and as a result Linux support for Power Macs — at least what remains — never completely depended on it.

At least for the Genesi machines, however, you have better and more supported options in MorphOS and AmigaOS 4, and while Power Macs should still work, they're (IMHO) more usefully served by one of the BSDs. For the real oddball machines, though, if you were running a bleeding-edge kernel this is the end of the line — unless you'd like to step in and maintain it.

Fedora 41

Just a quick note that Fedora 41 is out, the standard Linux distro we use here on our two Raptor systems. Based on kernel 6.11, he big updates are updated packaging with DNF 5 and RPM 4.20 plus Perl 5.40, Python 3.13, LLVM 19 and gcc 14.1/glibc 2.40. Fortunately there are apparently still optional Xorg support packages for GNOME and KDE Plasma. Once the package repos have stabilized a bit, I'll have the usual minireview. For now, refresh your memory with how things went with Fedora 40, and of course Fedora 38 will be EOL in about a month based on their standard schedule.

Rocky Linux 9.4

Rocky Linux 9.4 is out, based on RHEL 9.4, but, you know, free. (Note that Rocky Linux 8.9 doesn't come in a ppc64le version, so Rocky 9.x is your only choice.) If you want the stability of RHEL but don't like the pricetag and don't need the support, here's one of your options. As is typical for such point releases, this one primarily refreshes included software along with security updates. Boot, minimal and DVD ISOs are available for download.

Fedora 40 mini-review on the Blackbird and Talos II (and a taste of Chromium)

This is Chromium running on GNOME in Xorg in Fedora 40 on the Talos II. I think it says it all, really.

Now, I won't mince words here: I don't like Chromium on philosophical grounds and you shouldn't expect me to be complimentary. But I salute the work that went into making it run. I'll have more to say about that later.

Meanwhile, it's that time again: in the same way I preface all these mini-reviews, Fedora was one of the first mainstream distributions to support POWER9 out of the box, it's still one of the top distributions OpenPOWER denizens use and its position closest to the bleeding, ragged edge is where we see problems emerge first and get fixed (hopefully) before they move further downstream. That's why it's worth caring about it even if you yourself don't run it. Also, as always, recall both my T2 and Blackbird are configured to come up in a text boot instead of gdm and I start the GUI manually from there. I always recommend a non-graphical boot as a recovery mechanism in case your graphics card gets whacked by something or other, and on Fedora this is easily done by ensuring the symlink /etc/systemd/system/default.target points to /lib/systemd/system/multi-user.target.

F40 is the first release since the 2.10 Petitboot PNOR firmware update and I had high hopes that it would fix the problem with stuck XFS logs sometimes making it puke, since it leapfrogs to a much more recent kernel. Both my GPU-less 4-core Blackbird and WX7100 dual-8 Talos II were already upgraded to the current PNOR before beginning and I recommend you do the same. Don't forget to save a copy of your BOOTKERNFW flash partition if your GPU requires it since this operation will erase it (you can flash it back when it's done).

dnf still!!!!! doesn't update grub's config (bug 1921479, showing messages like 0ed84c0-p94177c1: integer expression expected during the process), so the process remains largely unchanged from F38 and F39:

dnf upgrade --refresh # upgrade prior system and DNF
grub2-mkconfig -o /boot/grub2/grub.cfg # force grub to update
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=40 # download F40 packages
dnf system-upgrade reboot # reboot into upgrader

I always do the Blackbird first as a checkpoint, and got this:

I'm not sure what the issue was, since the Blackbird mostly runs Workstation with only a few extra packages. This didn't happen on the T2. However, I crossed my fingers with --allowerasing and I was able to get it to download on the Blackbird and install.

I also should note that there was no installation screen on either the Blackbird or T2 this time around; for both systems I needed to log in as root on an alternate VTY (Ctrl-Alt-F2 or as appropriate) and dnf system-upgrade log --number=-1 intermittently to watch the updates. You can probably also still monitor it on the virtual TTY in the BMC web interface. Both systems then rebooted (fast reboot is disabled on both) and came up clean, so no XFS burp on the T2! One more grub2-mkconfig -o /boot/grub2/grub.cfg was needed to get Petitboot's menu looking right and the install was complete. I do note with approval that Fedora's boot from Petitboot to prompt was very quick this time around. Good work there.

Now, that desktop environment. I migrated to KDE from GNOME a few releases back after GNOME started messing with my themes, but KDE Plasma 6 in F40 is now Wayland-only; startplasma-x11 doesn't even exist. There is apparently an unofficial package to restore the X11 session but I haven't tried this yet due to a bigger problem I'll get to momentarily. On the GPU-less Blackbird this is a problem because Wayland remains limited to 1024x768 over the built-in HDMI output (Xorg can be coerced up to 1920x1200 with a modeline), so if you've decided to give up and embrace the KDE Wayland Wasteland, you either get to compute like it's 1999 or you get a GPU.

GNOME, on the other hand, does still work in Xorg and performs well on both the Blackbird and T2. Set your .xinitrc to

export XDG_SESSION_TYPE=x11
/usr/bin/gnome-session

and then the usual startx will bring it up from a text boot. Which brings us to this screenshot again:

Chromium is now officially available for the first time on ppc64le as a Fedora package. However, in Xorg it has many visual glitches, and this is true whether or not you have a GPU (this was taken on the Talos II, which has the Raptor BTO WX7100 workstation card).

The reason I entertained running it under Xorg at all was Plasma 6 pretty much broke my custom theme completely and a lot of my applets, even though its Wayland compositor runs fine on the Talos II. (Start it from the text prompt simply with startplasma-wayland.) But the application appears normally.

There are many problems with Chromium on ppc64le (big endian need not apply) and I suspect the major reason is because its JIT appears unfinished. In particular, it seems like most Wasm and certain other operations will make it trap, and as such it's not yet ready for prime time. I'm sure it will continue to improve and the porters are to be congratulated for their hard work on it, but I'll still be trying to get all the pieces in Firefox to go in the same direction, and once the next ESR (128) starts hitting the beta channel we'll at least have Baseline JIT acceleration available for it while I continue to struggle with Ion and Wasm.

Before that, though, I'm deciding what to do, whether to go back to GNOME or try to piece together my custom theme again in KDE. It'll need a fair bit of work. I guess this means it wasn't a good upgrade, though not because it doesn't work on OpenPOWER; it just wasn't a good upgrade, period. I certainly hope the churn will be less in F41.

Fedora 40

Fedora 40 is now out, the most current release that I personally use on my own Talos II and Blackbird systems. (This means that Fedora 38 will go EOL in about a month.) This release is presently based on kernel 6.8.7 and GNOME 46, but not the anticipated new Anaconda installer and DNF5 package manager updates. Also included are gcc 14.0, GNU binutils 2.41, glibc 2.39, gdb 14.1, Golang 1.22, LLVM 18, Ruby 3.3 and PHP 8.3.

Perhaps the biggest news for this release is that an official Chromium build is available once again for ppc64le while I still spin my wheels with the Firefox JIT (Wasm is now broken again and I have not been able to figure out why). I don't like Chromium for philosophical reasons but I'm sure it will make many of you happy.

That said, this release is probably more notorious for eliminating X11 support in the KDE Plasma 6 spin, which yours truly also uses. It will be interesting to see how well that works on the GPU-less Blackbird here since I haven't seen anything to suggest the issue with 1920x1080 through the onboard HDMI has been fixed, but the trusty old BTO WX7100 in the T2 should be fine in the Wayland Wasteland. Attempts to remove the X11 session from GNOME reportedly didn't land for this release either, so we'll see how that turns out too. I usually give the repos a few days to catch up before updating and then I'll post my usual mini-review (here's the one for F39).

OpenBSD 7.5

OpenBSD 7.5 is out with multiple kernel and SMP improvements (we love SMP improvements on our multicore beasts), more hardware support, and LibreSSL 3.9.0, OpenSSH 9.6/9.7, and LLVM-clang 16.0.6. The only headliner Power ISA specific improvement to the big-endian powerpc64 port is a smoother upgrade process, but all the other advancements are welcome too. Download from any of the many mirror sites.

Fedora 39 mini-review on the Blackbird and Talos II (and other woes)

Merry Christmas and Happy Holidays: what a long strange trip it's been trying to get to this point, between trying to get a place to live at the new $DAYJOB and fix the Blackbird, which seemed to have gotten its BMC setting scrambled, then get it and the Talos II upgraded to Fedora 39 so that I can get back to work on the Firefox JIT. Here we are finally, just in time to write up our usual mini-review (see what I wrote for Fedora 38). As I always say in these mini-reviews, Fedora was one of the first mainstream distributions to support POWER9 out of the box, it's still one of the top distributions OpenPOWER denizens use and its position closest to the bleeding, ragged edge is where we see problems emerge first and get fixed (hopefully) before they move further downstream. That's why it's worth caring about it even if you yourself don't run it.

Also, as usual, recall both my T2 and Blackbird are configured to come up in a text boot instead of gdm and I start KDE manually from there. I still test GNOME on both systems, but my primary desktop environment has been KDE Plasma for several versions now, and I always recommend a non-graphical boot as a recovery mechanism in case your graphics card gets whacked by something or other. On Fedora this is easily done by ensuring the symlink /etc/systemd/system/default.target points to /lib/systemd/system/multi-user.target.

Unfortunately, dnf kernel updates still! don't seem to properly update grub's config (basically bug 1921479, showing messages like 0ed84c0-p94177c1: integer expression expected during the process), so the process remains largely unchanged from F38:

dnf upgrade --refresh # upgrade prior system and DNF
grub2-mkconfig -o /boot/grub2/grub.cfg # force grub to update
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=39 # download F39 packages
dnf system-upgrade reboot # reboot into upgrader

I do the Blackbird first as a check, since it can afford to be down waiting for updates, but I can't have that happening with the Talos II since it's my primary workstation. Unfortunately, while the packages downloaded okay, the actual upgrade process didn't do anything. After a couple failed attempts where it rebooted back into 38 seemingly unchanged, I watched it like a hawk and observed the following:

The installer was saying that the Fedora 39 GPG signatures weren't valid "yet" and so nothing that was downloaded could be verified. This turns out to be the same basic issue that bedevils Raspberry Pi 4 and earlier models that lack a realtime clock (bug 2242759), but the Blackbird has an RTC, so what gives?

I checked the CMOS battery coin cell and got a full 3.0 volts, and couldn't find anything otherwise wrong with the installer itself. The answer was labouriously going through the logs and finding that the system time was about two years in the past, confirmed in the Petitboot shell. This gets fixed in a normal boot by chrony synching up with NTP, but that apparently doesn't happen with the installer. Worse, it looked like everything had somehow gotten scrambled in the Blackbird's BMC because I couldn't log on with the admin password and fix the time (either from the SSH server or the web interface). Time to crack the case and connect to the BMC's serial port.

Connect your terminal at 115200bps, 8-N-1 to the inside 9-pin serial port headers. We're basically following these instructions to reset the BMC's internal persistent storage, which will zap everything including the BMC password (if you're an old Mac user like me, think of this as the ASPEED equivalent of "zapping PRAM"). However, a wrinkle here is that the system can reboot multiple times, so it's important to change the bootargs as many times as it resets back to U-Boot until the kernel actually comes up.

Here's what you might see in your terminal program. Make sure the terminal program is running and the serial port is connected before you apply power to the system to start up the BMC, or you won't be early enough to talk to U-Boot.

DRAM Init-V12-DDR4
0abc1-4Gb-Done
Read margin-DL:0.3725/DH:0.3803 CK (min:0.30)


U-Boot 2016.07 (Feb 19 2020 - 11:51:39 +0000)

       Watchdog enabled
DRAM:  496 MiB
Flash: 32 MiB
In:    serial
Out:   serial
Err:   serial
Net:   aspeednic#0
Hit any key to stop autoboot:  0 
ast# setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw
ast# boot
## Loading kernel from FIT Image at 20080000 ...
   Using '[email protected]' configuration
   Trying 'kernel@1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x20080128
     Data Size:    2656176 Bytes = 2.5 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x80001000
     Entry Point:  0x80001000
     Hash algo:    sha1
     Hash value:   1815ece74a2e27241c471b9ed87885071dd9e143
   Verifying Hash Integrity ... sha1+ OK
## Loading ramdisk from FIT Image at 20080000 ...
   Using '[email protected]' configuration
   Trying 'ramdisk@1' ramdisk subimage
     Description:  obmc-phosphor-initramfs
     Type:         RAMDisk Image
     Compression:  lzma compressed
     Data Start:   0x20310f88
     Data Size:    1583941 Bytes = 1.5 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: unavailable
     Entry Point:  unavailable
     Hash algo:    sha1
     Hash value:   55e0853d6ad703d5ea225837d85223a73f7cf3a4
   Verifying Hash Integrity ... sha1
DRAM Init-V12-DDR4
0abc1-4Gb-Done
Read margin-DL:0.3745/DH:0.3784 CK (min:0.30)


U-Boot 2016.07 (Feb 19 2020 - 11:51:39 +0000)

       Watchdog enabled
DRAM:  496 MiB
Flash: 32 MiB
In:    serial
Out:   serial
Err:   serial
Net:   aspeednic#0
Hit any key to stop autoboot:  0 
ast# setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw
ast# boot
## Loading kernel from FIT Image at 20080000 ...
   Using '[email protected]' configuration
   Trying 'kernel@1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x20080128
     Data Size:    2656176 Bytes = 2.5 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x80001000
     Entry Point:  0x80001000
     Hash algo:    sha1
     Hash value:   1815ece74a2e27241c471b9ed87885071dd9e143
   Verifying Hash Integrity ... sha1+ OK
## Loading ramdisk from FIT Image at 20080000 ...
   Using '[email protected]' configuration
   Trying 'ramdisk@1' ramdisk subimage
     Description:  obmc-phosphor-initramfs
     Type:         RAMDisk Image
     Compression:  lzma compressed
     Data Start:   0x20310f88
     Data Size:    1583941 Bytes = 1.5 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: unavailable
     Entry Point:  unavailable
     Hash algo:    sha1
     Hash value:   55e0853d6ad703d5ea225837d85223a73f7cf3a4
   Verifying Hash Integrity ... sha1+ OK
## Loading fdt from FIT Image at 20080000 ...
   Using '[email protected]' configuration
   Trying '[email protected]' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x203089e8
     Data Size:    34013 Bytes = 33.2 KiB
     Architecture: ARM
     Hash algo:    sha1
     Hash value:   f53d8ad3a7c573a4903f910fd124507c73f0bbfb
   Verifying Hash Integrity ... sha1+ OK
   Booting using the fdt blob at 0x203089e8
   Loading Kernel Image ... OK
   Loading Ramdisk to 9ea16000, end 9eb98b45 ... OK
   Loading Device Tree to 9ea0a000, end 9ea154dc ... OK

Starting kernel ...

[...]

Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) 0.1.0 blackbird ttyS4

blackbird login: root
Password: 0penBmc
root@blackbird:~# flash_eraseall /dev/mtd/rwfs
Erasing 64 Kibyte @ 400000 - 100% complete.
root@blackbird:~# reboot

Notice that it fell back to U-Boot once before actually going into the OpenBMC kernel. You need to set the bootargs both times (because the watchdog is aggressive and will reboot after even a brief period of inactivity, I recommand cutting and pasting that setenv command instead of typing it). Once that's done, the default 0penBmc password will work, and we can reset the persistent storage. You may need to powercycle the BMC after this too as the manual reboot may not be enough.

With that corrected, I was able to log into the web interface and set the time manually, though I also ensured the time owner was Host (i.e., not the BMC) so that hopefully the system can deal with this itself on startup again. Petitboot confirmed the time was correct and the install succeeded. On my Blackbird I got the usual graphical progress bar on the ASPEED BMC framebuffer; on my T2 with a BTO AMD WX7100, as long as you ensure the kernel is manually selected through Petitboot on startup, you'll still get to see the install log live as text. If you forget to manually select the kernel and the system comes up to an apparently black screen, you can either monitor on the serial port, or from a connected system viewing the serial console over the BMC's web server, or by logging into another VTY with CTRL-ALT-F2 or as appropriate as root and periodically issuing dnf system-upgrade log --number=-1 to watch log updates.

The update did not cause a stuck XFS log entry on the Blackbird, but after the reboot I still had to do one more grub2-mkconfig -o /boot/grub2/grub.cfg and a restart to ensure the right kernel and version were being used. Currently the kernel version as of this writing is 6.6.7.

Fedora 39 is apparently the last hurrah for the X11 session of KDE Plasma, largely because the Fedora KDE SIG doesn't want to deal with it. While Plasma 6 still has a legacy X11 session, KDE as provided in the Fedora spin won't have it — Wayland Wasteland or bust. (In particular, this will obsolete both kwin-x11, presumably including /usr/bin/kwin_x11 and /usr/bin/startplasma-x11, and plasma-workspace-wayland.) Naturally this directly affects me personally, so let's start with the system with the worst Wayland support, our stripped-down Blackbird. All it has is the ASPEED BMC framebuffer connected over HDMI.

(GNOME Wayland started manually with XDG_SESSION_TYPE=wayland /usr/libexec/gnome-session-binary --builtin)

(KDE Plasma 5 Wayland started manually with /usr/bin/startplasma-wayland)

Both GNOME 45 (GNOME's own X11 session support removal is further behind, and doesn't appear imminent) and Plasma 5 in F39 are still limited to 1024x768 with the on-board HDMI. This is a terrible limitation that yet again remains unaddressed, especially for those people trying to run their systems completely blobless, and also affects Arctic Tern which uses the same IT66121FN HDMI transceiver PHY. While performance remains stably improved over the horrid morass it used to be, further fixes appear to be stuck on a plateau.

The good news is, a lot of the graphical glitches that plagued F38 and earlier in GNOME on the AST framebuffer (both Wayland and X11) appear to be corrected in GNOME 45. Window decorations and window movement seem to work properly again and performance is good enough. This places me in the unusual situation of recommending GNOME to blobless or no-GPU OpenPOWER users, even though it's not a particularly lightweight desktop environment, simply because you'll still be able to run it under X11 at least for a little while and you can add an X11 modeline for higher resolution — as I've done. Otherwise, start budgeting for a video card.

Plasma 5, of course, continues to work fine without a GPU in X11. Enjoy that while it lasts.

Next was the Talos II, which has a custom KDE theme and a lot more packages. Its clock is fine, so that wasn't the problem, but the install left a stuck log entry on the XFS root again and Petitboot duly puked. I haven't (quite, anyway) gotten to the point of replacing the XFS root with ext4, but I now have a script to automatically do the volume group gyrations and XFS repair on the Blackbird instead of typing in commands, and for future upgrades I'll be doing ipmitool chassis bootdev safe (at the suggestion of a commenter) before running the installer so that if I can't scan for devices, I can at least restart, immediately jump into the Petitboot shell and do repairs if necessary. Note that turning off the automatic device scan can impair autobooting, such as me remotely bringing up the system from the BMC after a power failure, so I've set it back to normal with ipmitool chassis bootdev none after the upgrade and repair were successful.

So far F39 is running fine on the T2 as well, with only a couple old compat packages that didn't make the jump (meanwhile, for those of you hanging on to F37, that support is already over). It's F40 where the problems are expected to start for me with KDE, which as of this writing is scheduled for mid-April 2024. As usual no one cares, least of all the Wayland community, which has simply fallen back on strong-arm tactics to drive adoption and functionality be damned. I'll have to evaluate my choices then based on my app mix and how well Wayland-only KDE can handle them, and it won't be just OpenPOWER users like me in that boat.

Fedora 39

Fedora 39 is out, the Linux distro I use personally on my Talos II and Blackbird systems. Since I'm doing a lot of remote work with the new $DAYJOB it might be a bit before I can sit down with it but there will be the usual mini-review (here's what I had to say about Fedora 38). This release is based on kernel 6.5 and GNOME 45, with LLVM 17, Perl 5.38, glibc 2.38 and gcc 13.2. As usual, Fedora 37 will EOL one month after this release.

However, F39 isn't really the big news, and probably won't be a very exciting (or bumpy) release. Instead, the bigger and possibly more obnoxious change will occur with Fedora 40 when the Plasma spin (which I use instead of GNOME) is expected to completely drop X11 support, and the same is likely coming for GNOME 46 which will occur in the same version. It will be interesting, to say the least, to see how this affects people running OpenPOWER systems without GPUs to be completely blob-less. I've not been terribly happy with Wayland on the Blackbird's ASpeed BMC framebuffer and I haven't seen anything to indicate that its deficiencies have improved. Wonder how well it would work on the X1 ...

OpenBSD 7.4

OpenBSD 7.4 is released, an increasingly mature BSD option for OpenPOWER systems. While there are relatively few improvements this time around specifically for the powerpc64 port, this release does have improvements for SMP (a big deal for our pervasively SMT cores), performance and security upgrades with the virtual memory manager, and updates and bugfixes to userland. You can download it, or read the entire changelog.

Linux 6.5

Linux 6.5 is released, including deprecation of the old SLAB allocator, faster PCIe waits (especially notable for us when things like SATA controllers start timing out), faster parallel direct I/O on ext4, and improvements to the workqueue. There's not a lot notable for Power ISA, though ELFv2 is now the default for 64-bit big-endian kernel builds, and if you're running Power10 this release adds support for the DEXCR SPR (Dynamic Execution Control Register) which helps to reduce speculative execution risk. Expect to see 6.5 in bleeding-edge distros like Fedora soon (and almost certainly in Fedora 39).

Linux 6.4

While we wait to see what Red Hat's new source code policy does to RHEL rebuilds like RockyLinux and Alma Linux downstream, Linux 6.4 came out this week. Aside from things like new hardware support, filesystem improvements (such as a small performance win for ext4) and more Rust code, and the removal of the old SLOB memory allocator, there's not a lot here on the Power ISA side this time around except for the removal of various older evaluation boards. Expect to see it in Fedora and more leading edge distributions shortly.

Maybe XFS truly is cursed

Bugs, bugs everywhere: for those of you — like me! — running XFS as your root in Fedora because you never got around to changing it, if Petitboot doesn't get you then the kernel will. Linux 6.3 apparently has one line of code missing in the XFS component which can corrupt metadata (thank goodness I'm on 6.2). The updated kernel is on its way to the Fedora testing repositories and should turn up as a 6.3 point release. I guess today is a good day to die go ext4 or something.

Rocky Linux 9.2 is rocky

Although Rocky Linux 9.2 emerged on Tuesday, one of the architectures wasn't ppc64le - the release was held back. This seems to be due to a Power-specific bug in the provided build of Python 3.9, and also affects RHEL 9.2 (but not, near as I can tell, Fedora 38, which ships with Python 3.11 and runs fine on my systems). There are also no build artifacts available and there is currently no ETA for repair. Because Rocky Linux's mirrorlist can't hold back just one architecture, you'll need to add --releasever 9.1 (or change /etc/dnf/vars/releasever) to ensure dnf update doesn't get polluted with later metadata until the revised architecture spin is available.

Fedora 38

Fedora 38 is out — a week early, for a change. Fedora matters to us here at Orbiting Floodgap HQ because it's what we run on our Talos II and Blackbird systems and it should matter to you because, being a bleeding edge distro, changes occur there first that tricke down to other distributions. That's why we make efforts to do mini-reviews of each release. With F38's release F36 will be End of Life in one month.

The changeset for 38 is typically extensive. Possibly the most controversial was the change to globally build with -fno-omit-frame-pointer to facilitate better profiling and debugging, particularly where debugging information is not available, but at a cost as this also takes a register out of circulation to hold the frame pointer. The performance impact seems to be limited on x86_64 but I doubt much testing was done on ppc64le, and it should be noted that PowerPC is one of the gcc targets where leaf functions wouldn't use a frame pointer anyway. Time will tell if this pays off. Builds are also now made with _FORTIFY_SOURCE=3 (up from 2) for better security, and another interesting though probably irrelevant change for most is reducing the shutdown timer in systemd to 45 seconds from 2 minutes.

On the back-end F38 ships with kernel 6.2.x and gcc 13, LLVM 16, gmake 4.4, binutils 2.39, glibc 2.37 and gdb 12.1. F38 also has a major upgrade to microdnf as dnf5, the "future of package management" that may ultimately replace dnf entirely. On the front-end F38 updates GNOME to version 44, finally with grid thumbnail view in the file picker, a big overhaul to the Settings app and many new applications, as well as more apps moving to the unthemable libadwaita (but I run KDE Plasma now, and haven't looked back). Xfce also updates to 4.18, there's a new spin for the Sway window manager, and the SDDM display manager now also defaults to Wayland (we use a text boot to log in and start X11 manually, avoiding any display manager completely).

This is the first release to include the change that blocks clients with different endianness from connecting to the X server, including XWayland, which means that the compositor has to support the configurable option too (GNOME 44 Mutter does, others may not). At least you still have the option!

We'll give the mirrors a week or two to catch up on builds and then start the transition on our own machines, with the usual mini-review to follow. Stay tuned.

FreeBSD 13.2

And hot on the heels of the latest OpenBSD release is the latest FreeBSD iteration, 13.2-RELEASE. FreeBSD has a longer track record on OpenPOWER and in my cursory estimates is the most commonly installed BSD on modern Power ISA. One big jump is that the bhyve hypervisor now supports more than 16 virtual CPUs and by default can create the same number of vCPUs as physical CPUs, which is quite useful to us once you get away from the smallest single-4 machines given all our cores are SMT-4. Additionally, for those of you running FreeBSD on a VM (such as an LPAR or under KVM), nested POWER9 radix MMU mappings are now supported on the pseries flavour, substantially reducing hypercall overhead. The Linux compatibility ABI has also been expanded and on the security side ASLR is now enabled for all 64-bit executables by default, configurable through proccontrol. Downloads are available for big-endian and little-endian. Note that the release notes indicate that all PowerPC and Power ISA releases right now must run kldxref /boot/kernel manually after an upgraded successful kernel and world installation.

OpenBSD 7.3

OpenBSD 7.3 is released. While most of the improvements are not specific to Power ISA, there's a lot we benefit from, including many kernel calls which are now "lock-free" (improving SMP performance) like mmap(2) and select(2), more device support, immutable permissions on address ranges to prevent permissions from being changed in the future — much of a running program's static address space like stack, code and most libraries is now automatically immutable — and support for execute-only memory on both Power ISA and the PowerPC 970 ("G5"). LibreSSL is updated to 3.7.2, OpenSSH is updated to 9.3, and the OS ships with LLVM/clang 13.0.0 and Perl 5.36.0. Download and install when ready, Puffy.

Linux 6.2

Linux 6.2 is out. Among its marquee updates are improved Rust-in-kernel support (strings, formatting and printing, memory allocation, macros, etc.), adding TCP Protective Load Balancing (PLB) for IPv6, reducing the overhead of read-copy update (RCU) operations using lazy callbacks, performance and RAID improvements for Btrfs, and userspace support for runtime verification with safety-critical systems. And, of course, support for Apple silicon and Retbleed sucks less on Skylake, but who cares about that around here anyway?

On the Power ISA side, probably the most noteworthy change is official support for big endian ELFv2 kernels. A nice upgrade for our Sir Mix-A-Lot brigade! Another interesting commit is the one to allow compile time support for the lharx and lbarx instructions (present on ISA v2.06/POWER7 and up). The lwarx (32-bit word) and ldarx (64-bit doubleword) load instructions, along with the corresponding store instructions stwcx. and stdcx. (and a conditional branch), are used to implement atomic load-store-compare/exchange operations by placing and checking reservations on particular memory locations. The newer instructions can do this at halfword (short) and byte level respectively (with sthcx. and stbcx.) instead of reserving at least an entire 32-bit word, reducing contention in tightly packed structs. In the future, it might also benefit the newly introduced Power ISA-specific spinlock implementation as well, which is also new in this release.

Expect 6.2 to make it to bleeding edge users and Fedora in the very near future.