What happened?
I have a strong suspicion that the 2bpp mode is emulated incorrectly on VGA.
Specifically, this is controlled by bits 5,6 of Graphics Controller register 0x05 being set to 01.
I believe that the correct behaviour for the 2bpp mode is such that, not only does the lower 2 planes get rearranged to a chunky rearrangement such that plane 0 is on the left and plane 1 is on the right, but the same happens with planes 2 and 3 as well, effectively doing some hybrid 2x2 chunky x planar mode.
The way that 86Box currently renders this mode is to completely ignore planes 2 and 3. However, the correct way to mask those planes out is via the Attribute Controller.
The documentation for the 82C451 VGA clone chipset explicitly spells out the behaviour performed there, while the OTI-037 documentation and the official IBM documentation state that it applies to both pairs of planes in a rather non-obvious way.
IBM (and almost the same for OTI-037):
When set to 1, the Shift Register Mode field (bit 5) directs the shift registers in the graphics controller to format the serial data stream with even-numbered bits from both maps on even-numbered maps, and odd-numbered bits from both maps on the odd-numbered maps. This bit is used for modes 4 and 5.
From the 82C451 documentation:

(Side note: The "Note" at the bottom is impossible to achieve on an EGA due to the Graphics Controllers being... well, 2 chips instead of 1, and they aren't connected up to allow that. If that's how VGA behaves then that could be a cheeky way to make something appear differently on VGA vs. EGA...)
For comparison:
- DOSBox-X agrees with my interpretation.
- The EGA emulation also agrees with my interpretation, although that's partly because I was the one who rewrote the scanline renderer, so there may be some confirmation bias.
- DOSBox 0.74-3 is completely wrong, as it appears to completely ignore the plane write selection register.
- QEMU seems to be treating what I've set up as being a planar mode.
Attached here is a VM and a bootable test program (assemble the disk image with FASM 1):
vga2bpp-86boxvm.zip
I believe this is the correct interpretation, and this is what the EGA renderer (as well as the SuperEGA renderer) produces:

This is what the VGA renderer produces, and at least one SVGA renderer produces:

Configuration file
[General]
vid_renderer = qt_software
dpi_scale = 0
[Machine]
machine = mr286
cpu_family = 286
cpu_speed = 25000000
cpu_multi = 1
cpu_use_dynarec = 0
fpu_softfloat = 0
time_sync = utc
cpu_waitstates = 1
cpu = 6
mem_size = 512
[Video]
gfxcard = vga
[Input devices]
mouse_type = none
[Sound]
fm_driver = nuked
[Network]
net_01_link = 0
net_02_link = 0
net_03_link = 0
net_04_link = 0
[Storage controllers]
hdc = none
cassette_mode = load
[Floppy and CD-ROM drives]
fdd_01_fn = 2x2tst0b.img
fdd_01_writeprot = 1
[SuperEGA]
memory = 256
monitor_type = 9
Operating system
Artix Linux
CPU
AMD Ryzen 9 7950X3D 16-Core Processor
86Box version
Commit at 2023-11-12T23:55:00-0500, hash cf8dea5
Build architecture
Linux - x64 (64-bit)
Build type
Download source
I built 86Box myself (please tell us more about your build configuration)
Additional context
I can be found on the RoL 86Box IRC as GreaseMonkey. I am happy to fix the 2bpp renderers (ideally by making them go via the 4bpp renderers with some remapping tables), but the prospect of doing something as extensive as the EGA renderer rewrite scares me from a compatibility perspective as well as a performance perspective - AFAIK, this affects every SVGA-capable card, too.
What happened?
I have a strong suspicion that the 2bpp mode is emulated incorrectly on VGA.
Specifically, this is controlled by bits 5,6 of Graphics Controller register 0x05 being set to 01.
I believe that the correct behaviour for the 2bpp mode is such that, not only does the lower 2 planes get rearranged to a chunky rearrangement such that plane 0 is on the left and plane 1 is on the right, but the same happens with planes 2 and 3 as well, effectively doing some hybrid 2x2 chunky x planar mode.
The way that 86Box currently renders this mode is to completely ignore planes 2 and 3. However, the correct way to mask those planes out is via the Attribute Controller.
The documentation for the 82C451 VGA clone chipset explicitly spells out the behaviour performed there, while the OTI-037 documentation and the official IBM documentation state that it applies to both pairs of planes in a rather non-obvious way.
IBM (and almost the same for OTI-037):
From the 82C451 documentation:
(Side note: The "Note" at the bottom is impossible to achieve on an EGA due to the Graphics Controllers being... well, 2 chips instead of 1, and they aren't connected up to allow that. If that's how VGA behaves then that could be a cheeky way to make something appear differently on VGA vs. EGA...)
For comparison:
Attached here is a VM and a bootable test program (assemble the disk image with FASM 1):
vga2bpp-86boxvm.zip
I believe this is the correct interpretation, and this is what the EGA renderer (as well as the SuperEGA renderer) produces:
This is what the VGA renderer produces, and at least one SVGA renderer produces:
Configuration file
Operating system
Artix Linux
CPU
AMD Ryzen 9 7950X3D 16-Core Processor
86Box version
Commit at 2023-11-12T23:55:00-0500, hash cf8dea5
Build architecture
Linux - x64 (64-bit)
Build type
Download source
I built 86Box myself (please tell us more about your build configuration)
Additional context
I can be found on the RoL 86Box IRC as GreaseMonkey. I am happy to fix the 2bpp renderers (ideally by making them go via the 4bpp renderers with some remapping tables), but the prospect of doing something as extensive as the EGA renderer rewrite scares me from a compatibility perspective as well as a performance perspective - AFAIK, this affects every SVGA-capable card, too.