Repository navigation
Description
/kind bug
Description
In a M1 Mac, Downloaded the VM via podman machine init and then when trying to start it via podman machine start I get thew following error:
Starting machine "podman-machine-default"
Waiting for VM ...
Error: dial unix /var/folders/kk/vvrx0k6x2b54wnnhc0zt6rv00000gr/T/podman/podman-machine-default_ready.sock: connect: connection refused
Tried to download the arch specific VM via podman machine init --image-path=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220424.3.0/aarch64/fedora-coreos-35.20220424.3.0-qemu.aarch64.qcow2.xz and starting it again with the same error.
Steps to reproduce the issue:
-
podman machine init -
podman machine start
Describe the results you received:
Error when trying to start the machine.
Describe the results you expected:
Expected the machine to start successfully.
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:56180/run/user/504/podman/podman.sock) failed: dial tcp [::1]:56180: connect: connection refused
Output of podman info --debug:
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:56180/run/user/504/podman/podman.sock) failed: dial tcp [::1]:56180: connect: connection refused
Package info (e.g. output of rpm -q podman or apt list podman):
podman: stable 4.1.0 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/4.1.0 (174 files, 48.2MB) *
Poured from bottle on 2022-05-19 at 17:51:55
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
Install HEAD version
==> Caveats
zsh completions have been installed to:
/usr/local/share/zsh/site-functions
==> Analytics
install: 20,290 (30 days), 55,705 (90 days), 143,980 (365 days)
install-on-request: 20,245 (30 days), 55,655 (90 days), 143,917 (365 days)
build-error: 0 (30 days)
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
Activity
phenri00 commentedon May 20, 2022
Experiencing similar(?) issue. M1 Mac.
nycnewman commentedon May 22, 2022
Getting same issue. If I run in debug mode the following command is run which produces the error if run standalone:
/opt/homebrew/bin/qemu-system-aarch64 -m 27000 -smp 10 -fw_cfg name=opt/com.coreos/config,file=/Users/edwardnewman/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix://var/folders/tz/4n8rgr6n6wj5cgqzjx1dh0_80000gq/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/tz/4n8rgr6n6wj5cgqzjx1dh0_80000gq/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=podman-machine-default_ready -device virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0 -accel hvf -accel tcg -cpu cortex-a57 -M virt,highmem=off -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/edwardnewman/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -virtfs local,path=/Users/edwardnewman,mount_tag=vol0,security_model=mapped-xattr -drive if=virtio,file=/Users/edwardnewman/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-36.20220511.dev.0-qemu.aarch64.qcow2
qemu-system-aarch64: -netdev socket,id=vlan,fd=3: can't get socket option SO_TYPE
nycnewman commentedon May 23, 2022
Note that it is not possible on MacOS to downgrade to previous podman version as the tap appears to only understand 4.1.0. Was proposing to test to see if that still works as issue appears to be with qemu.
mheon commentedon May 23, 2022
@baude @ashley-cui PTAL
ktram commentedon May 25, 2022
Also running into this, and I saw the same
qemu-system-aarch64: -netdev socket,id=vlan,fd=3: can't get socket option SO_TYPEerror message as @nycnewman. After downgrading qemu, I was able to run podman.lukas-vlcek commentedon May 25, 2022
I am running into the same issue.
@ktram Would you mind sharing your steps to downgrade qemu? I installed it using brew and it seems that downgrading packages in brew is full of magic.
ashley-cui commentedon May 25, 2022
Hmm, maybe it looks like something changed in qemu 7.0? Taking a look...
lukas-vlcek commentedon May 25, 2022
I can confirm that after downgrading the qemu to
6.2.0I was able to start the machine and connect to it.Note: But this is not to say that podman 4.1.0 will work correctly with qemu 6.2.0 !!!
Now, just a quick notes about how I downgraded the qemu.
A great help was this article by @sandipb
ktram commentedon May 25, 2022
@lukas-vlcek I followed the same steps to downgrade (used this article). I think your qemu will not update on the next
brew update, because you have[email protected]installed instead ofqemu.shanesmith commentedon May 25, 2022
In case it's of any use, I'm finding that the max memory I'm able to set on a machine is
3072, otherwise I see theconnection refusederror.✔️ The following works:
✖️ The following results in the
connection refusederror:justin-f-perez commentedon May 25, 2022
I am also experiencing this issue on an M1 mac running MacOS 12.4 (21F79) with podman 4.1.0 and qemu 7.0.0 (with qemu being installed automatically by homebrew as a dependency of podman). The comment from @shanesmith allowed me to initialize and start a podman machine but I still receive connection failures if I attempt to actually run anything.
🔧 pm-clean util function
I ran this before each of the below tests
❌
podman machine init --cpus=4 --disk-size=20 --memory=3072 --timezone=local --rootful=false --nowConfirming @shanesmith comment, rootless works with memory <= 3072I can initialize and start the machine, but can't actually connect to it.
$ podman machine init \ --cpus=4 \ --disk-size=20 \ --memory=3072 \ --timezone=local \ --rootful=false \ --now Extracting compressed file Image resized. Machine init complete Waiting for VM ... Mounting volume... /Users/podman-rootless-user:/Users/podman-rootless-user Error: exit status 255 $ podman machine ls NAME VM TYPE CREATED LAST UP CPUS MEMORY DISK SIZE podman-machine-default* qemu 3 minutes ago Currently running 4 3.221GB 21.47GB $ podman run ubuntu /bin/bash Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:64227/run/user/1117123858/podman/podman.sock) failed: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures $ ll ~/.ssh/podman-machine-default Permissions Size User Date Modified Name .rw------- 411 podman-rootless-user 25 May 17:34 /Users/podman-rootless-user/.ssh/podman-machine-default $ podman machine ssh Connecting to vm podman-machine-default. To close connection, use `~.` or `exit` Warning: Permanently added '[localhost]:64227' (ED25519) to the list of known hosts. Received disconnect from ::1 port 64227:2: Too many authentication failures Disconnected from ::1 port 64227 Error: exit status 255❌
podman machine init --cpus=4 --disk-size=20 --memory=3072 --timezone=local --rootful=true --now --log-level=traceSo does rootful (but only with memory <= 3072)Same result as attempting rootless above
❌
podman machine init --cpus=4 --disk-size=20 --memory=4096 --timezone=local --rootful=true --now --log-level=trace❌
podman machine init --cpus=4 --disk-size=20 --memory=4096 --timezone=local --rootful=false --now --log-level=traceshanesmith commentedon May 26, 2022
@justin-f-perez It's probably a different issue, but I've found that the
--nowflag forpodman machine initcan be unreliable. If I can't connect to a machine after using--nowI can fix it however by stopping it then starting it again.For this issue, perhaps you could avoid
--nowand instead manually run a start after the init?justin-f-perez commentedon May 26, 2022
@shanesmith Thanks for the suggestion! It doesn't seem to matter how I initialize the machine, I get the same meaningless error 😮💨
35 remaining items
carlosgorges commentedon Jun 21, 2022
After some time debugging, I found the cause of this problem.
This problem is caused due to qemu 7.0.0 startup latency (3-5s) that occour in every first qemu execution after Mac Machine machine boots.
Podman has some bug that doesn't expect that the creation of socks files, done by the qemu call, can be delayed some seconds, and when podman tries to access the socks files, the qemu is not created them yet, showing the error "Error: dial unix /podman/podman-machine-default_ready.sock: connect: connection refused".
To avoid this problem, just execute qemu, even with invalid options (just to initialize), before call "podman machine start".
I hope help.
Carlos Eduardo Gorges.
mixpix3ls commentedon Jun 25, 2022
👋🏽 @ashley-cui and all, thanks for looking into this. I'm able to start the machine (albeit after a long time given I used 2 vCPUs, 2GB of memory and 20GB disk specs for the VM).
cpolizzi commentedon Jun 25, 2022
@mixpix3ls With your host upgraded to macOS 12.4 and using Podman 4.1.1 and ensuring that the linked fix #14563 is applied to your Podman configuration file for the QEMU
CmdLineflags as described here and fixed in #14563 then you can run with more than your current VM sizing. Yesterday I confirmed with several folks that have M1 Pro Max SoC's that once they did these things they could size the VM to as the maximum number of performance cores available (8 in their case) and more than 3GB of memory without issue.momrak commentedon Jul 11, 2022
I still have problems getting this to work. First, since I am on Podman v4.1.1 should not
-cpubehostandhighmem=onbe that by default now? I still have to make that change manually here. And also after doing that I am unable to get it running.What am I missing?
I am using nix as package manager if that is of any help
baude commentedon Jul 11, 2022
we are cutting an 4.2 rc1 today. Any chance you couild build that and try?
momrak commentedon Jul 11, 2022
Unfortunately I don't know how to override and specify this with Nix, but I'll keep an eye out for the new version and see if I get it working when it is out
waikontse commentedon Jul 14, 2022
Here is a snippet from my m1 pro on 4.2 rc1. I hope it is useful. Also, I've use podman machine set --rootful
before starting the machine. Removed the old machine with the old settings set up by podman 4.1.1
`
podman --version
podman version 4.2.0-rc1
➜ podman-4.2.0-rc1 podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Error: qemu exited unexpectedly with exit code 1, stderr: qemu-system-aarch64: -fw_cfg name=opt/com.coreos/environment,string=bm9fcHJveHk9Ii5pbnRyYW5ldCwuaW5nLm5ldCwxMjcuMC4wLjEsbG9jYWxob3N0LCouZGV2LmF6dXJlLmNvbSx3ZWIud2hhdHNhcHAuY29tInxodHRwX3Byb3h5PSJodHRwOi8vbG9jYWxob3N0OjMxMjgifGh0dHBzX3Byb3h5PSJodHRwOi8vbG9jYWxob3N0OjMxMjgifA==: duplicate fw_cfg file name: opt/com.coreos/environment
`
JonasJonny commentedon Jul 18, 2022
Today I had:
podman/podman-machine-default_ready.sock: connect: connection refusedWhat worked for me:
ericcurtin commentedon Aug 3, 2022
Highly recommend this solution if you are happy to throw away everything by the way... Although I do tend to do "--cpus $(sysctl -n hw.ncpu)", to get all my CPUs in
mkenigs commentedon Sep 23, 2022
Also using nix, and for some reason I was still running into this until 4.2
paulatwork commentedon Feb 23, 2023
EXCELLENT fix from Carlos Eduardo Gorges, on stackoverflow.
Caused due to qemu 7.0.0 startup latency (3-5s).
To fix .. execute qemu, to initialize, before call "podman machine start".
Works for me ..
https://stackoverflow.com/questions/72404926/cant-get-podman-to-run-on-an-mac-m1-monterey
justkash commentedon Apr 17, 2023
This is a problem with virtualizing memory in newer versions of Qemu (7.0+) on aarch64 machines. I resolved the issue by updating the
~/.config/containers/podman/machine/qemu/podman-machine-default.jsonfile before starting the podman machine VM. Here are the steps that worked for me:podman machine init( with-v $HOME:$HOMEif you want to be able to mount volumes)-Mentry tovirt,highmem=offin~/.config/containers/podman/machine/qemu/podman-machine-default.jsonpodman machine startjohnhamelink commentedon Jul 11, 2023
Hi folks,
I'm also attempting to use podman from within a Nix shell on
aarch64-darwin(M1 Max).After entering the shell, and allowing podman to initialise, when I run
podman machine start vm, theqmp_<machine-name>.sockfile isn't found:I've tried adjusting the memory as mentioned by @justkash above, reducing the length of the machine name (down to 'vm'). I've also ensured
-cpu=hostandhighmem=onare set by default as @momrak suggests.