By default git pickups my ssh key from ~/.ssh, but for pull and push operations this fails for go-git.

Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

If I do an ssh-add with my key it will start working.

This is on OSX.