cannot authenticate through cli, with no error to debug on dnf(linux) based system

Describe the bug

problem:
cannot authenticate through cli, with no error to debug:

❯ gh auth login -w

error connecting to github.com
check your internet connection or githubstatus.com

❯ gh --version
gh version 1.13.1 (2021-07-20)
https://github.com/cli/cli/releases/tag/v1.13.1

ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
~ ❯

Expected vs actual behavior

should login/or at least have verbose logging/error to understand what's wrong

@ideepika

@ideepika Hi, this error is a generic network connectivity error. I don't think the gh process can provide you with any more detailed information. From the machine where you are running gh from, can you verify that networking is set up correctly? You can perhaps use curl https://api.github.com to see if you can make requests to GitHub.

@mislav

curl https://api.github.com

@mislav seems to work fine:

❯ curl https://api.github.com
{
  "current_user_url": "https://api.github.com/user",
  "current_user_authorizations_html_url": "https://github.com/settings/connections/applications{/client_id}",
  "authorizations_url": "https://api.github.com/authorizations",
  "code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",
  "commit_search_url": "https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}",
  "emails_url": "https://api.github.com/user/emails",
  "emojis_url": "https://api.github.com/emojis",
  "events_url": "https://api.github.com/events",
  "feeds_url": "https://api.github.com/feeds",
  "followers_url": "https://api.github.com/user/followers",
  "following_url": "https://api.github.com/user/following{/target}",
  "gists_url": "https://api.github.com/gists{/gist_id}",
  "hub_url": "https://api.github.com/hub",
  "issue_search_url": "https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}",
  "issues_url": "https://api.github.com/issues",
  "keys_url": "https://api.github.com/user/keys",
  "label_search_url": "https://api.github.com/search/labels?q={query}&repository_id={repository_id}{&page,per_page}",
  "notifications_url": "https://api.github.com/notifications",
  "organization_url": "https://api.github.com/orgs/{org}",
  "organization_repositories_url": "https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}",
  "organization_teams_url": "https://api.github.com/orgs/{org}/teams",
  "public_gists_url": "https://api.github.com/gists/public",
  "rate_limit_url": "https://api.github.com/rate_limit",
  "repository_url": "https://api.github.com/repos/{owner}/{repo}",
  "repository_search_url": "https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}",
  "current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
  "starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
  "starred_gists_url": "https://api.github.com/gists/starred",
  "user_url": "https://api.github.com/users/{user}",
  "user_organizations_url": "https://api.github.com/user/orgs",
  "user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
  "user_search_url": "https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"
}

@ideepika

@ideepika So strange; thanks for checking. How did you install gh on your system?

@ideepika

@ideepika So strange; thanks for checking. How did you install gh on your system?

using dnf, but IIRC the nature remains same with other modes of installation.

@mislav

@mislav Aren't there ways to look into failure debug logs?

@ideepika

@ideepika Setting DEBUG=api environment variable will print detailed information about HTTPS requests. This looks like a DNS error, meaning that gh was not able to resolve the hostname github.com. I cannot think of a reason why that would be.

@mislav

@mislav

❯ DEBUG=api gh auth login -w
* Request at 2021-08-05 05:02:31.59816361 +0530 IST m=+0.021690287
* Request to https://api.github.com/repos/cli/cli/releases/latest
> GET /repos/cli/cli/releases/latest HTTP/1.1
> Host: api.github.com
> Content-Type: application/json; charset=utf-8
> User-Agent: GitHub CLI 1.13.1

* dial tcp: lookup api.github.com on 127.0.0.1:53: server misbehaving
* Request took 2.654105ms

* Request at 2021-08-05 05:02:31.60151682 +0530 IST m=+0.025043535
* Request to https://github.com/login/device/code
> POST /login/device/code HTTP/1.1
> Host: github.com
> Content-Type: application/x-www-form-urlencoded

* dial tcp: lookup github.com on 127.0.0.1:53: server misbehaving
* Request took 643.855µs
error connecting to github.com
lookup github.com on 127.0.0.1:53: server misbehaving
check your internet connection or githubstatus.com

@ideepika

@ideepika Thanks for sharing! I can't tell from over here why there would be a "server misbehaving" error message. Do you have a proxy set via https_proxy/HTTPS_PROXY environment variables or custom DNS server on this network? You can check DNS resolution with nslookup github.com or dig github.com. Also take a peek at /etc/resolv.conf.

I'm sorry, but people who don't have access to your computer are not very well positioned to debug this issue. I think you might be able to get to the bottom of it after some digging, but I'm pretty sure that this is not a bug caused by gh. Closing until there is evidence that gh itself is broken.

Run below commands to fix this:
unset HTTP_PROXY
unset HTTPS_PROXY
unset http_proxy
unset https_proxy

I ran into this issue on Arch Linux. I had an empty /etc/resolv.conf, but dns lookups still worked across the system because it used systemd for dns resolution via nss. Go must not use the nss facility, so adding nameserver 1.1.1.1 to /etc/resolv.conf fixed the issue.

I ran into this issue on Arch Linux. I had an empty /etc/resolv.conf, but dns lookups still worked across the system because it used systemd for dns resolution via nss. Go must not use the nss facility, so adding nameserver 1.1.1.1 to /etc/resolv.conf fixed the issue.

I'm on Arch too and this fixed it, thanks!

I ran into this issue on Arch Linux. I had an empty /etc/resolv.conf, but dns lookups still worked across the system because it used systemd for dns resolution via nss. Go must not use the nss facility, so adding nameserver 1.1.1.1 to /etc/resolv.conf fixed the issue.

Very Thanks !!

What is the root cause? Is the Go runtime not using the Name Service Switch? The standard API to resolve a domain name on Linux is gethostbyname(3), which should automatically use the Name Service Switch.

I tried this on Arch Linux:

package main

import (
    "fmt"
    "log"
    "net/http"
)

func main() {
    resp, err := http.Get("https://github.com/")
    if err != nil {
        log.Fatalf("HTTP error: %v", err)
    }
    fmt.Printf("response: %v\n", resp.Status)
}

It gives me “200 Ok” as expected, so this do seem like an issue specific to GitHub-CLI. Shall we re-open?

@FranklinYu

@FranklinYu If your simple program works, can you see if you can reproduce the original issue by compiling gh from source?

It should be easy: just clone this repo, run make, and try bin/gh.

I quickly looked at the go source. It looks like it depends on how the binary is compiled. I think the pure go network stack simply reads and parses resolv.conf. There is a separate network stack that uses getaddrinfo which should use nss if compiled with build tag cgo. As best I can tell it will use the pure go stack if the build tag netgo is used or cgo is not used. I'm on my phone so I can't test anything to confirm.

@drewkett

@drewkett Thanks for that analysis. We do set CGO_ENABLED=0 in our official binary/packages, effectively disabling cgo for cross-compiled binaries, so gh compiled on the host system might behave differently than gh downloaded from our official binaries.

Indeed, when I directly make after cloning the repo, I can authenticate to GitHub with the binary. I see that the Arch Linux package did disable cgo. What’s the recommendation for Linux distribution packages? If I understand correctly, it might make sense for the official build to disable cgo for better compatibility, but if we know that cgo works on a distribution (e.g. Arch Linux), maybe we should enable cgo instead?

I can try later enabling cgo in the Arch Linux package metadata, and build the package to see whether this is the only problem. Is there a way to “log out” of the CLI, so that I can try the authentication workflow again? I assume, by deleting some “cookies” file?

@FranklinYu

@FranklinYu You could try to reach out to Arch linux package maintainers to see if they would enable cgo. I'm not familiar with Arch, but I know that for Ubuntu for example, a build with cgo enabled would work for one Ubuntu version but could fail on a different version. The solution, of course, would be that we offer a different gh build for each Ubuntu version, but as we currently cannot commit to that, our official Linux builds disable cgo to increase portability.

If I build a new gh and want to test the authentication, what’s the best way? Do I simply remove ~/.config/gh/hosts.yml and try gh auth login again? Or does gh auth login accept any flag to specify the configuration root?

I figured it out for Arch Linux! I was using systemd-resolved for name resolution. I should have followed Arch Wiki to set up systemd-resolved to use the “stub mode”, in other words

ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Compiling with cgo was rejected in https://bugs.archlinux.org/task/77057. Thank Mislav for help!

I figured it out for Arch Linux! I was using systemd-resolved for name resolution. I should have followed Arch Wiki to set up systemd-resolved to use the “stub mode”, in other words
ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Very Thanks! That helped a lot

This issue is discussed in detail here: https://wiki.archlinux.org/title/Systemd-resolved#DNS

TLDR; Not all programs use systemd's resolved service and instead rely on /etc/resolv.conf.

The fix:

# ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

DNS

Software that relies on glibc's getaddrinfo(3) (or similar) will work out of the box, since, by default, /etc/nsswitch.conf is configured to use nss-resolve(8) if it is available.

To provide domain name resolution for software that reads /etc/resolv.conf directly, such as web browsers and GnuPG, systemd-resolved has four different modes for handling the file—stub, static, uplink and foreign. They are described in systemd-resolved(8) § /ETC/RESOLV.CONF. We will focus here only on the recommended mode, i.e. the stub mode which uses /run/systemd/resolve/stub-resolv.conf.

/run/systemd/resolve/stub-resolv.conf contains the local stub 127.0.0.53 as the only DNS server and a list of search domains. This is the recommended mode of operation that propagates the systemd-resolved managed configuration to all clients.

Navigation Menu

Repository navigation

cannot authenticate through cli, with no error to debug on dnf(linux) based system #4083

Describe the bug

Expected vs actual behavior

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cannot authenticate through cli, with no error to debug on dnf(linux) based system #4083

Description

Describe the bug

Expected vs actual behavior

Activity

mislav commented on Aug 4, 2021

ideepika commented on Aug 4, 2021

mislav commented on Aug 4, 2021

ideepika commented on Aug 4, 2021

ideepika commented on Aug 4, 2021

mislav commented on Aug 4, 2021

ideepika commented on Aug 4, 2021

mislav commented on Aug 5, 2021

0xjas0 commented on Aug 21, 2021

drewkett commented on Feb 17, 2022

charbelnicolas commented on Jul 9, 2022

aureliolk commented on Jan 10, 2023

FranklinYu commented on Jan 20, 2023

FranklinYu commented on Jan 20, 2023

mislav commented on Jan 23, 2023

drewkett commented on Jan 23, 2023

mislav commented on Jan 23, 2023

FranklinYu commented on Jan 28, 2023

mislav commented on Jan 30, 2023

FranklinYu commented on Feb 17, 2023

FranklinYu commented on Feb 17, 2023

miakuon commented on Mar 1, 2023

freddiehaddad commented on Jun 5, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions