Take the default answer at the downloader's retry prompt when stdin is closed - #3965

In the reports behind nltk#3167 and nltk#3144 it is the index fetch that fails,
and incr_download then yields ErrorMessage(None, ...); for a list
request it yields the id string. Answering "y" at the prompt then
crashed on msg.package.id with AttributeError. The retry now takes the
Package's id when there is one, else the id or request that failed.

The retried attempt also reports to print_error_to, where the first
attempt reported, instead of to sys.stderr.

A proxy's HTML error page, an empty body, a duplicate attribute, or a
package or collection element missing an attribute its constructor
needs made _update_index raise ElementTree.ParseError or TypeError.
Neither is caught by the (OSError, ValueError) handler around the
lookup, so download() ended in a traceback, and a non-interactive run
such as the Docker build of nltk#3167 died on it. Both are now a ValueError
with the sanitised reason, reported like a refused URL and raised only
under raise_on_error.

The index is also assigned only after every entry parsed, so a refused
index is not cached as fresh: the same Downloader refetches on the next
request and installs once the server serves a good index. The test
drives each shape through a real local server and checks the error
line, the empty download directory and the refetch.

Attack-suite cases for the prompt against the local server: with stdin
at EOF or absent, a collection with one bad package installs the others
and reports the bad one once; "e" stops before the next package; "y"
refetches the failed package once and goes on; "y" after an unusable
index refetches the index. Each is judged by what is on disk afterwards
and by the captured lines staying clean.

nltk.set_proxy(proxy, user=..., password=...) only installed
ProxyBasicAuthHandler, which answers a 407 to a plain HTTP request. An
HTTPS request goes through a CONNECT tunnel that http.client opens
before any handler sees a response, so the proxy got no credentials and
nltk.download failed with "Tunnel connection failed: 407 Proxy
Authentication Required".

Put the credentials in the proxy URL given to ProxyHandler,
percent-encoded so that an "@", ":" or "/" in them stays data. urllib
sends them as Proxy-Authorization with every request it routes through
the proxy, and urllib.request.AbstractHTTPHandler.do_open moves that
header onto the CONNECT and keeps it from the origin server.
pathsec.urlopen already copies the proxy URL from the installed opener,
so the downloader picks them up. The basic and digest handlers stay for
plain HTTP.

The scheme and host come from urllib.request._parse_proxy, the parser
ProxyHandler itself uses, so the credentials go to the host the
requests go to, also when the proxy URL holds another URL in its path,
query or fragment. Credentials already in the URL are replaced, and a
proxy without a scheme still works.

ProxyHandler adds the credentials as an ordinary header, which urllib's
redirect handler copies onto a redirected request. So the opener that
set_proxy installs gets pathsec's _ProxyCredentialsRedirectHandler from
the previous commit: a redirect to a host that bypasses the proxy
(NO_PROXY) must not receive the credentials, also through a plain
urllib.request.urlopen, and ProxyHandler adds them again when the
redirected request is proxied.

password=None is treated like the default empty password. The
(user, password) tuple that web/data.rst passed as the second argument
is still accepted; a user or password that is not exactly a str now
raises TypeError instead of being sent in its str() form. Also fix the
web/data.rst example to pass user= and password=.

The SSRF attack harness gets the set_proxy cases: the redirect cases
of the previous commit, the CONNECT of an HTTPS request (with the
credentials from set_proxy or from HTTPS_PROXY), hostile credential
strings, and proxy URLs with another URL after the host.

Fixes nltk#3109.

…luster

Under a proxy, pathsec.urlopen left the pinning handlers out because they
could not open a CONNECT tunnel. The operator's opt in vouches for the
proxy, but urllib still makes some connections itself: to a host in
NO_PROXY or the system bypass list, to a redirect target that leaves the
proxy, and to the proxy a caller's Request names. Those went out by host
name after validate_network_url had resolved it once, so a name that
validated as public could rebind to loopback before the connection.

The pinning handlers are now always installed. They leave a connection to
urllib only when ProxyHandler routed the request to one of the configured
proxies: a plain HTTP request to it, or the CONNECT of a tunnel through
it. The proxy hosts are read with urllib.request._parse_proxy, the parser
ProxyHandler uses, and the environment proxies are read once and handed
to the ProxyHandler that uses them.

The SSRF harness gets section 21: a rebinding name behind a redirect off
the proxy (credentials from the environment or from set_proxy, 302 and
307, http and https), the same name as the first hop under set_proxy,
and a Request that names its own proxy. It also pins what
_ProxyCredentialsRedirectHandler is still for, a redirect to a second
proxy, with a teeth case that puts urllib's own redirect handling back
and sees the first proxy's credentials on the second proxy's CONNECT.

test_env_proxy_opt_in_skips_pinning_handlers asserted the old design, no
pinning handlers under an opted in proxy. It is replaced by a test that
the environment's ProxyHandler keeps its proxies and that both pinning
handlers trust exactly that proxy.

Refs nltk#3109.

set_proxy percent-encodes the user name and password into the proxy URL,
but ProxyHandler unquotes them again and sends "user:password" in Basic
form. A ":" in the user name therefore moves the split: for the user
"[email protected]:8080/" the proxy reads the user "[email protected]" and a
password that starts with "8080/". The hostile credential cases compared
the header string, which hid this. RFC 7617 also allows no control
character in either part.

set_proxy now raises ValueError for a ":" in the user name or a control
character (C0 or DEL) in the credentials, before it replaces the
installed opener. The message does not repeat the credentials. A ":" in
the password is still fine.

The two harness cases that hit this, the colon in the user name and the
CR LF in the password, move to a test that they are refused and that the
proxy receives nothing, with a NUL case next to them. Their places in the
"reach the proxy as data" cases go to a user name with "@" and "/" and to
a password that reads as another proxy.

Refs nltk#3109.

The retry prompt was the only one of the downloader's eight input() calls
that survived a closed or missing stdin. python -m nltk.downloader with no
package, or nltk.download() with no argument, opens the text shell when
there is no display, as in a Docker or CI build, and each of its prompts
still ended in EOFError, or RuntimeError for a lost sys.stdin (nltk#3167).

Taking an empty line at EOF everywhere would not do: the main menu and
the config menu ask again on an empty line, so they would ask forever.
All eight prompts now go through _answer(prompt, at_eof). At EOF it gives
the answer an empty line gives, except "q" at the main menu and "m" in
the config menu. So the retry prompt goes on as before, the listing pages
on to the end, and every other prompt is left. The prompt goes through
sanitize_terminal like any other text the downloader writes.

The download harness gets section 15: the shell driven against a real
local server with stdin at EOF after each sequence a user could type, and
with no stdin at all. Its stdin fails the test instead of hanging if a
prompt keeps asking after the end. A test runs python -m nltk.downloader
in a child process with stdin at EOF and the text shell forced.

Refs nltk#3167, nltk#3144.

… a proxy"

This reverts commit 0af531e. The proxy changes are being taken out of this PR so that it only carries the downloader prompt fix; the proxy question stays with nltk#3970.

…er cluster branch"

This reverts merge commit f760989. The proxy changes are being taken out of this PR so that it only carries the downloader prompt fix; the proxy question stays with nltk#3970.