bpo-34780: Fix potential hang during stdin initialization on Windows#9520
bpo-34780: Fix potential hang during stdin initialization on Windows#9520izbyshev wants to merge 1 commit intopython:2.7from
Conversation
On Windows, if there is an outstanding operation on a synchronous pipe handle (e.g., ReadFile()), many other operations will block until the first one completes. This is true even for operations that normally complete immediately (PeekNamedPipe(), SetFilePointer(), etc.). Python performs fstat(fd) during file object initialization to check for attempts to open a directory, which is forbidden. If fd refers to a pipe, msvcrt calls PeekNamedPipe() to fill in st_size member of struct stat, potentially hanging the interpreter before execution of user code when stdin object is initialized in the conditions described above. fstat() on Windows can't distinguish between a file and a directory because it relies on GetFileType(), which returns FILE_TYPE_DISK in both cases. Therefore, skipping the no-op check avoids the hang.
| return posix_error(); | ||
| } | ||
| #if defined(HAVE_FSTAT) && defined(S_IFDIR) && defined(EISDIR) | ||
| #if !defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(S_IFDIR) && \ |
There was a problem hiding this comment.
It's unlikely but still possible that fd is a directory. For example, say the directory was opened with CreateFile (via ctypes, PyWin32, etc) and the file descriptor was opened with msvcrt.open_osfhandle. Granted the fstat check has never worked on Windows, but we could actually make it work rather than skip it:
BY_HANDLE_FILE_INFORMATION info;
HANDLE hfile = (HANDLE)_get_osfhandle(fd);
if (GetFileType(hfile) == FILE_TYPE_DISK &&
GetFileInformationByHandle(hfile, &info) &&
info.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) {
...There was a problem hiding this comment.
Yes, I considered that fd can refer to a directory, but I wasn't sure whether we should bother to fix dircheck in 2.7. I mentioned in the bug report that if we do want to fix dircheck, we may use 3.x approach, which is the same that you propose (in _Py_fstat_noraise). If I get more feedback suggesting to fix dircheck, I'll update the patch. Thank you!
There was a problem hiding this comment.
This suggestion was only for this inline check in os.fdopen. I don't think dircheck in fileobject.c should even be called on Windows.
| dircheck(PyFileObject* f) | ||
| { | ||
| #if defined(HAVE_FSTAT) && defined(S_IFDIR) && defined(EISDIR) | ||
| #if !defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(S_IFDIR) && \ |
There was a problem hiding this comment.
We can't open a directory via [_w]fopen in Windows, so how about just skipping the dircheck calls in fill_file_fields and open_the_file?
There was a problem hiding this comment.
Yes, it's possible if we don't want to make dircheck work on Windows (I changed dircheck instead of its callers just because it makes a smaller patch). It also makes sense to put dircheck under #ifndef MS_WINDOWS in this case.
If dircheck is to be fixed instead, the call in fill_file_fields can't be removed because fill_file_fields is called by PyFile_FromFile which accepts FILE * of potentially unknown origin (not necessarily from fopen).
I may update the patch once there is a decision about fixing dircheck for Windows.
There was a problem hiding this comment.
For PyFile_FromFile I'm not really concerned about a C extension doing something silly. I don't think the check is necessary. It never worked, and no one ever complained that it didn't work, so my vote is to retain the status quo by either skipping the dircheck call or making it a no-op.
There was a problem hiding this comment.
Now I see your points here and above, but is there any reason to make os.fdopen behave differently compared to PyFile_FromFile (and, as a result, stdin check)? Directory checks never worked in both of them.
There was a problem hiding this comment.
My reasoning was that we can end up with an fd for a directory using PyWin32 and msvcrt.open_osfhandle. OTOH, C FILE streams are definitely outside of the domain of Python. If we consider code that directly calls CreateFile (via PyWin32 or ctypes) to be on the periphery, then the check in os.fdopen doesn't need to be fixed. It never worked, so the status quo is preserved by skipping it.
There was a problem hiding this comment.
Thank you, I understand your reasoning. A nitpick: we can still abuse PyFile_FromFile without a third-party extension module since it's used to wrap std streams.
import ctypes, msvcrt, subprocess, sys
h = ctypes.windll.kernel32.CreateFileA(
".",
0,
0,
0,
3, # OPEN_EXISTING
0x02000000, # FILE_FLAG_BACKUP_SEMANTICS
0,
)
fd = msvcrt.open_osfhandle(h, 0)
subprocess.check_call([sys.executable, '-c', 'import sys; sys.stdout.write("OK")'],
stdout=fd) The above prints the following:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
Personally, I'd either fix both os.fdopen and PyFile_FromFile or preserve the status quo, and I'm more inclined to the latter.
|
Hi @izbyshev! Unfortunately, this PR did not reach a successful conclusion before Python 2.7 reached EOL on January 1st. As this does not appear to be a critical security issue, there is almost no chance that it will be accepted in this brief limbo window between end-of-support and the final 2.7.18 release, so I'm going to go ahead and close it. As the attached issue indicates that it affects Python 3.x , please feel free to open a new PR against the Thanks for your contribution anyway, and I hope your next one is more fruitful! |
On Windows, if there is an outstanding operation on a synchronous
pipe handle (e.g., ReadFile()), many other operations will block until
the first one completes. This is true even for operations that
normally complete immediately (PeekNamedPipe(), SetFilePointer(),
etc.).
Python performs fstat(fd) during file object initialization to check for
attempts to open a directory, which is forbidden. If fd refers to a pipe,
msvcrt calls PeekNamedPipe() to fill in st_size member of struct stat,
potentially hanging the interpreter before execution of user code
when stdin object is initialized in the conditions described above.
fstat() on Windows can't distinguish between a file and a directory
because it relies on GetFileType(), which returns FILE_TYPE_DISK
in both cases. Therefore, skipping the no-op check avoids the hang.
https://bugs.python.org/issue34780