kernel/git - git - PowerEL Git System

Commit Graph

Author	SHA1	Message	Date
Alexandr Miloslavskiy	94f4d01932	mingw: workaround for hangs when sending STDIN Explanation ----------- The problem here is flawed `poll()` implementation. When it tries to see if pipe can be written without blocking, it eventually calls `NtQueryInformationFile()` and tests `WriteQuotaAvailable`. However, the meaning of quota was misunderstood. The value of quota is reduced when either some data was written to a pipe, or there is a pending read on the pipe. Therefore, if there is a pending read of size >= than the pipe's buffer size, poll() will think that pipe is not writable and will hang forever, usually that means deadlocking both pipe users. I have studied the problem and found that Windows pipes track two values: `QuotaUsed` and `BytesInQueue`. The code in `poll()` apparently wants to know `BytesInQueue` instead of quota. Unfortunately, `BytesInQueue` can only be requested from read end of the pipe, while `poll()` receives write end. The git's implementation of `poll()` was copied from gnulib, which also contains a flawed implementation up to today. I also had a look at implementation in cygwin, which is also broken in a subtle way. It uses this code in `pipe_data_available()`: fpli.WriteQuotaAvailable = (fpli.OutboundQuota - fpli.ReadDataAvailable) However, `ReadDataAvailable` always returns 0 for the write end of the pipe, turning the code into an obfuscated version of returning pipe's total buffer size, which I guess will in turn have `poll()` always say that pipe is writable. The commit that introduced the code doesn't say anything about this change, so it could be some debugging code that slipped in. These are the typical sizes used in git: 0x2000 - default read size in `strbuf_read()` 0x1000 - default read size in CRT, used by `strbuf_getwholeline()` 0x2000 - pipe buffer size in compat\mingw.c As a consequence, as soon as child process uses `strbuf_read()`, `poll()` in parent process will hang forever, deadlocking both processes. This results in two observable behaviors: 1) If parent process begins sending STDIN quickly (and usually that's the case), then first `poll()` will succeed and first block will go through. MAX_IO_SIZE_DEFAULT is 8MB, so if STDIN exceeds 8MB, then it will deadlock. 2) If parent process waits a little bit for any reason (including OS scheduler) and child is first to issue `strbuf_read()`, then it will deadlock immediately even on small STDINs. The problem is illustrated by `git stash push`, which will currently read the entire patch into memory and then send it to `git apply` via STDIN. If patch exceeds 8MB, git hangs on Windows. Possible solutions ------------------ 1) Somehow obtain `BytesInQueue` instead of `QuotaUsed` I did a pretty thorough search and didn't find any ways to obtain the value from write end of the pipe. 2) Also give read end of the pipe to `poll()` That can be done, but it will probably invite some dirty code, because `poll()` * can accept multiple pipes at once * can accept things that are not pipes * is expected to have a well known signature. 3) Make `poll()` always reply "writable" for write end of the pipe Afterall it seems that cygwin (accidentally?) does that for years. Also, it should be noted that `pump_io_round()` writes 8MB blocks, completely ignoring the fact that pipe's buffer size is only 8KB, which means that pipe gets clogged many times during that single write. This may invite a deadlock, if child's STDERR/STDOUT gets clogged while it's trying to deal with 8MB of STDIN. Such deadlocks could be defeated with writing less than pipe's buffer size per round, and always reading everything from STDOUT/STDERR before starting next round. Therefore, making `poll()` always reply "writable" shouldn't cause any new issues or block any future solutions. 4) Increase the size of the pipe's buffer The difference between `BytesInQueue` and `QuotaUsed` is the size of pending reads. Therefore, if buffer is bigger than size of reads, `poll()` won't hang so easily. However, I found that for example `strbuf_read()` will get more and more hungry as it reads large inputs, eventually surpassing any reasonable pipe buffer size. Chosen solution --------------- Make `poll()` always reply "writable" for write end of the pipe. Hopefully one day someone will find a way to implement it properly. Reproduction ------------ printf "%8388608s" X >large_file.txt git stash push --include-untracked -- large_file.txt I have decided not to include this as test to avoid slowing down the test suite. I don't expect the specific problem to come back, and chances are that `git stash push` will be reworked to avoid sending the entire patch via STDIN. Signed-off-by: Alexandr Miloslavskiy <alexandr.miloslavskiy@syntevo.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Johannes Schindelin	94238859b9	mingw: use Unicode functions explicitly Many Win32 API functions actually exist in two variants: one with the `A` suffix that takes ANSI parameters (`char ` or `const char `) and one with the `W` suffix that takes Unicode parameters (`wchar_t ` or `const wchar_t `). The ANSI variant assumes that the strings are encoded according to whatever is the current locale. This is not what Git wants to use on Windows: we assume that `char *` variables point to strings encoded in UTF-8. There is a pseudo UTF-8 locale on Windows, but it does not work as one might expect. In addition, if we overrode the user's locale, that would modify the behavior of programs spawned by Git (such as editors, difftools, etc), therefore we cannot use that pseudo locale. Further, it is actually highly encouraged to use the Unicode versions instead of the ANSI versions, so let's do precisely that. Note: when calling the Win32 API functions _without_ any suffix, it depends whether the `UNICODE` constant is defined before the relevant headers are #include'd. Without that constant, the ANSI variants are used. Let's be explicit and avoid that ambiguity. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	8d4679fe09	poll (mingw): allow compiling with GCC 8 and DEVELOPER=1 The return type of the `GetProcAddress()` function is `FARPROC` which evaluates to `long long int ()()`, i.e. it cannot be cast to the correct function signature by GCC 8. To work around that, we first cast to `void ` and go on with our merry lives. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Steve Hoelzer	e8dfcace31	poll: use GetTickCount64() to avoid wrap-around issues The value of timeout starts as an int value, and for this reason it cannot overflow unsigned long long aka ULONGLONG. The unsigned version of this initial value is available in orig_timeout. The difference (orig_timeout - elapsed) cannot wrap around because it is protected by a conditional (as can be seen in the patch text). Hence, the ULONGLONG difference can only have values that are smaller than the initial timeout value and truncation to int cannot overflow. Signed-off-by: Steve Hoelzer <shoelzer@gmail.com> [j6t: improved both implementation and log message] Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	d7e357fb9c	compat/poll: prepare for targeting Windows Vista Windows Vista (and later) actually have a working poll(), but we still cannot use it because it only works on sockets. So let's detect when we are targeting Windows Vista and undefine those constants, and define `pollfd` so that we can declare our own pollfd struct. We also need to make sure that we override those constants after `winsock2.h` has been `#include`d (otherwise we would not really override those constants). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Todd Zullinger	484257925f	Replace Free Software Foundation address in license notices The mailing address for the FSF has changed over the years. Rather than updating the address across all files, refer readers to gnu.org, as the GNU GPL documentation now suggests for license notices. The mailing address is retained in the full license files (COPYING and LGPL-2.1). The old address is still present in t/diff-lib/COPYING. This is intentional, as the file is used in tests and the contents are not expected to change. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Randall S. Becker	61b2a1acaa	poll.c: always set revents, even if to zero Match what is done to pfd[i].revents when compute_revents() returns 0 to the upstream gnulib's commit d42461c3 ("poll: fixes for large fds", 2015-02-20). The revents field is set to 0, without incrementing the value rc to be returned from the function. The original code left the field to whatever random value the field was initialized to. This fixes occasional hangs in git-upload-pack on HPE NonStop. Signed-off-by: Randall S. Becker <randall.becker@nexbridge.ca> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Johannes Schindelin	7c00bc39eb	mingw: avoid warnings when casting HANDLEs to int HANDLE is defined internally as a void *, but in many cases it is actually guaranteed to be a 32-bit integer. In these cases, GCC should not warn about a cast of a pointer to an integer of a different type because we know exactly what we are doing. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Edward Thomson	ef8b53e78c	poll: honor the timeout on Win32 Ensure that when passing a pipe, the gnulib poll replacement will not return 0 before the timeout has passed. Not obeying the timeout (and merely returning 0) causes pathological behavior when preparing a packfile for a repository and taking a long time to do so. If poll were to return 0 immediately, this would cause keep-alives to get sent as quickly as possible until the packfile was created. Such deviance from the standard would cause megabytes (or more) of keep-alive packets to be sent. GetTickCount is used as it is efficient, stable and monotonically increasing. (Neither GetSystemTime nor QueryPerformanceCounter have all three of these properties.) Signed-off-by: Edward Thomson <ethomson@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Theodore Leblond	76e7c8a7ed	compat/poll: sleep 1 millisecond to avoid busy wait SwitchToThread() only gives away the rest of the current time slice to another thread in the current process. So if the thread that feeds the file decscriptor we're polling is not in the current process, we get busy-waiting. I played around with this quite a bit. After trying some more complex schemes, I found that what worked best is to just sleep 1 millisecond between iterations. Though it's a very short time, it still completely eliminates the busy wait condition, without hurting perf. There code uses SleepEx(1, TRUE) to sleep. See this page for a good discussion of why that is better than calling SwitchToThread, which is what was used previously: http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1 Note that calling SleepEx(0, TRUE) does not solve the busy wait. The most striking case was when testing on a UNC share with a large repo, on a single CPU machine. Without the fix, it took 4 minutes 15 seconds, and with the fix it took just 1:08! I think it's because git-upload-pack's busy wait was eating the CPU away from the git process that's doing the real work. With multi-proc, the timing is not much different, but tons of CPU time is still wasted, which can be a killer on a server that needs to do bunch of other things. I also tested the very fast local case, and didn't see any measurable difference. On a big repo with 4500 files, the upload-pack took about 2 seconds with and without the fix. [jc: this was first accepted in msysgit tree in May 2012 via a pull request and Paolo Bonzini has also accepted the same fix to Gnulib around the same time; see $gmane/247518 for a bit more detail] Signed-off-by: Stepan Kasal <kasal@ucw.cz> Acked-by: Johannes Sixt <j6t@kdbg.org> Acked-by: Erik Faye-Lund <kusmabite@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	11 years ago
Sebastian Schuberth	8453c1259a	Windows: do not redefine _WIN32_WINNT With MinGW runtime version 4.0 this interferes with the previous definition from sdkddkver.h. Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	11 years ago
Ramsay Jones	1c31596a4b	compat/poll/poll.c: Fix a sparse warning Sparse issues an 'Using plain integer as NULL pointer' warning when passing the constant '0' as the second parameter in the call to the WSAEventSelect() function. The function parameter has a pointer type (WSAEVENT, aka HANDLE, aka void *) so that, in order to suppress the warning, we simply pass NULL for that parameter in the function call expression. Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>	12 years ago
Joachim Schmitz	a6772946a5	make poll() work on platforms that can't recv() on a non-socket This way it just got added to gnulib too the other day. Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	12 years ago
Joachim Schmitz	32fde6575e	poll() exits too early with EFAULT if 1st arg is NULL If poll() is used as a milli-second sleep, like in help.c, by passing a NULL in the 1st and a 0 in the 2nd arg, it exits with EFAULT. As per Paolo Bonzini, the original author, this is a bug and to be fixed Like in this commit, which is not to exit if the 2nd arg is 0. It got fixed In gnulib in the same manner the other day. Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	12 years ago
Joachim Schmitz	98c573a902	fix some win32 specific dependencies in poll.c In order for non-win32 platforms to be able to use poll.c, #ifdef the inclusion of two header files properly Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	12 years ago
Joachim Schmitz	6d45eb1720	make poll available for other platforms lacking it move poll.[ch] out of compat/win32/ into compat/poll/ and adjust Makefile with the changed paths. Adding comments to Makefile about how/when to enable it and add logic for this Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	12 years ago

17 Commits (7da656f1e083b065400c4ab92405c6332481eb7a)