kernel/git - git - PowerEL Git System

Commit Graph

Author	SHA1	Message	Date
David Turner	37ee680d9b	http.postbuffer: allow full range of ssize_t values Unfortunately, in order to push some large repos where a server does not support chunked encoding, the http postbuffer must sometimes exceed two gigabytes. On a 64-bit system, this is OK: we just malloc a larger buffer. This means that we need to use CURLOPT_POSTFIELDSIZE_LARGE to set the buffer size. Signed-off-by: David Turner <dturner@twosigma.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Sergey Ryazanov	ae51d91105	http: fix the silent ignoring of proxy misconfiguraion Earlier, the whole http.proxy option string was passed to curl without any preprocessing so curl could complain about the invalid proxy configuration. After the commit `372370f167` ("http: use credential API to handle proxy authentication", 2016-01-26), if the user specified an invalid HTTP proxy option in the configuration, then the option parsing silently fails and NULL will be passed to curl as a proxy. This forces curl to fall back to detecting the proxy configuration from the environment, causing the http.proxy option ignoring. Fix this issue by checking the proxy option parsing result. If parsing failed then print an error message and die. Such behaviour allows the user to quickly figure the proxy misconfiguration and correct it. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Sergey Ryazanov <ryazanov.s.a@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Sergey Ryazanov	57415089bd	http: honor empty http.proxy option to bypass proxy Curl distinguishes between an empty proxy address and a NULL proxy address. In the first case it completely disables proxy usage, but if the proxy address option is NULL then curl attempts to determine the proxy address from the http_proxy environment variable. According to the documentation, if the http.proxy option is set to an empty string, git should bypass proxy and connect to the server directly: export http_proxy=http://network-proxy/ cd ~/foobar-project git config remote.origin.proxy "" git fetch Previously, proxy host was configured by one line: curl_easy_setopt(result, CURLOPT_PROXY, curl_http_proxy); Commit `372370f167` ("http: use credential API to handle proxy authentication", 2016-01-26) parses the proxy option, then extracts the proxy host address and updates the curl configuration, making the previous call a noop: credential_from_url(&proxy_auth, curl_http_proxy); curl_easy_setopt(result, CURLOPT_PROXY, proxy_auth.host); But if the proxy option is empty then the proxy host field becomes NULL. This forces curl to fall back to detecting the proxy configuration from the environment, causing the http.proxy option to not work anymore. Fix this issue by explicitly handling http.proxy being set the empty string. This also makes the code a bit more clear and should help us avoid such regressions in the future. Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Sergey Ryazanov <ryazanov.s.a@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	1a168e5c86	convert unchecked snprintf into xsnprintf These calls to snprintf should always succeed, because their input is small and fixed. Let's use xsnprintf to make sure this is the case (and to make auditing for actual truncation easier). These could be candidates for turning into heap buffers, but they fall into a few broad categories that make it not worth doing: - formatting single numbers is simple enough that we can see the result should fit - the size of a sha1 is likewise well-known, and I didn't want to cause unnecessary conflicts with the ongoing process to convert these constants to GIT_MAX_HEXSZ - the interface for curl_errorstr is dictated by curl Signed-off-by: Jeff King <peff@peff.net>	8 years ago
Jonathan Tan	8e27391a5f	http: attempt updating base URL only if no error http.c supports HTTP redirects of the form http://foo/info/refs?service=git-upload-pack -> http://anything -> http://bar/info/refs?service=git-upload-pack (that is to say, as long as the Git part of the path and the query string is preserved in the final redirect destination, the intermediate steps can have any URL). However, if one of the intermediate steps results in an HTTP exception, a confusing "unable to update url base from redirection" message is printed instead of a Curl error message with the HTTP exception code. This was introduced by 2 commits. Commit `c93c92f` ("http: update base URLs when we see redirects", 2013-09-28) introduced a best-effort optimization that required checking if only the "base" part of the URL differed between the initial request and the final redirect destination, but it performed the check before any HTTP status checking was done. If something went wrong, the normal code path was still followed, so this did not cause any confusing error messages until commit `6628eb4` ("http: always update the base URL for redirects", 2016-12-06), which taught http to die if the non-"base" part of the URL differed. Therefore, teach http to check the HTTP status before attempting to check if only the "base" part of the URL differed. This commit teaches http_request_reauth to return early without updating options->base_url upon an error; the only invoker of this function that passes a non-NULL "options" is remote-curl.c (through "http_get_strbuf"), which only uses options->base_url for an informational message in the situations that this commit cares about (that is, when the return value is not HTTP_OK). The included test checks that the redirect scheme at the beginning of this commit message works, and that returning a 502 in the middle of the redirect scheme produces the correct result. Note that this is different from the test in commit `6628eb4` ("http: always update the base URL for redirects", 2016-12-06) in that this commit tests that a Git-shaped URL (http://.../info/refs?service=git-upload-pack) works, whereas commit `6628eb4` tests that a non-Git-shaped URL (http://.../info/refs/foo?service=git-upload-pack) does not work (even though Git is processing that URL) and is an error that is fatal, not silently swallowed. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	40a18fc77c	http: add an "auto" mode for http.emptyauth This variable needs to be specified to make some types of non-basic authentication work, but ideally this would just work out of the box for everyone. However, simply setting it to "1" by default introduces an extra round-trip for cases where it _isn't_ useful. We end up sending a bogus empty credential that the server rejects. Instead, let's introduce an automatic mode, that works like this: 1. We won't try to send the bogus credential on the first request. We'll wait to get an HTTP 401, as usual. 2. After seeing an HTTP 401, the empty-auth hack will kick in only when we know there is an auth method available that might make use of it (i.e., something besides "Basic" or "Digest"). That should make it work out of the box, without incurring any extra round-trips for people hitting Basic-only servers. This _does_ incur an extra round-trip if you really want to use "Basic" but your server advertises other methods (the emptyauth hack will kick in but fail, and then Git will actually ask for a password). The auto mode may incur an extra round-trip over setting http.emptyauth=true, because part of the emptyauth hack is to feed this blank password to curl even before we've made a single request. Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	840398feb8	http: restrict auth methods to what the server advertises By default, we tell curl to use CURLAUTH_ANY, which does not limit its set of auth methods. However, this results in an extra round-trip to the server when authentication is required. After we've fed the credential to curl, it wants to probe the server to find its list of available methods before sending an Authorization header. We can shortcut this by limiting our http_auth_methods by what the server told us it supports. In some cases (such as when the server only supports Basic), that lets curl skip the extra probe request. The end result should look the same to the user, but you can use GIT_TRACE_CURL to verify the sequence of requests: GIT_TRACE_CURL=1 \ git ls-remote https://example.com/repo.git \ 2>&1 >/dev/null \| egrep '(Send\|Recv) header: (GET\|HTTP\|Auth)' Before this patch, hitting a Basic-only server like github.com results in: Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1 Recv header: HTTP/1.1 401 Authorization Required Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1 Recv header: HTTP/1.1 401 Authorization Required Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1 Send header: Authorization: Basic <redacted> Recv header: HTTP/1.1 200 OK And after: Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1 Recv header: HTTP/1.1 401 Authorization Required Send header: GET /repo.git/info/refs?service=git-upload-pack HTTP/1.1 Send header: Authorization: Basic <redacted> Recv header: HTTP/1.1 200 OK The possible downsides are: - This only helps for a Basic-only server; for a server with multiple auth options, curl may still send a probe request to see which ones are available (IOW, there's no way to say "don't probe, I already know what the server will say"). - The http_auth_methods variable is global, so this will apply to all further requests. That's acceptable for Git's usage of curl, though, which also treats the credentials as global. I.e., in any given program invocation we hit only one conceptual server (we may be redirected at the outset, but in that case that's whose auth_avail field we'd see). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Brandon Williams	a768a02265	transport: add from_user parameter to is_transport_allowed Add a from_user parameter to is_transport_allowed() to allow http to be able to distinguish between protocol restrictions for redirects versus initial requests. CURLOPT_REDIR_PROTOCOLS can now be set differently from CURLOPT_PROTOCOLS to disallow use of protocols with the "user" policy in redirects. This change allows callers to query if a transport protocol is allowed, given that the caller knows that the protocol is coming from the user (1) or not from the user (0) such as redirects in libcurl. If unknown a -1 should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` to determine if the protocol came from the user. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Brandon Williams	aeae4db174	http: create function to get curl allowed protocols Move the creation of an allowed protocols whitelist to a helper function. This will be useful when we need to compute the set of allowed protocols differently for normal and redirect cases. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Brandon Williams	f962ddf6ed	http: always warn if libcurl version is too old Always warn if libcurl version is too old because: 1. Even without a protocol whitelist, newer versions of curl have all non-standard protocols disabled by default. 2. A future patch will introduce default "known-good" and "known-bad" protocols which are allowed/disallowed by 'is_transport_allowed' which older version of libcurl can't respect. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	3680f16f9d	http-walker: complain about non-404 loose object errors Since commit `17966c0a6` (http: avoid disconnecting on 404s for loose objects, 2016-07-11), we turn off curl's FAILONERROR option and instead manually deal with failing HTTP codes. However, the logic to do so only recognizes HTTP 404 as a failure. This is probably the most common result, but if we were to get another code, the curl result remains CURLE_OK, and we treat it as success. We still end up detecting the failure when we try to zlib-inflate the object (which will fail), but instead of reporting the HTTP error, we just claim that the object is corrupt. Instead, let's catch anything in the 300's or above as an error (300's are redirects which are not an error at the HTTP level, but are an indication that we've explicitly disabled redirects, so we should treat them as such; we certainly don't have the resulting object content). Note that we also fill in req->errorstr, which we didn't do before. Without FAILONERROR, curl will not have filled this in, and it will remain a blank string. This never mattered for the 404 case, because in the logic below we hit the "missing_target()" branch and print nothing. But for other errors, we'd want to say _something_, if only to fill in the blank slot in the error message. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	cb4d2d35c4	http: treat http-alternates like redirects The previous commit made HTTP redirects more obvious and tightened up the default behavior. However, there's another way for a server to ask a git client to fetch arbitrary content: by having an http-alternates file (or a regular alternates file, which is used as a backup). Similar to the HTTP redirect case, a malicious server can claim to have refs pointing at object X, return a 404 when the client asks for X, but point to some other URL via http-alternates, which the client will transparently fetch. The end result is that it looks from the user's perspective like the objects came from the malicious server, as the other URL is not mentioned at all. Worse, because we feed the new URL to curl ourselves, the usual protocol restrictions do not kick in (neither curl's default of disallowing file://, nor the protocol whitelisting in `f4113cac0` (http: limit redirection to protocol-whitelist, 2015-09-22). Let's apply the same rules here as we do for HTTP redirects. Namely: - unless http.followRedirects is set to "always", we will not follow remote redirects from http-alternates (or alternates) at all - set CURLOPT_PROTOCOLS alongside CURLOPT_REDIR_PROTOCOLS restrict ourselves to a known-safe set and respect any user-provided whitelist. - mention alternate object stores on stderr so that the user is aware another source of objects may be involved The first item may prove to be too restrictive. The most common use of alternates is to point to another path on the same server. While it's possible for a single-server redirect to be an attack, it takes a fairly obscure setup (victim and evil repository on the same host, host speaks dumb http, and evil repository has access to edit its own http-alternates file). So we could make the checks more specific, and only cover cross-server redirects. But that means parsing the URLs ourselves, rather than letting curl handle them. This patch goes for the simpler approach. Given that they are only used with dumb http, http-alternates are probably pretty rare. And there's an escape hatch: the user can allow redirects on a specific server by setting http.<url>.followRedirects to "always". Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	50d3413740	http: make redirects more obvious We instruct curl to always follow HTTP redirects. This is convenient, but it creates opportunities for malicious servers to create confusing situations. For instance, imagine Alice is a git user with access to a private repository on Bob's server. Mallory runs her own server and wants to access objects from Bob's repository. Mallory may try a few tricks that involve asking Alice to clone from her, build on top, and then push the result: 1. Mallory may simply redirect all fetch requests to Bob's server. Git will transparently follow those redirects and fetch Bob's history, which Alice may believe she got from Mallory. The subsequent push seems like it is just feeding Mallory back her own objects, but is actually leaking Bob's objects. There is nothing in git's output to indicate that Bob's repository was involved at all. The downside (for Mallory) of this attack is that Alice will have received Bob's entire repository, and is likely to notice that when building on top of it. 2. If Mallory happens to know the sha1 of some object X in Bob's repository, she can instead build her own history that references that object. She then runs a dumb http server, and Alice's client will fetch each object individually. When it asks for X, Mallory redirects her to Bob's server. The end result is that Alice obtains objects from Bob, but they may be buried deep in history. Alice is less likely to notice. Both of these attacks are fairly hard to pull off. There's a social component in getting Mallory to convince Alice to work with her. Alice may be prompted for credentials in accessing Bob's repository (but not always, if she is using a credential helper that caches). Attack (1) requires a certain amount of obliviousness on Alice's part while making a new commit. Attack (2) requires that Mallory knows a sha1 in Bob's repository, that Bob's server supports dumb http, and that the object in question is loose on Bob's server. But we can probably make things a bit more obvious without any loss of functionality. This patch does two things to that end. First, when we encounter a whole-repo redirect during the initial ref discovery, we now inform the user on stderr, making attack (1) much more obvious. Second, the decision to follow redirects is now configurable. The truly paranoid can set the new http.followRedirects to false to avoid any redirection entirely. But for a more practical default, we will disallow redirects only after the initial ref discovery. This is enough to thwart attacks similar to (2), while still allowing the common use of redirects at the repository level. Since `c93c92f30` (http: update base URLs when we see redirects, 2013-09-28) we re-root all further requests from the redirect destination, which should generally mean that no further redirection is necessary. As an escape hatch, in case there really is a server that needs to redirect individual requests, the user can set http.followRedirects to "true" (and this can be done on a per-server basis via http.*.followRedirects config). Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	6628eb41db	http: always update the base URL for redirects If a malicious server redirects the initial ref advertisement, it may be able to leak sha1s from other, unrelated servers that the client has access to. For example, imagine that Alice is a git user, she has access to a private repository on a server hosted by Bob, and Mallory runs a malicious server and wants to find out about Bob's private repository. Mallory asks Alice to clone an unrelated repository from her over HTTP. When Alice's client contacts Mallory's server for the initial ref advertisement, the server issues an HTTP redirect for Bob's server. Alice contacts Bob's server and gets the ref advertisement for the private repository. If there is anything to fetch, she then follows up by asking the server for one or more sha1 objects. But who is the server? If it is still Mallory's server, then Alice will leak the existence of those sha1s to her. Since commit `c93c92f30` (http: update base URLs when we see redirects, 2013-09-28), the client usually rewrites the base URL such that all further requests will go to Bob's server. But this is done by textually matching the URL. If we were originally looking for "http://mallory/repo.git/info/refs", and we got pointed at "http://bob/other.git/info/refs", then we know that the right root is "http://bob/other.git". If the redirect appears to change more than just the root, we punt and continue to use the original server. E.g., imagine the redirect adds a URL component that Bob's server will ignore, like "http://bob/other.git/info/refs?dummy=1". We can solve this by aborting in this case rather than silently continuing to use Mallory's server. In addition to protecting from sha1 leakage, it's arguably safer and more sane to refuse a confusing redirect like that in general. For example, part of the motivation in `c93c92f30` is avoiding accidentally sending credentials over clear http, just to get a response that says "try again over https". So even in a non-malicious case, we'd prefer to err on the side of caution. The downside is that it's possible this will break a legitimate but complicated server-side redirection scheme. The setup given in the newly added test does work, but it's convoluted enough that we don't need to care about it. A more plausible case would be a server which redirects a request for "info/refs?service=git-upload-pack" to just "info/refs" (because it does not do smart HTTP, and for some reason really dislikes query parameters). Right now we would transparently downgrade to dumb-http, but with this patch, we'd complain (and the user would have to set GIT_SMART_HTTP=0 to fetch). Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	986d7f4d37	http: simplify update_url_from_redirect This function looks for a common tail between what we asked for and where we were redirected to, but it open-codes the comparison. We can avoid some confusing subtractions by using strip_suffix_mem(). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
David Turner	5275c3081c	http: http.emptyauth should allow empty (not just NULL) usernames When using Kerberos authentication with newer versions of libcurl, CURLOPT_USERPWD must be set to a value, even if it is an empty value. The value is never sent to the server. Previous versions of libcurl did not require this variable to be set. One way that some users express the empty username/password is http://:@gitserver.example.com, which http.emptyauth was designed to support. Another, equivalent, URL is http://@gitserver.example.com. The latter leads to a username of zero-length, rather than a NULL username, but CURLOPT_USERPWD still needs to be set (if http.emptyauth is set). Do so. Signed-off-by: David Turner <dturner@twosigma.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Petr Stodulka	26a7b23429	http: control GSSAPI credential delegation Delegation of credentials is disabled by default in libcurl since version 7.21.7 due to security vulnerability CVE-2011-2192. Which makes troubles with GSS/kerberos authentication when delegation of credentials is required. This can be changed with option CURLOPT_GSSAPI_DELEGATION in libcurl with set expected parameter since libcurl version 7.22.0. This patch provides new configuration variable http.delegation which corresponds to curl parameter "--delegation" (see man 1 curl). The following values are supported: * none (default). * policy * always Signed-off-by: Petr Stodulka <pstodulk@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Eric Wong	2abc848d54	http: always remove curl easy from curlm session on release We must call curl_multi_remove_handle when releasing the slot to prevent subsequent calls to curl_multi_add_handle from failing with CURLM_ADDED_ALREADY (in curl 7.32.1+; older versions returned CURLM_BAD_EASY_HANDLE) Signed-off-by: Eric Wong <e@80x24.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Eric Wong	d8b6b84df0	http: consolidate #ifdefs for curl_multi_remove_handle I find #ifdefs makes code difficult-to-follow. An early version of this patch had error checking for curl_multi_remove_handle calls, but caused some tests (e.g. t5541) to fail under curl 7.26.0 on old Debian wheezy. Signed-off-by: Eric Wong <e@80x24.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Eric Wong	9f1b58842a	http: warn on curl_multi_add_handle failures This will be useful for tracking down curl usage errors. Signed-off-by: Eric Wong <e@80x24.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Jeff King	d63ed6ef24	remote-curl: handle URLs without protocol Generally remote-curl would never see a URL that did not have "proto:" at the beginning, as that is what tells git to run the "git-remote-proto" helper (and git-remote-http, etc, are aliases for git-remote-curl). However, the special syntax "proto::something" will run git-remote-proto with only "something" as the URL. So a malformed URL like: http::/example.com/repo.git will feed the URL "/example.com/repo.git" to git-remote-http. The resulting URL has no protocol, but the code added by `372370f` (http: use credential API to handle proxy authentication, 2016-01-26) does not handle this case and segfaults. For the purposes of this code, we don't really care what the exact protocol; only whether or not it is https. So let's just assume that a missing protocol is not, and curl will handle the real error (which is that the URL is nonsense). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
René Scharfe	bc57b9c0cc	use strbuf_addstr() instead of strbuf_addf() with "%s" Call strbuf_addstr() for adding a simple string to a strbuf instead of using the heavier strbuf_addf(). This is shorter and documents the intent more clearly. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Eric Wong	17966c0a63	http: avoid disconnecting on 404s for loose objects 404s are common when fetching loose objects on static HTTP servers, and reestablishing a connection for every single 404 adds additional latency. Signed-off-by: Eric Wong <e@80x24.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Elia Pinto	74c682d3c6	http.c: implement the GIT_TRACE_CURL environment variable Implement the GIT_TRACE_CURL environment variable to allow a greater degree of detail of GIT_CURL_VERBOSE, in particular the complete transport header and all the data payload exchanged. It might be useful if a particular situation could require a more thorough debugging analysis. Document the new GIT_TRACE_CURL environment variable. Helped-by: Torsten Bögershausen <tboegi@web.de> Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Helped-by: Junio C Hamano <gitster@pobox.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Nguyễn Thái Ngọc Duy	d2e255eefa	http.c: use error_errno() and warning_errno() Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Brian Norris	e5a39ad8e6	http: expand http.cookieFile as a path This should handle .gitconfig files that specify things like: [http] cookieFile = "~/.gitcookies" Signed-off-by: Brian Norris <computersforpeace@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Johannes Schindelin	8cb01e2fd3	http: support sending custom HTTP headers We introduce a way to send custom HTTP headers with all requests. This allows us, for example, to send an extra token from build agents for temporary access to private repositories. (This is the use case that triggered this patch.) This feature can be used like this: git -c http.extraheader='Secret: sssh!' fetch $URL $REF Note that `curl_easy_setopt(..., CURLOPT_HTTPHEADER, ...)` takes only a single list, overriding any previous call. This means we have to collect _all_ of the headers we want to use into a single list, and feed it to cURL in one shot. Since we already unconditionally set a "pragma" header when initializing the curl handles, we can add our new headers to that list. For callers which override the default header list (like probe_rpc), we provide `http_copy_default_headers()` so they can do the same trick. Big thanks to Jeff King and Junio Hamano for their outstanding help and patient reviews. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Junio C Hamano	87f8a0b279	http: differentiate socks5:// and socks5h:// Felix Ruess <felix.ruess@gmail.com> noticed that with configuration $ git config --global 'http.proxy=socks5h://127.0.0.1:1080' connections to remote sites time out, waiting for DNS resolution. The logic to detect various flavours of SOCKS proxy and ask the libcurl layer to use appropriate one understands the proxy string that begin with socks5, socks4a, etc., but does not know socks5h, and we end up using CURLPROXY_SOCKS5. The correct one to use is CURLPROXY_SOCKS5_HOSTNAME. https://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html says ..., socks5h:// (the last one to enable socks5 and asking the proxy to do the resolving, also known as CURLPROXY_SOCKS5_HOSTNAME type). which is consistent with the way the breakage was reported. Tested-by: Felix Ruess <felix.ruess@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Jiang Xin	d445fda44d	http: honor no_http env variable to bypass proxy Curl and its families honor several proxy related environment variables: * http_proxy and https_proxy define proxy for http/https connections. * no_proxy (a comma separated hosts) defines hosts bypass the proxy. This command will bypass the bad-proxy and connect to the host directly: no_proxy=* https_proxy=http://bad-proxy/ \ curl -sk https://google.com/ Before commit `372370f` (http: use credential API to handle proxy auth...), Environment variable "no_proxy" will take effect if the config variable "http.proxy" is not set. So the following comamnd won't fail if not behind a firewall. no_proxy=* https_proxy=http://bad-proxy/ \ git ls-remote https://github.com/git/git But commit `372370f` not only read git config variable "http.proxy", but also read "http_proxy" and "https_proxy" environment variables, and set the curl option using: curl_easy_setopt(result, CURLOPT_PROXY, proxy_auth.host); This caused "no_proxy" environment variable not working any more. Set extra curl option "CURLOPT_NOPROXY" will fix this issue. Signed-off-by: Jiang Xin <xin.jiang@huawei.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Christoph Egger	aeff8a6121	http: implement public key pinning Add the http.pinnedpubkey configuration option for public key pinning. It allows any string supported by libcurl -- base64(sha256(pubkey)) or filename of the full public key. If cURL does not support pinning (is too old) output a warning to the user. Signed-off-by: Christoph Egger <christoph@christoph-egger.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
brian m. carlson	121061f67f	http: add option to try authentication without username Performing GSS-Negotiate authentication using Kerberos does not require specifying a username or password, since that information is already included in the ticket itself. However, libcurl refuses to perform authentication if it has not been provided with a username and password. Add an option, http.emptyAuth, that provides libcurl with an empty username and password to make it attempt authentication anyway. Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Eric Wong	c915f11eb4	connect & http: support -4 and -6 switches for remote operations Sometimes it is necessary to force IPv4-only or IPv6-only operation on networks where name lookups may return a non-routable address and stall remote operations. The ssh(1) command has an equivalent switches which we may pass when we run them. There may be old ssh(1) implementations out there which do not support these switches; they should report the appropriate error in that case. rsync support is untouched for now since it is deprecated and scheduled to be removed. Signed-off-by: Eric Wong <normalperson@yhbt.net> Reviewed-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Knut Franke	372370f167	http: use credential API to handle proxy authentication Currently, the only way to pass proxy credentials to curl is by including them in the proxy URL. Usually, this means they will end up on disk unencrypted, one way or another (by inclusion in ~/.gitconfig, shell profile or history). Since proxy authentication often uses a domain user, credentials can be security sensitive; therefore, a safer way of passing credentials is desirable. If the configured proxy contains a username but not a password, query the credential API for one. Also, make sure we approve/reject proxy credentials properly. For consistency reasons, add parsing of http_proxy/https_proxy/all_proxy environment variables, which would otherwise be evaluated as a fallback by curl. Without this, we would have different semantics for git configuration and environment variables. Helped-by: Junio C Hamano <gitster@pobox.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Knut Franke <k.franke@science-computing.de> Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Knut Franke	ef976395e2	http: allow selection of proxy authentication method CURLAUTH_ANY does not work with proxies which answer unauthenticated requests with a 307 redirect to an error page instead of a 407 listing supported authentication methods. Therefore, allow the authentication method to be set using the environment variable GIT_HTTP_PROXY_AUTHMETHOD or configuration variables http.proxyAuthmethod and remote.<name>.proxyAuthmethod (in analogy to http.proxy and remote.<name>.proxy). The following values are supported: * anyauth (default) * basic * digest * negotiate * ntlm Signed-off-by: Knut Franke <k.franke@science-computing.de> Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Helped-by: Junio C Hamano <gitster@pobox.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Elia Pinto <gitter.spiros@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Charles Bailey	bf9acba2c1	http: treat config options sslCAPath and sslCAInfo as paths This enables ~ and ~user expansion for these config options. Signed-off-by: Charles Bailey <cbailey32@bloomberg.net> Signed-off-by: Jeff King <peff@peff.net>	9 years ago
brian m. carlson	f4e54d02b8	Convert struct ref to use object_id. Use struct object_id in three fields in struct ref and convert all the necessary places that use it. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Jeff King <peff@peff.net>	9 years ago
Pat Thoyts	6d7afe07f2	remote-http(s): support SOCKS proxies With this patch we properly support SOCKS proxies, configured e.g. like this: git config http.proxy socks5://192.168.67.1:32767 Without this patch, Git mistakenly tries to use SOCKS proxies as if they were HTTP proxies, resulting in a error message like: fatal: unable to access 'http://.../': Proxy CONNECT aborted This patch was required to work behind a faulty AP and scraped from http://stackoverflow.com/questions/15227130/#15228479 and guarded with an appropriate cURL version check by Johannes Schindelin. Signed-off-by: Pat Thoyts <patthoyts@users.sourceforge.net> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Ramsay Jones	838ecf0b0f	http: fix some printf format warnings Commit `f8117f55` ("http: use off_t to store partial file size", 02-11-2015) changed the type of some variables from long to off_t. Unfortunately, the off_t type is not portable and can be represented by several different actual types (even multiple types on the same platform). This makes it difficult to print an off_t variable in a platform independent way. As a result, this commit causes gcc to issue some printf format warnings on a couple of different platforms. In order to suppress the warnings, change the format specifier to use the PRIuMAX macro and cast the off_t argument to uintmax_t. (See also the http_opt_request_remainder() function, which uses the same solution). Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Jeff King <peff@peff.net>	9 years ago
Jeff King	f8117f550b	http: use off_t to store partial file size When we try to resume transfer of a partially-downloaded object or pack, we fopen() the existing file for append, then use ftell() to get the current position. We use a "long", which can hold only 2GB on a 32-bit system, even though packfiles may be larger than that. Let's switch to using off_t, which should hold any file size our system is capable of storing. We need to use ftello() to get the off_t. This is in POSIX and hopefully available everywhere; if not, we should be able to wrap it by falling back to ftell(), which would presumably return "-1" on such a large file (and we would simply skip resuming in that case). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
David Turner	835c4d3689	http.c: use CURLOPT_RANGE for range requests A HTTP server is permitted to return a non-range response to a HTTP range request (and Apache httpd in fact does this in some cases). While libcurl knows how to correctly handle this (by skipping bytes before and after the requested range), it only turns on this handling if it is aware that a range request is being made. By manually setting the range header instead of using CURLOPT_RANGE, we were hiding the fact that this was a range request from libcurl. This could cause corruption. Signed-off-by: David Turner <dturner@twopensource.com> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago
Blake Burkhart	b258116462	http: limit redirection depth By default, libcurl will follow circular http redirects forever. Let's put a cap on this so that somebody who can trigger an automated fetch of an arbitrary repository (e.g., for CI) cannot convince git to loop infinitely. The value chosen is 20, which is the same default that Firefox uses. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Blake Burkhart	f4113cac0c	http: limit redirection to protocol-whitelist Previously, libcurl would follow redirection to any protocol it was compiled for support with. This is desirable to allow redirection from HTTP to HTTPS. However, it would even successfully allow redirection from HTTP to SFTP, a protocol that git does not otherwise support at all. Furthermore git's new protocol-whitelisting could be bypassed by following a redirect within the remote helper, as it was only enforced at transport selection time. This patch limits redirects within libcurl to HTTP, HTTPS, FTP and FTPS. If there is a protocol-whitelist present, this list is limited to those also allowed by the whitelist. As redirection happens from within libcurl, it is impossible for an HTTP redirect to a protocol implemented within another remote helper. When the curl version git was compiled with is too old to support restrictions on protocol redirection, we warn the user if GIT_ALLOW_PROTOCOL restrictions were requested. This is a little inaccurate, as even without that variable in the environment, we would still restrict SFTP, etc, and we do not warn in that case. But anything else means we would literally warn every time git accesses an http remote. This commit includes a test, but it is not as robust as we would hope. It redirects an http request to ftp, and checks that curl complained about the protocol, which means that we are relying on curl's specific error message to know what happened. Ideally we would redirect to a working ftp server and confirm that we can clone without protocol restrictions, and not with them. But we do not have a portable way of providing an ftp server, nor any other protocol that curl supports (https is the closest, but we would have to deal with certificates). [jk: added test and version warning] Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Jeff King	5096d4909f	convert trivial sprintf / strcpy calls to xsnprintf We sometimes sprintf into fixed-size buffers when we know that the buffer is large enough to fit the input (either because it's a constant, or because it's numeric input that is bounded in size). Likewise with strcpy of constant strings. However, these sites make it hard to audit sprintf and strcpy calls for buffer overflows, as a reader has to cross-reference the size of the array with the input. Let's use xsnprintf instead, which communicates to a reader that we don't expect this to overflow (and catches the mistake in case we do). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Jeff King	9ae97018fb	use strip_suffix and xstrfmt to replace suffix When we want to convert "foo.pack" to "foo.idx", we do it by duplicating the original string and then munging the bytes in place. Let's use strip_suffix and xstrfmt instead, which has several advantages: 1. It's more clear what the intent is. 2. It does not implicitly rely on the fact that strlen(".idx") <= strlen(".pack") to avoid an overflow. 3. We communicate the assumption that the input file ends with ".pack" (and get a run-time check that this is so). 4. We drop calls to strcpy, which makes auditing the code base easier. Likewise, we can do this to convert ".pack" to ".bitmap", avoiding some manual memory computation. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Elia Pinto	01861cb7a2	http: add support for specifying the SSL version Teach git about a new option, "http.sslVersion", which permits one to specify the SSL version to use when negotiating SSL connections. The setting can be overridden by the GIT_SSL_VERSION environment variable. Signed-off-by: Elia Pinto <gitter.spiros@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Junio C Hamano	cb5add5868	sha1_file.c: rename move_temp_to_file() to finalize_object_file() Since `5a688fe4` ("core.sharedrepository = 0mode" should set, not loosen, 2009-03-25), we kept reminding ourselves: NEEDSWORK: this should be renamed to finalize_temp_file() as "moving" is only a part of what it does, when no patch between master to pu changes the call sites of this function. without doing anything about it. Let's do so. The purpose of this function was not to move but to finalize. The detail of the primarily implementation of finalizing was to link the temporary file to its final name and then to unlink, which wasn't even "moving". The alternative implementation did "move" by calling rename(2), which is a fun tangent. Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Enrique Tobis	5841520b03	http: always use any proxy auth method available We set CURLOPT_PROXYAUTH to use the most secure authentication method available only when the user has set configuration variables to specify a proxy. However, libcurl also supports specifying a proxy through environment variables. In that case libcurl defaults to only using the Basic proxy authentication method, because we do not use CURLOPT_PROXYAUTH. Set CURLOPT_PROXYAUTH to always use the most secure authentication method available, even when there is no git configuration telling us to use a proxy. This allows the user to use environment variables to configure a proxy that requires an authentication method different from Basic. Signed-off-by: Enrique A. Tobis <etobis@twosigma.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Lars Kellogg-Stedman	f6f2a9e42d	http: add support for specifying an SSL cipher list Teach git about a new option, "http.sslCipherList", which permits one to specify a list of ciphers to use when negotiating SSL connections. The setting can be overwridden by the GIT_SSL_CIPHER_LIST environment variable. Signed-off-by: Lars Kellogg-Stedman <lars@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Stefan Beller	826aed50cb	http: release the memory of a http pack request as well The cleanup function is used in 4 places now and it's always safe to free up the memory as well. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago
Jeff King	93f7d9108a	gettext.c: move get_preferred_languages() from http.c Calling setlocale(LC_MESSAGES, ...) directly from http.c, without including <locale.h>, was causing compilation warnings. Move the helper function to gettext.c that already includes the header and where locale-related issues are handled. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	10 years ago

1 2 3 4 5 ...

291 Commits (099a912a279415dd27716ee5de07ff347bfc49da)