From 8b8d9a229888adb737851c4d7eeaa9a50e37afe1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Dec 2022 15:14:07 +0000 Subject: [PATCH 01/11] protocol v2: add server-side "bundle-uri" skeleton MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++ bundle-uri.c | 36 ++++++ bundle-uri.h | 7 ++ serve.c | 6 + t/t5701-git-serve.sh | 40 +++++- 5 files changed, 289 insertions(+), 1 deletion(-) diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt index 59bf41cefb..10bd2d40ce 100644 --- a/Documentation/gitprotocol-v2.txt +++ b/Documentation/gitprotocol-v2.txt @@ -578,6 +578,207 @@ and associated requested information, each separated by a single space. obj-info = obj-id SP obj-size +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. To be useful to clients it's expected that it'll be issued +after an `ls-refs` and before `fetch`, but CAN be issued at any time +in the dialog. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +One way that servers could take advantage of these bundles is that the +server would anticipate that fresh clones will download a known bundle, +followed by catching up to the current state of the repository using ref +tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` request, the response is a +list of key-value pairs provided as packet lines with value +`=`. Each `` should be interpreted as a config key from +the `bundle.*` namespace to construct a list of bundles. These keys are +grouped by a `bundle..` subsection, where each key corresponding to a +given `` contributes attributes to the bundle defined by that ``. +See linkgit:git-config[1] for the specific details of these keys and how +the Git client will interpret their values. + +Clients MUST parse the line according to the above format, lines that do +not conform to the format SHOULD be discarded. The user MAY be warned in +such a case. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +URI CONTENTS:: +The content at the advertised URIs MUST be one of two types. ++ +The advertised URI may contain a bundle file that `git bundle verify` +would accept. I.e. they MUST contain one or more reference tips for +use by the client, MUST indicate prerequisites (in any) with standard +"-" prefixes, and MUST indicate their "object-format", if +applicable. ++ +The advertised URI may alternatively contain a plaintext file that `git +config --list` would accept (with the `--file` option). The key-value +pairs in this list are in the `bundle.*` namespace (see +linkgit:git-config[1]). + +bundle-uri CLIENT ERROR RECOVERY:: +A client MUST above all gracefully degrade on errors, whether that +error is because of bad missing/data in the bundle URI(s), because +that client is too dumb to e.g. understand and fully parse out bundle +headers and their prerequisite relationships, or something else. ++ +Server operators should feel confident in turning on "bundle-uri" and +not worry if e.g. their CDN goes down that clones or fetches will run +into hard failures. Even if the server bundle bundle(s) are +incomplete, or bad in some way the client should still end up with a +functioning repository, just as if it had chosen not to use this +protocol extension. ++ +All subsequent discussion on client and server interaction MUST keep +this in mind. + +bundle-uri SERVER TO CLIENT:: +The ordering of the returned bundle uris is not significant. Clients +MUST parse their headers to discover their contained OIDS and +prerequisites. A client MUST consider the content of the bundle(s) +themselves and their header as the ultimate source of truth. ++ +A server MAY even return bundle(s) that don't have any direct +relationship to the repository being cloned (either through accident, +or intentional "clever" configuration), and expect a client to sort +out what data they'd like from the bundle(s), if any. + +bundle-uri CLIENT TO SERVER:: +The client SHOULD provide reference tips found in the bundle header(s) +as 'have' lines in any subsequent `fetch` request. A client MAY also +ignore the bundle(s) entirely if doing so is deemed worse for some +reason, e.g. if the bundles can't be downloaded, it doesn't like the +tips it finds etc. + +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION:: +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s) +of the bundle(s) the client finds that the ref tips it wants can be +retrieved entirely from advertised bundle(s), the client MAY disconnect +from the Git server. The results of such a 'clone' or 'fetch' should be +indistinguishable from the state attained without using bundle-uri. + +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY:: +A client MAY perform an early disconnect while still downloading the +bundle(s) (having streamed and parsed their headers). In such a case +the client MUST gracefully recover from any errors related to +finishing the download and validation of the bundle(s). ++ +I.e. a client might need to re-connect and issue a 'fetch' command, +and possibly fall back to not making use of 'bundle-uri' at all. ++ +This "MAY" behavior is specified as such (and not a "SHOULD") on the +assumption that a server advertising bundle uris is more likely than +not to be serving up a relatively large repository, and to be pointing +to URIs that have a good chance of being in working order. A client +MAY e.g. look at the payload size of the bundles as a heuristic to see +if an early disconnect is worth it, should falling back on a full +"fetch" dialog be necessary. + +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION:: +A client SHOULD commence a negotiation of a PACK from the server via +the "fetch" command using the OID tips found in advertised bundles, +even if's still in the process of downloading those bundle(s). ++ +This allows for aggressive early disconnects from any interactive +server dialog. The client blindly trusts that the advertised OID tips +are relevant, and issues them as 'have' lines, it then requests any +tips it would like (usually from the "ls-refs" advertisement) via +'want' lines. The server will then compute a (hopefully small) PACK +with the expected difference between the tips from the bundle(s) and +the data requested. ++ +The only connection the client then needs to keep active is to the +concurrently downloading static bundle(s), when those and the +incremental PACK are retrieved they should be inflated and +validated. Any errors at this point should be gracefully recovered +from, see above. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The client constructs a bundle list from the `=` pairs +provided by the server. These pairs are part of the `bundle.*` namespace +as documented in linkgit:git-config[1]. In this section, we discuss some +of these keys and describe the actions the client will do in response to +this information. + +In particular, the `bundle.version` key specifies an integer value. The +only accepted value at the moment is `1`, but if the client sees an +unexpected value here then the client MUST ignore the bundle list. + +As long as `bundle.version` is understood, all other unknown keys MAY be +ignored by the client. The server will guarantee compatibility with older +clients, though newer clients may be better able to use the extra keys to +minimize downloads. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new `bundle.version` value or values in 'bundle-uri' +capability advertisement itself, and/or by new future `bundle-uri` +request arguments. + +Some example key-value pairs that are not currently implemented but could +be implemented in the future include: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "oid=" shortcut and "prerequisite=" shortcut. For + expressing the common case of a bundle with one tip and no + prerequisites, or one tip and one prerequisite. ++ +This would allow for optimizing the common case of servers who'd like +to provide one "big bundle" containing only their "main" branch, +and/or incremental updates thereof. ++ +A client receiving such a a response MAY assume that they can skip +retrieving the header from a bundle at the indicated URI, and thus +save themselves and the server(s) the request(s) needed to inspect the +headers of that bundle or bundles. + GIT --- Part of the linkgit:git[1] suite diff --git a/bundle-uri.c b/bundle-uri.c index 79a914f961..28d8966005 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -563,6 +563,42 @@ cleanup: return result; } +/** + * API for serve.c. + */ + +int bundle_uri_advertise(struct repository *r, struct strbuf *value UNUSED) +{ + static int advertise_bundle_uri = -1; + + if (advertise_bundle_uri != -1) + goto cached; + + advertise_bundle_uri = 0; + repo_config_get_maybe_bool(r, "uploadpack.advertisebundleuris", &advertise_bundle_uri); + +cached: + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) + die(_("bundle-uri: expected flush after arguments")); + + /* TODO: Implement the communication */ + + packet_writer_flush(&writer); + + return 0; +} + /** * General API for {transport,connect}.c etc. */ diff --git a/bundle-uri.h b/bundle-uri.h index 4dbc269823..357111ecce 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -4,6 +4,7 @@ #include "hashmap.h" #include "strbuf.h" +struct packet_reader; struct repository; struct string_list; @@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * API for serve.c. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + /** * General API for {transport,connect}.c etc. */ diff --git a/serve.c b/serve.c index 733347f602..cbf4a143cf 100644 --- a/serve.c +++ b/serve.c @@ -7,6 +7,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; static int client_hash_algo = GIT_HASH_SHA1; @@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 1896f671cb..f21e5e9d33 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' ' + test_config uploadpack.advertiseBundleURIs true && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From 8f788eb8b75502acb5c7d771279e486bae626dde Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Dec 2022 15:14:08 +0000 Subject: [PATCH 02/11] t: create test harness for 'bundle-uri' command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous change allowed for a Git server to advertise the 'bundle-uri' command as a capability based on the uploadPack.advertiseBundleURIs config option. Create a set of tests that check that this capability is advertised using 'git ls-remote'. In order to test this functionality across three protocols (file, git, and http), create lib-bundle-uri-protocol.sh to generalize the tests, allowing the other test scripts to set an environment variable and otherwise inherit the setup and tests from this script. The tests currently only test that the 'bundle-uri' command is advertised or not. Other actions will be tested as the Git client learns to request the 'bundle-uri' command and parse its response. To help with URI escaping, specifically for file paths with a space in them, extract a 'sed' invocation from t9199-git-svn-info.sh into a helper function for use here, too. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- t/lib-bundle-uri-protocol.sh | 85 ++++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++++++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++++++ t/t9119-git-svn-info.sh | 2 +- t/test-lib-functions.sh | 7 +++ 6 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 t/lib-bundle-uri-protocol.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh new file mode 100644 index 0000000000..2da22a39cb --- /dev/null +++ b/t/lib-bundle-uri-protocol.sh @@ -0,0 +1,85 @@ +# Set up and run tests of the 'bundle-uri' command in protocol v2 +# +# The test that includes this script should set BUNDLE_URI_PROTOCOL +# to one of "file", "git", or "http". + +BUNDLE_URI_TEST_PARENT= +BUNDLE_URI_TEST_URI= +BUNDLE_URI_TEST_BUNDLE_URI= +case "$BUNDLE_URI_PROTOCOL" in +file) + BUNDLE_URI_PARENT=file_parent + BUNDLE_URI_REPO_URI="file://$PWD/file_parent" + BUNDLE_URI_BUNDLE_URI="$BUNDLE_URI_REPO_URI/fake.bdl" + test_set_prereq BUNDLE_URI_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + BUNDLE_URI_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + BUNDLE_URI_REPO_URI="$GIT_DAEMON_URL/parent" + BUNDLE_URI_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq BUNDLE_URI_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + BUNDLE_URI_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + BUNDLE_URI_REPO_URI="$HTTPD_URL/smart/http_parent" + BUNDLE_URI_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq BUNDLE_URI_HTTP + ;; +*) + BUG "Need to pass valid BUNDLE_URI_PROTOCOL (was \"$BUNDLE_URI_PROTOCOL\")" + ;; +esac + +test_expect_success "setup protocol v2 $BUNDLE_URI_PROTOCOL:// tests" ' + git init "$BUNDLE_URI_PARENT" && + test_commit -C "$BUNDLE_URI_PARENT" one && + git -C "$BUNDLE_URI_PARENT" config uploadpack.advertiseBundleURIs true +' + +case "$BUNDLE_URI_PROTOCOL" in +http) + test_expect_success "setup config for $BUNDLE_URI_PROTOCOL:// tests" ' + git -C "$BUNDLE_URI_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +BUNDLE_URI_BUNDLE_URI_ESCAPED=$(echo "$BUNDLE_URI_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + test_when_finished "git -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs true" && + git -C "$BUNDLE_URI_PARENT" config uploadpack.advertiseBundleURIs false && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$BUNDLE_URI_REPO_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$BUNDLE_URI_REPO_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 0000000000..37bdb725bc --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +BUNDLE_URI_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 0000000000..8add1b37ab --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +BUNDLE_URI_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 0000000000..129daa0226 --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'http://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'http://' transport +# +BUNDLE_URI_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t9119-git-svn-info.sh b/t/t9119-git-svn-info.sh index 8201c3e808..088d1c57a8 100755 --- a/t/t9119-git-svn-info.sh +++ b/t/t9119-git-svn-info.sh @@ -28,7 +28,7 @@ test_cmp_info () { rm -f tmp.expect tmp.actual } -quoted_svnrepo="$(echo $svnrepo | sed 's/ /%20/')" +quoted_svnrepo="$(echo $svnrepo | test_uri_escape)" test_expect_success 'setup repository and import' ' mkdir info && diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 796093a7b3..0fd7d4a200 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1751,6 +1751,13 @@ test_path_is_hidden () { return 1 } +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + # Check that the given command was invoked as part of the # trace2-format trace on stdin. # From 0cfde740f0b2c9474aae3a381d1d6e97c7468e7a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Dec 2022 15:14:09 +0000 Subject: [PATCH 03/11] clone: request the 'bundle-uri' command when available MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Set up all the needed client parts of the 'bundle-uri' protocol v2 command, without actually doing anything with the bundle URIs. If the server says it supports 'bundle-uri' teach Git to issue the 'bundle-uri' command after the 'ls-refs' during 'git clone'. The returned key=value pairs are passed to the bundle list code which is tested using a different ingest mechanism in t5750-bundle-uri-parse.sh. At this point, Git does nothing with that bundle list. It will not download any of the bundles. That will come in a later change after these protocol bits are finalized. The no-op client is initially used only by 'git clone' to test the basic functionality, and eventually will bootstrap the initial download of Git objects during a fresh clone. The bundle URI client will not be integrated into other fetches until a mechanism is created to select a subset of bundles for download. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- builtin/clone.c | 6 +++++ connect.c | 44 +++++++++++++++++++++++++++++++ remote.h | 5 ++++ t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++ transport-helper.c | 13 +++++++++ transport-internal.h | 7 +++++ transport.c | 51 ++++++++++++++++++++++++++++++++++++ transport.h | 19 ++++++++++++++ 8 files changed, 164 insertions(+) diff --git a/builtin/clone.c b/builtin/clone.c index 0e4348686b..5866873d26 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1271,6 +1271,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/connect.c b/connect.c index 5ea53deda2..624a10f18e 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -491,6 +492,49 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundles, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 1c4621b414..1ebbe42792 100644 --- a/remote.h +++ b/remote.h @@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +struct bundle_list; +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 2da22a39cb..d44c6e10f9 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -83,3 +83,22 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav # Server advertised bundle-uri capability grep "< bundle-uri" log ' + +test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' + test_when_finished "rm -rf log cloned" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + clone "$BUNDLE_URI_REPO_URI" cloned \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log && + + # Client issued bundle-uri command + grep "> command=bundle-uri" log +' diff --git a/transport-helper.c b/transport-helper.c index e95267a4ab..3ea7c2bb5a 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733a..90ea749e5c 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index e7b97194c1..b6f279e92c 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -359,6 +360,32 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + + if (!transport->bundles) { + CALLOC_ARRAY(transport->bundles, 1); + init_bundle_list(transport->bundles); + } + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + transport->bundles, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -902,6 +929,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1054,6 +1082,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1068,6 +1097,9 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + CALLOC_ARRAY(ret->bundles, 1); + init_bundle_list(ret->bundles); + if (!remote) BUG("No remote provided to transport_get()"); @@ -1482,6 +1514,23 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Check config only once. */ + if (transport->got_remote_bundle_uri) + return 0; + transport->got_remote_bundle_uri = 1; + + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport, unsigned int flags) { int in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER); @@ -1512,6 +1561,8 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + clear_bundle_list(transport->bundles); + free(transport->bundles); free(transport); return ret; } diff --git a/transport.h b/transport.h index b5bf7b3e70..85150f504f 100644 --- a/transport.h +++ b/transport.h @@ -62,6 +62,7 @@ enum transport_family { TRANSPORT_FAMILY_IPV6 }; +struct bundle_list; struct transport { const struct transport_vtable *vtable; @@ -76,6 +77,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct bundle_list *bundles; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts); const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From 7cce9074a728fb32501054577963d8ae31d007be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Dec 2022 15:14:10 +0000 Subject: [PATCH 04/11] bundle-uri client: add boolean transfer.bundleURI setting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The yet-to-be introduced client support for bundle-uri will always fall back on a full clone, but we'd still like to be able to ignore a server's bundle-uri advertisement entirely. The new transfer.bundleURI config option defaults to 'false', but a user can set it to 'true' to enable checking for bundle URIs from the origin Git server using protocol v2. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- Documentation/config/transfer.txt | 6 ++++++ t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++++++- transport.c | 8 ++++++++ 3 files changed, 32 insertions(+), 1 deletion(-) diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt index 264812cca4..c3ac767d1e 100644 --- a/Documentation/config/transfer.txt +++ b/Documentation/config/transfer.txt @@ -115,3 +115,9 @@ transfer.unpackLimit:: transfer.advertiseSID:: Boolean. When true, client and server processes will advertise their unique session IDs to their remote counterpart. Defaults to false. + +transfer.bundleURI:: + When `true`, local `git clone` commands will request bundle + information from the remote server (if advertised) and download + bundles before continuing the clone through the Git protocol. + Defaults to `false`. diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index d44c6e10f9..75ea8c4418 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -85,10 +85,11 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav ' test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' - test_when_finished "rm -rf log cloned" && + test_when_finished "rm -rf log cloned cloned2" && GIT_TRACE_PACKET="$PWD/log" \ git \ + -c transfer.bundleURI=false \ -c protocol.version=2 \ clone "$BUNDLE_URI_REPO_URI" cloned \ >actual 2>err && @@ -99,6 +100,22 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque # Server advertised bundle-uri capability grep "< bundle-uri" log && + # Client did not issue bundle-uri command + ! grep "> command=bundle-uri" log && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c transfer.bundleURI=true \ + -c protocol.version=2 \ + clone "$BUNDLE_URI_REPO_URI" cloned2 \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log && + # Client issued bundle-uri command grep "> command=bundle-uri" log ' diff --git a/transport.c b/transport.c index b6f279e92c..b4cf2c0252 100644 --- a/transport.c +++ b/transport.c @@ -1516,6 +1516,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) int transport_get_remote_bundle_uri(struct transport *transport) { + int value = 0; const struct transport_vtable *vtable = transport->vtable; /* Check config only once. */ @@ -1523,6 +1524,13 @@ int transport_get_remote_bundle_uri(struct transport *transport) return 0; transport->got_remote_bundle_uri = 1; + /* + * Don't request bundle-uri from the server unless configured to + * do so by the transfer.bundleURI=true config option. + */ + if (git_config_get_bool("transfer.bundleuri", &value) || !value) + return 0; + if (!vtable->get_bundle_uri) return error(_("bundle-uri operation not supported by protocol")); From 1b759e0cf1c18c637e40c6c191a0b614e4e4ea43 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 22 Dec 2022 15:14:11 +0000 Subject: [PATCH 05/11] transport: rename got_remote_heads The 'got_remote_heads' member of 'struct git_transport_data' was used historically to indicate that the initial server connection was made and the ref advertisement was returned. With protocol v2, that initial handshake does not necessarily include the ref advertisement, so this member is not an accurate name. Thankfully, all uses of the member are only checking to see if the handshake should take place, not whether or not some local data has the ref advertisement. Rename the member to 'finished_handshake' to represent the proper state. Note that the variable is only set to 1 during the handshake() method. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- transport.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/transport.c b/transport.c index b4cf2c0252..757ad552bf 100644 --- a/transport.c +++ b/transport.c @@ -198,7 +198,7 @@ struct git_transport_data { struct git_transport_options options; struct child_process *conn; int fd[2]; - unsigned got_remote_heads : 1; + unsigned finished_handshake : 1; enum protocol_version version; struct oid_array extra_have; struct oid_array shallow; @@ -345,7 +345,7 @@ static struct ref *handshake(struct transport *transport, int for_push, case protocol_unknown_version: BUG("unknown protocol version"); } - data->got_remote_heads = 1; + data->finished_handshake = 1; transport->hash_algo = reader.hash_algo; if (reader.line_peeked) @@ -421,7 +421,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.negotiation_tips = data->options.negotiation_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow; - if (!data->got_remote_heads) { + if (!data->finished_handshake) { int i; int must_list_refs = 0; for (i = 0; i < nr_heads; i++) { @@ -461,7 +461,7 @@ static int fetch_refs_via_pack(struct transport *transport, to_fetch, nr_heads, &data->shallow, &transport->pack_lockfiles, data->version); - data->got_remote_heads = 0; + data->finished_handshake = 0; data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked; @@ -846,7 +846,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re if (transport_color_config() < 0) return -1; - if (!data->got_remote_heads) + if (!data->finished_handshake) get_refs_via_connect(transport, 1, NULL); memset(&args, 0, sizeof(args)); @@ -894,7 +894,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re else ret = finish_connect(data->conn); data->conn = NULL; - data->got_remote_heads = 0; + data->finished_handshake = 0; return ret; } @@ -914,7 +914,7 @@ static int disconnect_git(struct transport *transport) { struct git_transport_data *data = transport->data; if (data->conn) { - if (data->got_remote_heads && !transport->stateless_rpc) + if (data->finished_handshake && !transport->stateless_rpc) packet_flush(data->fd[1]); close(data->fd[0]); if (data->fd[1] >= 0) @@ -949,7 +949,7 @@ void transport_take_over(struct transport *transport, data->conn = child; data->fd[0] = data->conn->out; data->fd[1] = data->conn->in; - data->got_remote_heads = 0; + data->finished_handshake = 0; transport->data = data; transport->vtable = &taken_over_vtable; @@ -1150,7 +1150,7 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->smart_options = &(data->options); data->conn = NULL; - data->got_remote_heads = 0; + data->finished_handshake = 0; } else { /* Unknown protocol in URL. Pass to external handler. */ int len = external_specification_len(url); From 70b9c1037325ee82bc0832f4ca2d30c6ebf4808e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Thu, 22 Dec 2022 15:14:12 +0000 Subject: [PATCH 06/11] bundle-uri client: add helper for testing server MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. In the "git clone" case we'll have already done the handshake(), but not here. Add an extra case to check for this handshake in get_bundle_uri() for ease of use for future callers. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- t/helper/test-bundle-uri.c | 46 ++++++++++++++++++++++++++++++++++++ t/lib-bundle-uri-protocol.sh | 46 ++++++++++++++++++++++++++++++++++++ transport.c | 7 ++++++ 3 files changed, 99 insertions(+) diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 25afd39342..f815918701 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -3,6 +3,10 @@ #include "bundle-uri.h" #include "strbuf.h" #include "string-list.h" +#include "transport.h" +#include "ref-filter.h" +#include "remote.h" +#include "refs.h" enum input_mode { KEY_VALUE_PAIRS, @@ -68,6 +72,46 @@ usage: usage_with_options(usage, options); } +static int cmd_ls_remote(int argc, const char **argv) +{ + const char *uploadpack = NULL; + struct string_list server_options = STRING_LIST_INIT_DUP; + const char *dest; + struct remote *remote; + struct transport *transport; + int status = 0; + + dest = argc > 1 ? argv[1] : NULL; + + remote = remote_get(dest); + if (!remote) { + if (dest) + die(_("bad repository '%s'"), dest); + die(_("no remote configured to get bundle URIs from")); + } + if (!remote->url_nr) + die(_("remote '%s' has no configured URL"), dest); + + transport = transport_get(remote, NULL); + if (uploadpack) + transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack); + if (server_options.nr) + transport->server_options = &server_options; + + if (transport_get_remote_bundle_uri(transport) < 0) { + error(_("could not get the bundle-uri list")); + status = 1; + goto cleanup; + } + + print_bundle_list(stdout, transport->bundles); + +cleanup: + if (transport_disconnect(transport)) + return 1; + return status; +} + int cmd__bundle_uri(int argc, const char **argv) { const char *usage[] = { @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv) return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); if (!strcmp(argv[1], "parse-config")) return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); + if (!strcmp(argv[1], "ls-remote")) + return cmd_ls_remote(argc - 1, argv + 1); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 75ea8c4418..5620e23038 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -119,3 +119,49 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque # Client issued bundle-uri command grep "> command=bundle-uri" log ' + +# The remaining tests will all assume transfer.bundleURI=true +# +# This test can be removed when transfer.bundleURI is enabled by default. +test_expect_success 'enable transfer.bundleURI for remaining tests' ' + git config --global transfer.bundleURI true +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.only.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 and extra data" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.only.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED" && + + # Extra data should be ignored + test_config -C "$BUNDLE_URI_PARENT" bundle.only.extra bogus && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' diff --git a/transport.c b/transport.c index 757ad552bf..0f35114a13 100644 --- a/transport.c +++ b/transport.c @@ -371,6 +371,13 @@ static int get_bundle_uri(struct transport *transport) init_bundle_list(transport->bundles); } + if (!data->finished_handshake) { + struct ref *refs = handshake(transport, 0, NULL, 0); + + if (refs) + free_refs(refs); + } + /* * "Support" protocol v0 and v2 without bundle-uri support by * silently degrading to a NOOP. From 738dc7d4a5f42376bc748008dd8792aa4eb25627 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 22 Dec 2022 15:14:13 +0000 Subject: [PATCH 07/11] bundle-uri: serve bundle.* keys from config Implement the "bundle-uri" protocol v2 capability by populating the key=value packet lines from the local Git config. The list of bundles is provided from the keys beginning with "bundle.". In the future, we may want to filter this list to be more specific to the exact known keys that the server intends to share, but for flexibility at the moment we will assume that the config values are well-formed. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- bundle-uri.c | 16 +++++++++++++++- t/lib-bundle-uri-protocol.sh | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 28d8966005..26ff4b062d 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -581,6 +581,16 @@ cached: return advertise_bundle_uri; } +static int config_to_packet_line(const char *key, const char *value, void *data) +{ + struct packet_reader *writer = data; + + if (!strncmp(key, "bundle.", 7)) + packet_write_fmt(writer->fd, "%s=%s", key, value); + + return 0; +} + int bundle_uri_command(struct repository *r, struct packet_reader *request) { @@ -592,7 +602,11 @@ int bundle_uri_command(struct repository *r, if (request->status != PACKET_READ_FLUSH) die(_("bundle-uri: expected flush after arguments")); - /* TODO: Implement the communication */ + /* + * Read all "bundle.*" config lines to the client as key=value + * packet lines. + */ + git_config(config_to_packet_line, &writer); packet_writer_flush(&writer); diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 5620e23038..3022ea4a95 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -136,6 +136,8 @@ test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol [bundle] version = 1 mode = all + [bundle "only"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED EOF test-tool bundle-uri \ @@ -157,6 +159,36 @@ test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol [bundle] version = 1 mode = all + [bundle "only"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 with list" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle1.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl" && + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle2.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl" && + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle3.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + [bundle "bundle1"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl + [bundle "bundle2"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl + [bundle "bundle3"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl EOF test-tool bundle-uri \ From 9ea57964953dec11dcbbd5d4bf44a5e3781f5880 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 22 Dec 2022 15:14:14 +0000 Subject: [PATCH 08/11] strbuf: introduce strbuf_strip_file_from_path() The strbuf_parent_directory() method was added as a static method in contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended config and starts maintenance, 2021-12-03) and then removed in 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now there is a need for a similar method in the bundle URI feature. Re-add the method, this time in strbuf.c, but with a new name: strbuf_strip_file_from_path(). The method requirements are slightly modified to allow a trailing slash, in which case nothing is done, which makes the name change valuable. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- strbuf.c | 6 ++++++ strbuf.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/strbuf.c b/strbuf.c index 0890b1405c..c383f41a3c 100644 --- a/strbuf.c +++ b/strbuf.c @@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path, free(path2); return res; } + +void strbuf_strip_file_from_path(struct strbuf *sb) +{ + char *path_sep = find_last_dir_sep(sb->buf); + strbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0); +} diff --git a/strbuf.h b/strbuf.h index 76965a17d4..f6dbb9681e 100644 --- a/strbuf.h +++ b/strbuf.h @@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer, int strbuf_edit_interactively(struct strbuf *buffer, const char *path, const char *const *env); +/* + * Remove the filename from the provided path string. If the path + * contains a trailing separator, then the path is considered a directory + * and nothing is modified. + * + * Examples: + * - "/path/to/file" -> "/path/to/" + * - "/path/to/dir/" -> "/path/to/dir/" + */ +void strbuf_strip_file_from_path(struct strbuf *sb); + void strbuf_add_lines(struct strbuf *sb, const char *prefix, const char *buf, From ebc39479558fb4893c72a35065e6792515acef86 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 22 Dec 2022 15:14:15 +0000 Subject: [PATCH 09/11] bundle-uri: allow relative URLs in bundle lists Bundle providers may want to distribute that data across multiple CDNs. This might require a change in the base URI, all the way to the domain name. If all bundles require an absolute URI in their 'uri' value, then every push to a CDN would require altering the table of contents to match the expected domain and exact location within it. Allow a bundle list to specify a relative URI for the bundles. This URI is based on where the client received the bundle list. For a list provided in the 'bundle-uri' protocol v2 command, the Git remote URI is the base URI. Otherwise, the bundle list was provided from an HTTP URI not using the Git protocol, and that URI is the base URI. This allows easier distribution of bundle data. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- bundle-uri.c | 16 +++++++- bundle-uri.h | 14 +++++++ t/helper/test-bundle-uri.c | 2 + t/t5750-bundle-uri-parse.sh | 82 +++++++++++++++++++++++++++++++++++++ transport.c | 3 ++ 5 files changed, 116 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 26ff4b062d..69929d363c 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -7,6 +7,7 @@ #include "hashmap.h" #include "pkt-line.h" #include "config.h" +#include "remote.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list) for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); + free(list->baseURI); } int for_all_bundles_in_list(struct bundle_list *list, @@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value, if (!strcmp(subkey, "uri")) { if (bundle->uri) return -1; - bundle->uri = xstrdup(value); + bundle->uri = relative_url(list->baseURI, value, NULL); return 0; } @@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri, .error_action = CONFIG_ERROR_ERROR, }; + if (!list->baseURI) { + struct strbuf baseURI = STRBUF_INIT; + strbuf_addstr(&baseURI, uri); + + /* + * If the URI does not end with a trailing slash, then + * remove the filename portion of the path. This is + * important for relative URIs. + */ + strbuf_strip_file_from_path(&baseURI); + list->baseURI = strbuf_detach(&baseURI, NULL); + } result = git_config_from_file_with_options(config_to_bundle_list, filename, list, &opts); diff --git a/bundle-uri.h b/bundle-uri.h index 357111ecce..c505444bc7 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -61,6 +61,20 @@ struct bundle_list { int version; enum bundle_list_mode mode; struct hashmap bundles; + + /** + * The baseURI of a bundle_list is the URI that provided the list. + * + * In the case of the 'bundle-uri' protocol v2 command, the base + * URI is the URI of the Git remote. + * + * Otherwise, the bundle list was downloaded over HTTP from some + * known URI. 'baseURI' is set to that value. + * + * The baseURI is used as the base for any relative URIs + * advertised by the bundle list at that location. + */ + char *baseURI; }; void init_bundle_list(struct bundle_list *list); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index f815918701..5df5bc3b89 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo init_bundle_list(&list); + list.baseURI = xstrdup(""); + switch (mode) { case KEY_VALUE_PAIRS: if (argc != 1) diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index c2fe3f9c5a..7b4f930e53 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -30,6 +30,58 @@ test_expect_success 'bundle_uri_parse_line() just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'bundle_uri_parse_line(): relative URIs' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line(): relative URIs and parent paths' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=../../bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /../bundle.bdl + EOF + + # TODO: We would prefer if parsing a bundle list would not cause + # a die() and instead would give a warning and allow the rest of + # a Git command to continue. This test_must_fail is necessary for + # now until the interface for relative_url() allows for reporting + # an error instead of die()ing. + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + grep "fatal: cannot strip one component off url" err +' + test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' cat >in <<-\EOF && =bogus-value @@ -136,6 +188,36 @@ test_expect_success 'parse config format: just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'parse config format: relative URIs' ' + cat >in <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = bundle.bdl + [bundle "two"] + uri = ../bundle.bdl + [bundle "three"] + uri = sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-config in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'parse config format edge cases: empty key or value' ' cat >in1 <<-\EOF && = bogus-value diff --git a/transport.c b/transport.c index 0f35114a13..241f8a6ba2 100644 --- a/transport.c +++ b/transport.c @@ -1538,6 +1538,9 @@ int transport_get_remote_bundle_uri(struct transport *transport) if (git_config_get_bool("transfer.bundleuri", &value) || !value) return 0; + if (!transport->bundles->baseURI) + transport->bundles->baseURI = xstrdup(transport->url); + if (!vtable->get_bundle_uri) return error(_("bundle-uri operation not supported by protocol")); From 12b0a14b9e5f140a9a4d5a3e3c47fae00a53c6fc Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 22 Dec 2022 15:14:16 +0000 Subject: [PATCH 10/11] bundle-uri: download bundles from an advertised list The logic in fetch_bundle_uri() is useful for the --bundle-uri option of 'git clone', but is not helpful when the clone operation discovers a list of URIs from the bundle-uri protocol v2 command. To actually download and unbundle the advertised bundles, we need a different mechanism. Create the new fetch_bundle_list() method which is very similar to fetch_bundle_uri() except that it relies on download_bundle_list() instead of fetch_bundle_uri_internal(). The download_bundle_list() method will recursively call fetch_bundle_uri_internal() if any of the advertised URIs serve a bundle list instead of a bundle. This will also follow the bundle.list.mode setting from the input list: "any" will download only one such URI while "all" will download data from all of the URIs. In an identical way to fetch_bundle_uri(), the bundles are unbundled after all of the bundle lists have been expanded and all necessary URIs. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- bundle-uri.c | 21 +++++++++++++++++++++ bundle-uri.h | 14 ++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index 69929d363c..36268dda17 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -577,6 +577,27 @@ cleanup: return result; } +int fetch_bundle_list(struct repository *r, struct bundle_list *list) +{ + int result; + struct bundle_list global_list; + + init_bundle_list(&global_list); + + /* If a bundle is added to this global list, then it is required. */ + global_list.mode = BUNDLE_MODE_ALL; + + if ((result = download_bundle_list(r, list, &global_list, 0))) + goto cleanup; + + result = unbundle_all_bundles(r, &global_list); + +cleanup: + for_all_bundles_in_list(&global_list, unlink_bundle, NULL); + clear_bundle_list(&global_list); + return result; +} + /** * API for serve.c. */ diff --git a/bundle-uri.h b/bundle-uri.h index c505444bc7..d5e89f1671 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -107,6 +107,20 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * Given a bundle list that was already advertised (likely by the + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the + * bundles according to the bundle strategy of that list. + * + * It is expected that the given 'list' is initialized, including its + * 'baseURI' value. + * + * Returns non-zero if there was an error trying to download the list + * or any of its advertised bundles. + */ +int fetch_bundle_list(struct repository *r, + struct bundle_list *list); + /** * API for serve.c. */ From 876094ac16870727207e6afc0b26060101a52359 Mon Sep 17 00:00:00 2001 From: Derrick Stolee Date: Thu, 22 Dec 2022 15:14:17 +0000 Subject: [PATCH 11/11] clone: unbundle the advertised bundles A previous change introduced the transport methods to acquire a bundle list from the 'bundle-uri' protocol v2 command, when advertised _and_ when the client has chosen to enable the feature. Teach Git to download and unbundle the data advertised by those bundles during 'git clone'. This takes place between the ref advertisement and the object data download, and stateful connections will linger while the client downloads bundles. In the future, we should consider closing the remote connection during this process. Also, since the --bundle-uri option exists, we do not want to mix the advertised bundles with the user-specified bundles. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano --- builtin/clone.c | 25 ++++++++++++--- t/lib-bundle-uri-protocol.sh | 21 +++++++++++-- t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 7 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 5866873d26..430b2e981e 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1271,11 +1271,26 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); - /* - * Populate transport->got_remote_bundle_uri and - * transport->bundle_uri. We might get nothing. - */ - transport_get_remote_bundle_uri(transport); + if (!bundle_uri) { + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + + if (transport->bundles && + hashmap_get_size(&transport->bundles->bundles)) { + /* At this point, we need the_repository to match the cloned repo. */ + if (repo_init(the_repository, git_dir, work_tree)) + warning(_("failed to initialize the repo, skipping bundle URI")); + else if (fetch_bundle_list(the_repository, + transport->bundles)) + warning(_("failed to fetch advertised bundles")); + } else { + clear_bundle_list(transport->bundles); + FREE_AND_NULL(transport->bundles); + } + } if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 3022ea4a95..a4a1af8d02 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -85,7 +85,7 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav ' test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' - test_when_finished "rm -rf log cloned cloned2" && + test_when_finished "rm -rf log* cloned*" && GIT_TRACE_PACKET="$PWD/log" \ git \ @@ -117,7 +117,24 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque grep "< bundle-uri" log && # Client issued bundle-uri command - grep "> command=bundle-uri" log + grep "> command=bundle-uri" log && + + GIT_TRACE_PACKET="$PWD/log3" \ + git \ + -c transfer.bundleURI=true \ + -c protocol.version=2 \ + clone --bundle-uri="$BUNDLE_URI_BUNDLE_URI" \ + "$BUNDLE_URI_REPO_URI" cloned3 \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log3 && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log3 && + + # Client did not issue bundle-uri command (--bundle-uri override) + ! grep "> command=bundle-uri" log3 ' # The remaining tests will all assume transfer.bundleURI=true diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index b2524a24c2..1928ea1dd7 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -772,6 +772,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' ' git clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo ' +test_expect_success 'auto-discover bundle URI from HTTP clone' ' + test_when_finished rm -rf trace.txt repo2 "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle" --all && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.mode all && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 \ + -c transfer.bundleURI=true clone \ + $HTTPD_URL/smart/repo2.git repo2 && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt +' + +test_expect_success 'auto-discover multiple bundles from HTTP clone' ' + test_when_finished rm -rf trace.txt repo3 "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + test_commit -C src new && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/new.bundle" HEAD~1..HEAD && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.mode all && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.new.uri "$HTTPD_URL/new.bundle" && + + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 \ + -c transfer.bundleURI=true clone \ + $HTTPD_URL/smart/repo3.git repo3 && + + # We should fetch _both_ bundles + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/new.bundle"\] + EOF + grep -f pattern trace.txt +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.