Browse Source

bundle: add new version for use with SHA-256

Currently we detect the hash algorithm in use by the length of the
object ID.  This is inelegant and prevents us from using a different
hash algorithm that is also 256 bits in length.

Since we cannot extend the v2 format in a backward-compatible way, let's
add a v3 format, which is identical, except for the addition of
capabilities, which are prefixed by an at sign.  We add "object-format"
as the only capability and reject unknown capabilities, since we do not
have a network connection and therefore cannot negotiate with the other
side.

For compatibility, default to the v2 format for SHA-1 and require v3
for SHA-256.

In t5510, always use format v3 so we can be sure we produce consistent
results across hash algorithms.  Since head -n N lists the top N lines
instead of the Nth line, let's run our output through sed to normalize
it and compare it against a fixed value, which will make sure we get
exactly what we're expecting.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Reviewed-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
maint
brian m. carlson 5 years ago committed by Junio C Hamano
parent
commit
c5aecfc866
  1. 9
      Documentation/git-bundle.txt
  2. 30
      Documentation/technical/bundle-format.txt
  3. 5
      builtin/bundle.c
  4. 83
      bundle.c
  5. 4
      bundle.h
  6. 16
      t/t5510-fetch.sh
  7. 31
      t/t5607-clone-bundle.sh

9
Documentation/git-bundle.txt

@ -9,7 +9,8 @@ git-bundle - Move objects and refs by archive
SYNOPSIS SYNOPSIS
-------- --------
[verse] [verse]
'git bundle' create [-q | --quiet | --progress | --all-progress] [--all-progress-implied] <file> <git-rev-list-args> 'git bundle' create [-q | --quiet | --progress | --all-progress] [--all-progress-implied]
[--version=<version>] <file> <git-rev-list-args>
'git bundle' verify [-q | --quiet] <file> 'git bundle' verify [-q | --quiet] <file>
'git bundle' list-heads <file> [<refname>...] 'git bundle' list-heads <file> [<refname>...]
'git bundle' unbundle <file> [<refname>...] 'git bundle' unbundle <file> [<refname>...]
@ -102,6 +103,12 @@ unbundle <file>::
is activated. Unlike --all-progress this flag doesn't actually is activated. Unlike --all-progress this flag doesn't actually
force any progress display by itself. force any progress display by itself.


--version=<version>::
Specify the bundle version. Version 2 is the older format and can only be
used with SHA-1 repositories; the newer version 3 contains capabilities that
permit extensions. The default is the oldest supported format, based on the
hash algorithm in use.

-q:: -q::
--quiet:: --quiet::
This flag makes the command not to report its progress This flag makes the command not to report its progress

30
Documentation/technical/bundle-format.txt

@ -7,6 +7,8 @@ The Git bundle format is a format that represents both refs and Git objects.
We will use ABNF notation to define the Git bundle format. See We will use ABNF notation to define the Git bundle format. See
protocol-common.txt for the details. protocol-common.txt for the details.


A v2 bundle looks like this:

---- ----
bundle = signature *prerequisite *reference LF pack bundle = signature *prerequisite *reference LF pack
signature = "# v2 git bundle" LF signature = "# v2 git bundle" LF
@ -18,9 +20,28 @@ reference = obj-id SP refname LF
pack = ... ; packfile pack = ... ; packfile
---- ----


A v3 bundle looks like this:

----
bundle = signature *capability *prerequisite *reference LF pack
signature = "# v3 git bundle" LF

capability = "@" key ["=" value] LF
prerequisite = "-" obj-id SP comment LF
comment = *CHAR
reference = obj-id SP refname LF
key = 1*(ALPHA / DIGIT / "-")
value = *(%01-09 / %0b-FF)

pack = ... ; packfile
----

== Semantics == Semantics


A Git bundle consists of three parts. A Git bundle consists of several parts.

* "Capabilities", which are only in the v3 format, indicate functionality that
the bundle requires to be read properly.


* "Prerequisites" lists the objects that are NOT included in the bundle and the * "Prerequisites" lists the objects that are NOT included in the bundle and the
reader of the bundle MUST already have, in order to use the data in the reader of the bundle MUST already have, in order to use the data in the
@ -46,3 +67,10 @@ put any string here. The reader of the bundle MUST ignore the comment.
Note that the prerequisites does not represent a shallow-clone boundary. The Note that the prerequisites does not represent a shallow-clone boundary. The
semantics of the prerequisites and the shallow-clone boundaries are different, semantics of the prerequisites and the shallow-clone boundaries are different,
and the Git bundle v2 format cannot represent a shallow clone repository. and the Git bundle v2 format cannot represent a shallow clone repository.

== Capabilities

Because there is no opportunity for negotiation, unknown capabilities cause 'git
bundle' to abort. The only known capability is `object-format`, which specifies
the hash algorithm in use, and can take the same values as the
`extensions.objectFormat` configuration value.

5
builtin/bundle.c

@ -60,6 +60,7 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {
int all_progress_implied = 0; int all_progress_implied = 0;
int progress = isatty(STDERR_FILENO); int progress = isatty(STDERR_FILENO);
struct argv_array pack_opts; struct argv_array pack_opts;
int version = -1;


struct option options[] = { struct option options[] = {
OPT_SET_INT('q', "quiet", &progress, OPT_SET_INT('q', "quiet", &progress,
@ -71,6 +72,8 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {
OPT_BOOL(0, "all-progress-implied", OPT_BOOL(0, "all-progress-implied",
&all_progress_implied, &all_progress_implied,
N_("similar to --all-progress when progress meter is shown")), N_("similar to --all-progress when progress meter is shown")),
OPT_INTEGER(0, "version", &version,
N_("specify bundle format version")),
OPT_END() OPT_END()
}; };
const char* bundle_file; const char* bundle_file;
@ -91,7 +94,7 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {


if (!startup_info->have_repository) if (!startup_info->have_repository)
die(_("Need a repository to create a bundle.")); die(_("Need a repository to create a bundle."));
return !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts); return !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);
} }


static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) { static int cmd_bundle_verify(int argc, const char **argv, const char *prefix) {

83
bundle.c

@ -12,7 +12,16 @@
#include "refs.h" #include "refs.h"
#include "argv-array.h" #include "argv-array.h"


static const char bundle_signature[] = "# v2 git bundle\n";
static const char v2_bundle_signature[] = "# v2 git bundle\n";
static const char v3_bundle_signature[] = "# v3 git bundle\n";
static struct {
int version;
const char *signature;
} bundle_sigs[] = {
{ 2, v2_bundle_signature },
{ 3, v3_bundle_signature },
};


static void add_to_ref_list(const struct object_id *oid, const char *name, static void add_to_ref_list(const struct object_id *oid, const char *name,
struct ref_list *list) struct ref_list *list)
@ -23,15 +32,30 @@ static void add_to_ref_list(const struct object_id *oid, const char *name,
list->nr++; list->nr++;
} }


static const struct git_hash_algo *detect_hash_algo(struct strbuf *buf) static int parse_capability(struct bundle_header *header, const char *capability)
{
const char *arg;
if (skip_prefix(capability, "object-format=", &arg)) {
int algo = hash_algo_by_name(arg);
if (algo == GIT_HASH_UNKNOWN)
return error(_("unrecognized bundle hash algorithm: %s"), arg);
header->hash_algo = &hash_algos[algo];
return 0;
}
return error(_("unknown capability '%s'"), capability);
}

static int parse_bundle_signature(struct bundle_header *header, const char *line)
{ {
size_t len = strcspn(buf->buf, " \n"); int i;
int algo;


algo = hash_algo_by_length(len / 2); for (i = 0; i < ARRAY_SIZE(bundle_sigs); i++) {
if (algo == GIT_HASH_UNKNOWN) if (!strcmp(line, bundle_sigs[i].signature)) {
return NULL; header->version = bundle_sigs[i].version;
return &hash_algos[algo]; return 0;
}
}
return -1;
} }


static int parse_bundle_header(int fd, struct bundle_header *header, static int parse_bundle_header(int fd, struct bundle_header *header,
@ -42,14 +66,16 @@ static int parse_bundle_header(int fd, struct bundle_header *header,


/* The bundle header begins with the signature */ /* The bundle header begins with the signature */
if (strbuf_getwholeline_fd(&buf, fd, '\n') || if (strbuf_getwholeline_fd(&buf, fd, '\n') ||
strcmp(buf.buf, bundle_signature)) { parse_bundle_signature(header, buf.buf)) {
if (report_path) if (report_path)
error(_("'%s' does not look like a v2 bundle file"), error(_("'%s' does not look like a v2 or v3 bundle file"),
report_path); report_path);
status = -1; status = -1;
goto abort; goto abort;
} }


header->hash_algo = the_hash_algo;

/* The bundle header ends with an empty line */ /* The bundle header ends with an empty line */
while (!strbuf_getwholeline_fd(&buf, fd, '\n') && while (!strbuf_getwholeline_fd(&buf, fd, '\n') &&
buf.len && buf.buf[0] != '\n') { buf.len && buf.buf[0] != '\n') {
@ -57,19 +83,19 @@ static int parse_bundle_header(int fd, struct bundle_header *header,
int is_prereq = 0; int is_prereq = 0;
const char *p; const char *p;


if (*buf.buf == '-') {
is_prereq = 1;
strbuf_remove(&buf, 0, 1);
}
strbuf_rtrim(&buf); strbuf_rtrim(&buf);


if (!header->hash_algo) { if (header->version == 3 && *buf.buf == '@') {
header->hash_algo = detect_hash_algo(&buf); if (parse_capability(header, buf.buf + 1)) {
if (!header->hash_algo) {
error(_("unknown hash algorithm length"));
status = -1; status = -1;
break; break;
} }
continue;
}

if (*buf.buf == '-') {
is_prereq = 1;
strbuf_remove(&buf, 0, 1);
} }


/* /*
@ -449,13 +475,14 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)
} }


int create_bundle(struct repository *r, const char *path, int create_bundle(struct repository *r, const char *path,
int argc, const char **argv, struct argv_array *pack_options) int argc, const char **argv, struct argv_array *pack_options, int version)
{ {
struct lock_file lock = LOCK_INIT; struct lock_file lock = LOCK_INIT;
int bundle_fd = -1; int bundle_fd = -1;
int bundle_to_stdout; int bundle_to_stdout;
int ref_count = 0; int ref_count = 0;
struct rev_info revs; struct rev_info revs;
int min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3;


bundle_to_stdout = !strcmp(path, "-"); bundle_to_stdout = !strcmp(path, "-");
if (bundle_to_stdout) if (bundle_to_stdout)
@ -464,8 +491,22 @@ int create_bundle(struct repository *r, const char *path,
bundle_fd = hold_lock_file_for_update(&lock, path, bundle_fd = hold_lock_file_for_update(&lock, path,
LOCK_DIE_ON_ERROR); LOCK_DIE_ON_ERROR);


/* write signature */ if (version == -1)
write_or_die(bundle_fd, bundle_signature, strlen(bundle_signature)); version = min_version;

if (version < 2 || version > 3) {
die(_("unsupported bundle version %d"), version);
} else if (version < min_version) {
die(_("cannot write bundle version %d with algorithm %s"), version, the_hash_algo->name);
} else if (version == 2) {
write_or_die(bundle_fd, v2_bundle_signature, strlen(v2_bundle_signature));
} else {
const char *capability = "@object-format=";
write_or_die(bundle_fd, v3_bundle_signature, strlen(v3_bundle_signature));
write_or_die(bundle_fd, capability, strlen(capability));
write_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));
write_or_die(bundle_fd, "\n", 1);
}


/* init revs to list objects for pack-objects later */ /* init revs to list objects for pack-objects later */
save_commit_buffer = 0; save_commit_buffer = 0;

4
bundle.h

@ -13,6 +13,7 @@ struct ref_list {
}; };


struct bundle_header { struct bundle_header {
unsigned version;
struct ref_list prerequisites; struct ref_list prerequisites;
struct ref_list references; struct ref_list references;
const struct git_hash_algo *hash_algo; const struct git_hash_algo *hash_algo;
@ -21,7 +22,8 @@ struct bundle_header {
int is_bundle(const char *path, int quiet); int is_bundle(const char *path, int quiet);
int read_bundle_header(const char *path, struct bundle_header *header); int read_bundle_header(const char *path, struct bundle_header *header);
int create_bundle(struct repository *r, const char *path, int create_bundle(struct repository *r, const char *path,
int argc, const char **argv, struct argv_array *pack_options); int argc, const char **argv, struct argv_array *pack_options,
int version);
int verify_bundle(struct repository *r, struct bundle_header *header, int verbose); int verify_bundle(struct repository *r, struct bundle_header *header, int verbose);
#define BUNDLE_VERBOSE 1 #define BUNDLE_VERBOSE 1
int unbundle(struct repository *r, struct bundle_header *header, int unbundle(struct repository *r, struct bundle_header *header,

16
t/t5510-fetch.sh

@ -281,15 +281,19 @@ test_expect_success 'create bundle 1' '
cd "$D" && cd "$D" &&
echo >file updated again by origin && echo >file updated again by origin &&
git commit -a -m "tip" && git commit -a -m "tip" &&
git bundle create bundle1 master^..master git bundle create --version=3 bundle1 master^..master
' '


test_expect_success 'header of bundle looks right' ' test_expect_success 'header of bundle looks right' '
head -n 4 "$D"/bundle1 && cat >expect <<-EOF &&
head -n 1 "$D"/bundle1 | grep "^#" && # v3 git bundle
head -n 2 "$D"/bundle1 | grep "^-$OID_REGEX " && @object-format=$(test_oid algo)
head -n 3 "$D"/bundle1 | grep "^$OID_REGEX " && -OID updated by origin
head -n 4 "$D"/bundle1 | grep "^$" OID refs/heads/master

EOF
sed -e "s/$OID_REGEX/OID/g" -e "5q" "$D"/bundle1 >actual &&
test_cmp expect actual
' '


test_expect_success 'create bundle 2' ' test_expect_success 'create bundle 2' '

31
t/t5607-clone-bundle.sh

@ -4,6 +4,10 @@ test_description='some bundle related tests'
. ./test-lib.sh . ./test-lib.sh


test_expect_success 'setup' ' test_expect_success 'setup' '
test_oid_cache <<-EOF &&
version sha1:2
version sha256:3
EOF
test_commit initial && test_commit initial &&
test_tick && test_tick &&
git tag -m tag tag && git tag -m tag tag &&
@ -94,4 +98,31 @@ test_expect_success 'fetch SHA-1 from bundle' '
git fetch --no-tags foo/tip.bundle "$(cat hash)" git fetch --no-tags foo/tip.bundle "$(cat hash)"
' '


test_expect_success 'git bundle uses expected default format' '
git bundle create bundle HEAD^.. &&
head -n1 bundle | grep "^# v$(test_oid version) git bundle$"
'

test_expect_success 'git bundle v3 has expected contents' '
git branch side HEAD &&
git bundle create --version=3 bundle HEAD^..side &&
head -n2 bundle >actual &&
cat >expect <<-EOF &&
# v3 git bundle
@object-format=$(test_oid algo)
EOF
test_cmp expect actual &&
git bundle verify bundle
'

test_expect_success 'git bundle v3 rejects unknown capabilities' '
cat >new <<-EOF &&
# v3 git bundle
@object-format=$(test_oid algo)
@unknown=silly
EOF
test_must_fail git bundle verify new 2>output &&
test_i18ngrep "unknown capability .unknown=silly." output
'

test_done test_done

Loading…
Cancel
Save