kernel/git - git - PowerEL Git System

34237 Commits

6 Branches

36 Tags

131 MiB

Tree: 05efb7b757

Commit Graph

Author	SHA1	Message	Date
Nguyễn Thái Ngọc Duy	c6807a40dc	clone: open a shortcut for connectivity check In order to make sure the cloned repository is good, we run "rev-list --objects --not --all $new_refs" on the repository. This is expensive on large repositories. This patch attempts to mitigate the impact in this special case. In the "good" clone case, we only have one pack. If all of the following are met, we can be sure that all objects reachable from the new refs exist, which is the intention of running "rev-list ...": - all refs point to an object in the pack - there are no dangling pointers in any object in the pack - no objects in the pack point to objects outside the pack The second and third checks can be done with the help of index-pack as a slight variation of --strict check (which introduces a new condition for the shortcut: pack transfer must be used and the number of objects large enough to call index-pack). The first is checked in check_everything_connected after we get an "ok" from index-pack. "index-pack + new checks" is still faster than the current "index-pack + rev-list", which is the whole point of this patch. If any of the conditions fail, we fall back to the good old but expensive "rev-list ..". In that case it's even more expensive because we have to pay for the new checks in index-pack. But that should only happen when the other side is either buggy or malicious. Cloning linux-2.6 over file:// before after real 3m25.693s 2m53.050s user 5m2.037s 4m42.396s sys 0m13.750s 0m16.574s A more realistic test with ssh:// over wireless before after real 11m26.629s 10m4.213s user 5m43.196s 5m19.444s sys 0m35.812s 0m37.630s This shortcut is not applied to shallow clones, partly because shallow clones should have no more objects than a usual fetch and the cost of rev-list is acceptable, partly to avoid dealing with corner cases when grafting is involved. This shortcut does not apply to unpack-objects code path either because the number of objects must be small in order to trigger that code path. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	12 years ago
Junio C Hamano	d21c463d55	fetch/receive: remove over-pessimistic connectivity check Git 1.7.8 introduced an object and history re-validation step after "fetch" or "push" causes new history to be added to a receiving repository. This is to protect a malicious server or pushing client from corrupting the repository by taking advantage of an existing corrupt object that is unconnected to existing history. But this check is way over-pessimistic. During "fetch" or "receive-pack" (the server side of "push"), unpack-objects and index-pack already validate individual objects that are received, and the only thing we would want to catch are corrupted objects that already happen to exist in our repository but are not referenced from our refs. Such objects must have been written by an earlier run of our codepaths that write out loose objects or packfiles, and they must have done the validation of individual objects when they did so. The only thing left to worry about is the connectivity integrity, which can be checked with "rev-list --objects", which is much cheaper. We have been paying the 5x to 8x runtime overhead the --verify-objects often adds for no real gain. Revert check_everything_connected() not to use this over-pessimistic check. Credit goes to Nguyễn Thái Ngọc Duy, who originally identified the performance regression and endured multiple rounds of reviews to fix it. Signed-off-by: Junio C Hamano <gitster@pobox.com>	13 years ago
Junio C Hamano	f96400cb46	check_everything_connected(): libify Extract the helper function and the type definition of the iterator function it uses out of builtin/fetch.c into a separate source and a header file. Signed-off-by: Junio C Hamano <gitster@pobox.com>	14 years ago

Author

SHA1

Message

Date

Nguyễn Thái Ngọc Duy

c6807a40dc

clone: open a shortcut for connectivity check

In order to make sure the cloned repository is good, we run "rev-list
--objects --not --all $new_refs" on the repository. This is expensive
on large repositories. This patch attempts to mitigate the impact in
this special case.

In the "good" clone case, we only have one pack. If all of the
following are met, we can be sure that all objects reachable from the
new refs exist, which is the intention of running "rev-list ...":

 - all refs point to an object in the pack
 - there are no dangling pointers in any object in the pack
 - no objects in the pack point to objects outside the pack

The second and third checks can be done with the help of index-pack as
a slight variation of --strict check (which introduces a new condition
for the shortcut: pack transfer must be used and the number of objects
large enough to call index-pack). The first is checked in
check_everything_connected after we get an "ok" from index-pack.

"index-pack + new checks" is still faster than the current "index-pack
+ rev-list", which is the whole point of this patch. If any of the
conditions fail, we fall back to the good old but expensive "rev-list
..". In that case it's even more expensive because we have to pay for
the new checks in index-pack. But that should only happen when the
other side is either buggy or malicious.

Cloning linux-2.6 over file://

        before         after
real    3m25.693s      2m53.050s
user    5m2.037s       4m42.396s
sys     0m13.750s      0m16.574s

A more realistic test with ssh:// over wireless

        before         after
real    11m26.629s     10m4.213s
user    5m43.196s      5m19.444s
sys     0m35.812s      0m37.630s

This shortcut is not applied to shallow clones, partly because shallow
clones should have no more objects than a usual fetch and the cost of
rev-list is acceptable, partly to avoid dealing with corner cases when
grafting is involved.

This shortcut does not apply to unpack-objects code path either
because the number of objects must be small in order to trigger that
code path.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

12 years ago

Junio C Hamano

d21c463d55

fetch/receive: remove over-pessimistic connectivity check

Git 1.7.8 introduced an object and history re-validation step after
"fetch" or "push" causes new history to be added to a receiving
repository. This is to protect a malicious server or pushing client from
corrupting the repository by taking advantage of an existing corrupt
object that is unconnected to existing history.

But this check is way over-pessimistic.  During "fetch" or "receive-pack"
(the server side of "push"), unpack-objects and index-pack already
validate individual objects that are received, and the only thing we would
want to catch are corrupted objects that already happen to exist in our
repository but are not referenced from our refs.  Such objects must have
been written by an earlier run of our codepaths that write out loose
objects or packfiles, and they must have done the validation of individual
objects when they did so.  The only thing left to worry about is the
connectivity integrity, which can be checked with "rev-list --objects",
which is much cheaper.  We have been paying the 5x to 8x runtime overhead
the --verify-objects often adds for no real gain.

Revert check_everything_connected() not to use this over-pessimistic
check.

Credit goes to Nguyễn Thái Ngọc Duy, who originally identified the
performance regression and endured multiple rounds of reviews to fix it.

Signed-off-by: Junio C Hamano <gitster@pobox.com>

13 years ago

Junio C Hamano

f96400cb46

check_everything_connected(): libify

Extract the helper function and the type definition of the iterator
function it uses out of builtin/fetch.c into a separate source and a
header file.

Signed-off-by: Junio C Hamano <gitster@pobox.com>

14 years ago

3 Commits (05efb7b7575dd60a5997a0f7616d0cc222c5dbf8)