Go to file
Patrick Steinhardt c7b190dabd fetch: implement support for atomic reference updates
When executing a fetch, then git will currently allocate one reference
transaction per reference update and directly commit it. This means that
fetches are non-atomic: even if some of the reference updates fail,
others may still succeed and modify local references.

This is fine in many scenarios, but this strategy has its downsides.

- The view of remote references may be inconsistent and may show a
  bastardized state of the remote repository.

- Batching together updates may improve performance in certain
  scenarios. While the impact probably isn't as pronounced with loose
  references, the upcoming reftable backend may benefit as it needs to
  write less files in case the update is batched.

- The reference-update hook is currently being executed twice per
  updated reference. While this doesn't matter when there is no such
  hook, we have seen severe performance regressions when doing a
  git-fetch(1) with reference-transaction hook when the remote
  repository has hundreds of thousands of references.

Similar to `git push --atomic`, this commit thus introduces atomic
fetches. Instead of allocating one reference transaction per updated
reference, it causes us to only allocate a single transaction and commit
it as soon as all updates were received. If locking of any reference
fails, then we abort the complete transaction and don't update any
reference, which gives us an all-or-nothing fetch.

Note that this may not completely fix the first of above downsides, as
the consistent view also depends on the server-side. If the server
doesn't have a consistent view of its own references during the
reference negotiation phase, then the client would get the same
inconsistent view the server has. This is a separate problem though and,
if it actually exists, can be fixed at a later point.

This commit also changes the way we write FETCH_HEAD in case `--atomic`
is passed. Instead of writing changes as we go, we need to accumulate
all changes first and only commit them at the end when we know that all
reference updates succeeded. Ideally, we'd just do so via a temporary
file so that we don't need to carry all updates in-memory. This isn't
trivially doable though considering the `--append` mode, where we do not
truncate the file but simply append to it. And given that we support
concurrent processes appending to FETCH_HEAD at the same time without
any loss of data, seeding the temporary file with current contents of
FETCH_HEAD initially and then doing a rename wouldn't work either. So
this commit implements the simple strategy of buffering all changes and
appending them to the file on commit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-12 12:06:15 -08:00
.github Merge branch 'da/vs-build-iconv-fix' 2020-12-14 10:21:38 -08:00
Documentation fetch: implement support for atomic reference updates 2021-01-12 12:06:15 -08:00
block-sha1 block-sha1: take a size_t length parameter 2020-11-16 13:41:35 -08:00
builtin fetch: implement support for atomic reference updates 2021-01-12 12:06:15 -08:00
ci ci: avoid `set-env` construct in print-test-failures.sh 2020-11-17 12:12:30 -08:00
compat Merge branch 'da/vs-build-iconv-fix' 2020-12-14 10:21:38 -08:00
contrib Merge branch 'js/cmake-extra-built-ins-fix' 2020-12-14 10:21:38 -08:00
ewah bitmap: implement bitmap_is_subset() 2020-12-08 14:48:16 -08:00
git-gui Merge https://github.com/prati0100/git-gui 2020-12-18 15:07:10 -08:00
gitk-git
gitweb gitweb/Makefile: conditionally include ../GIT-VERSION-FILE 2020-12-08 16:56:56 -08:00
mergetools Merge branch 'pd/mergetool-nvimdiff' 2020-11-21 15:14:39 -08:00
negotiator
perl Merge branch 'jk/perl-warning' 2020-11-09 14:06:25 -08:00
po l10n: zh_CN: for git v2.30.0 l10n round 1 and 2 2020-12-27 19:23:27 +08:00
ppc
refs
sha1collisiondetection@855827c583
sha1dc
sha256
t fetch: implement support for atomic reference updates 2021-01-12 12:06:15 -08:00
templates
trace2
vcs-svn
xdiff
.cirrus.yml
.clang-format
.editorconfig
.gitattributes
.gitignore Merge branch 'fc/random-cleanup' 2020-12-08 15:11:21 -08:00
.gitmodules
.mailmap
.travis.yml
.tsan-suppressions
CODE_OF_CONDUCT.md
COPYING
GIT-VERSION-GEN The first batch in 2.31 cycle 2021-01-06 23:33:44 -08:00
INSTALL doc: mention Python 3.x supports 2020-12-14 15:01:03 -08:00
LGPL-2.1
Makefile Merge branch 'dd/doc-p4-requirements-update' 2020-12-18 15:15:17 -08:00
README.md
RelNotes The first batch in 2.31 cycle 2021-01-06 23:33:44 -08:00
abspath.c
aclocal.m4
add-interactive.c Merge branch 'js/add-i-color-fix' 2020-12-08 15:11:17 -08:00
add-interactive.h
add-patch.c Merge branch 'js/add-i-color-fix' 2020-12-08 15:11:17 -08:00
advice.c
advice.h
alias.c
alias.h
alloc.c
alloc.h
apply.c Merge branch 'ab/unreachable-break' 2020-12-18 15:15:18 -08:00
apply.h
archive-tar.c archive: support compression levels beyond 9 2020-11-09 11:25:45 -08:00
archive-zip.c
archive.c Merge branch 'rs/archive-plug-leak-refname' 2020-11-25 15:24:53 -08:00
archive.h Merge branch 'rs/archive-plug-leak-refname' 2020-11-25 15:24:53 -08:00
attr.c Use new HASHMAP_INIT macro to simplify hashmap initialization 2020-11-11 12:55:27 -08:00
attr.h
banned.h banned.h: mark ctime_r() and asctime_r() as banned 2020-12-02 14:30:39 -08:00
base85.c
bisect.c Merge branch 'sg/bisect-approximately-halfway' 2020-11-25 15:24:52 -08:00
bisect.h
blame.c Merge branch 'en/strmap' 2020-11-21 15:14:38 -08:00
blame.h
blob.c
blob.h
bloom.c Use new HASHMAP_INIT macro to simplify hashmap initialization 2020-11-11 12:55:27 -08:00
bloom.h
branch.c
branch.h
builtin.h Merge branch 'ds/maintenance-part-3' 2020-11-18 13:32:53 -08:00
bulk-checkin.c
bulk-checkin.h
bundle.c
bundle.h
cache-tree.c
cache-tree.h
cache.h Merge branch 'jk/oid-array-cleanup' 2020-12-17 15:06:40 -08:00
chdir-notify.c
chdir-notify.h
check-builtins.sh
check_bindir
checkout.c
checkout.h
color.c
color.h
column.c
column.h
combine-diff.c
command-list.txt Merge branch 'ab/retire-parse-remote' 2020-12-03 00:18:06 -08:00
commit-graph.c commit-graph: use size_t for array allocation and indexing 2020-12-07 12:32:04 -08:00
commit-graph.h
commit-reach.c
commit-reach.h
commit-slab-decl.h
commit-slab-impl.h
commit-slab.h
commit.c Merge branch 'en/merge-ort-recursive' 2021-01-06 23:33:44 -08:00
commit.h Merge branch 'en/merge-ort-recursive' 2021-01-06 23:33:44 -08:00
common-main.c
config.c Merge branch 'ds/config-literal-value' 2020-12-08 15:11:19 -08:00
config.h config: plumb --fixed-value into config API 2020-11-25 14:43:48 -08:00
config.mak.dev Merge branch 'jc/sparse-error-for-developer-build' 2020-11-18 13:32:54 -08:00
config.mak.in
config.mak.uname Merge branch 'rb/nonstop-config-mak-uname-update' 2020-12-18 15:15:18 -08:00
configure.ac
connect.c
connect.h
connected.c
connected.h
convert.c
convert.h
copy.c
credential.c
credential.h
csum-file.c
csum-file.h csum-file: add hashwrite_be64() 2020-11-12 09:40:06 -08:00
ctype.c
daemon.c
date.c
decorate.c
decorate.h
delta-islands.c
delta-islands.h
delta.h
detect-compiler
diff-delta.c
diff-lib.c Merge branch 'rs/plug-diff-cache-leak' 2020-11-25 15:24:53 -08:00
diff-no-index.c
diff.c Merge branch 'jc/diff-I-status-fix' 2020-12-18 15:15:18 -08:00
diff.h
diffcore-break.c
diffcore-delta.c
diffcore-order.c
diffcore-pickaxe.c
diffcore-rename.c
diffcore.h
dir-iterator.c
dir-iterator.h
dir.c Merge branch 'en/strmap' 2020-11-21 15:14:38 -08:00
dir.h
editor.c
entry.c
environment.c
exec-cmd.c
exec-cmd.h
fetch-negotiator.c
fetch-negotiator.h
fetch-pack.c Merge branch 'rs/fetch-pack-invalid-lockfile' 2020-12-08 15:11:20 -08:00
fetch-pack.h
fmt-merge-msg.c
fmt-merge-msg.h
fsck.c
fsck.h
fsmonitor.c
fsmonitor.h
fuzz-commit-graph.c
fuzz-pack-headers.c
fuzz-pack-idx.c
generate-cmdlist.sh
generate-configlist.sh
gettext.c
gettext.h
git-add--interactive.perl Merge branch 'js/add-i-color-fix' 2020-12-08 15:11:17 -08:00
git-archimport.perl
git-bisect.sh Merge branch 'mr/bisect-in-c-3' 2020-11-09 14:06:25 -08:00
git-compat-util.h Merge branch 'jc/compat-util-setitimer-fix' 2020-12-18 15:15:17 -08:00
git-cvsexportcommit.perl
git-cvsimport.perl
git-cvsserver.perl
git-difftool--helper.sh
git-filter-branch.sh
git-instaweb.sh
git-merge-octopus.sh
git-merge-one-file.sh
git-merge-resolve.sh
git-mergetool--lib.sh mergetool: avoid letting `list_tool_variants` break user-defined setups 2020-11-11 13:00:11 -08:00
git-mergetool.sh
git-p4.py Merge branch 'js/p4-default-branch' 2020-11-11 13:18:38 -08:00
git-quiltimport.sh
git-rebase--preserve-merges.sh
git-request-pull.sh
git-send-email.perl
git-sh-i18n.sh
git-sh-setup.sh
git-submodule.sh submodule: fix fetch_in_submodule logic 2020-11-24 13:14:09 -08:00
git-svn.perl
git-web--browse.sh
git.c maintenance: fix SEGFAULT when no repository 2020-11-30 13:44:15 -08:00
git.rc
gpg-interface.c
gpg-interface.h
graph.c
graph.h
grep.c grep: copy struct in one fell swoop 2020-11-30 13:55:54 -08:00
grep.h grep: use designated initializers for `grep_defaults` 2020-11-21 14:50:33 -08:00
hash.h cache.h: move hash/oid functions to hash.h 2020-12-04 13:55:14 -08:00
hashmap.c
hashmap.h
help.c help.c: help.autocorrect=never means "do not compute suggestions" 2020-11-25 13:02:15 -08:00
help.h
hex.c
http-backend.c
http-fetch.c
http-push.c
http-walker.c
http.c
http.h
ident.c
imap-send.c imap-send: parse default git config 2020-12-01 11:10:59 -08:00
iterator.h
json-writer.c
json-writer.h
khash.h
kwset.c
kwset.h
levenshtein.c
levenshtein.h
line-log.c
line-log.h
line-range.c
line-range.h
linear-assignment.c
linear-assignment.h
list-objects-filter-options.c list-objects-filter-options: fix function name in BUG 2020-11-16 14:28:25 -08:00
list-objects-filter-options.h
list-objects-filter.c
list-objects-filter.h
list-objects.c
list-objects.h
list.h
ll-merge.c
ll-merge.h
lockfile.c
lockfile.h
log-tree.c format-patch: make output filename configurable 2020-11-09 17:44:41 -08:00
log-tree.h format-patch: make output filename configurable 2020-11-09 17:44:41 -08:00
ls-refs.c
ls-refs.h
mailinfo.c
mailinfo.h
mailmap.c
mailmap.h
match-trees.c
mem-pool.c
mem-pool.h
merge-blobs.c
merge-blobs.h
merge-ort-wrappers.c
merge-ort-wrappers.h
merge-ort.c merge-ort: implement merge_incore_recursive() 2020-12-16 21:56:39 -08:00
merge-ort.h merge-ort: implement merge_incore_recursive() 2020-12-16 21:56:39 -08:00
merge-recursive.c commit: move reverse_commit_list() from merge-recursive 2020-12-16 21:56:39 -08:00
merge-recursive.h
merge.c
mergesort.c
mergesort.h
midx.c Merge branch 'tb/idx-midx-race-fix' 2020-12-08 15:11:18 -08:00
midx.h
name-hash.c
notes-cache.c
notes-cache.h
notes-merge.c
notes-merge.h
notes-utils.c
notes-utils.h
notes.c Merge branch 'na/notes-displayref-is-not-boolean' 2020-11-30 14:49:44 -08:00
notes.h
object-store.h
object.c Merge branch 'en/strmap' 2020-11-21 15:14:38 -08:00
object.h
oid-array.c oid-array: provide a for-loop iterator 2020-12-07 12:32:04 -08:00
oid-array.h oid-array: provide a for-loop iterator 2020-12-07 12:32:04 -08:00
oidmap.c
oidmap.h
oidset.c blame: silently ignore invalid ignore file objects 2020-11-10 13:05:06 -08:00
oidset.h
pack-bitmap-write.c pack-bitmap-write: better reuse bitmaps 2020-12-08 14:49:07 -08:00
pack-bitmap.c pack-bitmap: factor out 'add_commit_to_bitmap()' 2020-12-08 14:49:06 -08:00
pack-bitmap.h pack-bitmap: factor out 'bitmap_for_commit()' 2020-12-08 14:49:04 -08:00
pack-check.c fsck: correctly compute checksums on idx files larger than 4GB 2020-11-16 13:41:35 -08:00
pack-objects.c
pack-objects.h
pack-revindex.c compute pack .idx byte offsets using size_t 2020-11-16 13:41:35 -08:00
pack-revindex.h
pack-write.c Merge branch 'rs/hashwrite-be64' 2020-11-25 15:24:52 -08:00
pack.h
packfile.c Merge branch 'tb/idx-midx-race-fix' 2020-12-08 15:11:18 -08:00
packfile.h
pager.c
parse-options-cb.c
parse-options.c
parse-options.h
patch-delta.c
patch-ids.c
patch-ids.h
path.c
path.h
pathspec.c
pathspec.h
pkt-line.c
pkt-line.h
preload-index.c
pretty.c pretty format %(trailers): add a "key_value_separator" 2020-12-09 14:16:42 -08:00
pretty.h
prio-queue.c
prio-queue.h
progress.c
progress.h
promisor-remote.c
promisor-remote.h
prompt.c
prompt.h
protocol.c
protocol.h
prune-packed.c
prune-packed.h
quote.c
quote.h
range-diff.c Use new HASHMAP_INIT macro to simplify hashmap initialization 2020-11-11 12:55:27 -08:00
range-diff.h
reachable.c
reachable.h
read-cache.c
rebase-interactive.c
rebase-interactive.h
rebase.c
rebase.h
ref-filter.c Merge branch 'en/strmap' 2020-11-21 15:14:38 -08:00
ref-filter.h
reflog-walk.c
reflog-walk.h
refs.c init: provide useful advice about init.defaultBranch 2020-12-13 15:53:51 -08:00
refs.h get_default_branch_name(): prepare for showing some advice 2020-12-13 15:53:50 -08:00
refspec.c Merge branch 'fc/atmark-in-refspec' 2020-12-14 10:21:36 -08:00
refspec.h
remote-curl.c
remote.c Merge branch 'nk/refspecs-negative-fix' 2020-12-23 13:59:46 -08:00
remote.h fetch: extract writing to FETCH_HEAD 2021-01-12 12:06:14 -08:00
replace-object.c
replace-object.h
repo-settings.c
repository.c
repository.h
rerere.c
rerere.h
reset.c
reset.h
resolve-undo.c
resolve-undo.h
revision.c Merge branch 'ma/grep-init-default' 2020-12-08 15:11:20 -08:00
revision.h format-patch: make output filename configurable 2020-11-09 17:44:41 -08:00
run-command.c
run-command.h
send-pack.c Merge branch 'js/trace2-session-id' 2020-12-08 15:11:20 -08:00
send-pack.h
sequencer.c Merge branch 'en/strmap' 2020-11-21 15:14:38 -08:00
sequencer.h Merge branch 'en/merge-ort-api-null-impl' 2020-11-18 13:32:53 -08:00
serve.c upload-pack, serve: log received client session ID 2020-11-11 18:26:53 -08:00
serve.h
server-info.c
setup.c
sh-i18n--envsubst.c
sha1-file.c apply: don't use core.sharedRepository to create working tree files 2020-12-02 14:35:51 -08:00
sha1-lookup.c
sha1-lookup.h
sha1-name.c
sha1dc_git.c
sha1dc_git.h
shallow.c
shallow.h
shell.c
shortlog.h
sideband.c Merge branch 'jk/sideband-more-error-checking' 2020-11-09 14:06:29 -08:00
sideband.h
sigchain.c
sigchain.h
split-index.c
split-index.h
stable-qsort.c
strbuf.c
strbuf.h
streaming.c
streaming.h
string-list.c
string-list.h
strmap.c strmap: take advantage of FLEXPTR_ALLOC_STR when relevant 2020-11-11 12:55:27 -08:00
strmap.h strmap: make callers of strmap_remove() to call it in void context 2020-12-15 15:30:44 -08:00
strvec.c
strvec.h
sub-process.c
sub-process.h
submodule-config.c
submodule-config.h
submodule.c submodules: fix of regression on fetching of non-init subsub-repo 2020-12-09 12:32:07 -08:00
submodule.h
symlinks.c
tag.c
tag.h
tar.h
tempfile.c
tempfile.h
thread-utils.c
thread-utils.h
tmp-objdir.c
tmp-objdir.h
trace.c
trace.h
trace2.c trace2: add a public function for getting the SID 2020-11-11 18:26:52 -08:00
trace2.h trace2: add a public function for getting the SID 2020-11-11 18:26:52 -08:00
trailer.c pretty format %(trailers): add a "key_value_separator" 2020-12-09 14:16:42 -08:00
trailer.h pretty format %(trailers): add a "key_value_separator" 2020-12-09 14:16:42 -08:00
transport-helper.c
transport-internal.h
transport.c transport: log received server session ID 2020-11-11 18:26:53 -08:00
transport.h
tree-diff.c
tree-walk.c
tree-walk.h
tree.c tree: enable cmp_cache_name_compare() to be used elsewhere 2020-12-13 14:18:20 -08:00
tree.h tree: enable cmp_cache_name_compare() to be used elsewhere 2020-12-13 14:18:20 -08:00
unicode-width.h
unimplemented.sh
unix-socket.c
unix-socket.h
unpack-trees.c
unpack-trees.h
upload-pack.c Merge branch 'tb/partial-clone-filters-fix' 2020-12-17 15:06:40 -08:00
upload-pack.h
url.c
url.h
urlmatch.c
urlmatch.h
usage.c Merge branch 'jt/trace-error-on-warning' 2020-12-08 15:11:17 -08:00
userdiff.c
userdiff.h
utf8.c
utf8.h
varint.c
varint.h
version.c
version.h
versioncmp.c
walker.c
walker.h
wildmatch.c
wildmatch.h
worktree.c worktree: teach `repair` to fix multi-directional breakage 2020-12-21 13:44:28 -08:00
worktree.h
wrap-for-bin.sh
wrapper.c
write-or-die.c
ws.c
wt-status.c
wt-status.h
xdiff-interface.c
xdiff-interface.h
zlib.c

README.md

Build status

Git - fast, scalable, distributed revision control system

Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals.

Git is an Open Source project covered by the GNU General Public License version 2 (some parts of it are under different licenses, compatible with the GPLv2). It was originally written by Linus Torvalds with help of a group of hackers around the net.

Please read the file INSTALL for installation instructions.

Many Git online resources are accessible from https://git-scm.com/ including full documentation and Git related tools.

See Documentation/gittutorial.txt to get started, then see Documentation/giteveryday.txt for a useful minimum set of commands, and Documentation/git-<commandname>.txt for documentation of each command. If git has been correctly installed, then the tutorial can also be read with man gittutorial or git help tutorial, and the documentation of each command with man git-<commandname> or git help <commandname>.

CVS users may also want to read Documentation/gitcvs-migration.txt (man gitcvs-migration or git help cvs-migration if git is installed).

The user discussion and development of Git take place on the Git mailing list -- everyone is welcome to post bug reports, feature requests, comments and patches to git@vger.kernel.org (read Documentation/SubmittingPatches for instructions on patch submission). To subscribe to the list, send an email with just "subscribe git" in the body to majordomo@vger.kernel.org. The mailing list archives are available at https://lore.kernel.org/git/, http://marc.info/?l=git and other archival sites.

Issues which are security relevant should be disclosed privately to the Git Security mailing list git-security@googlegroups.com.

The maintainer frequently sends the "What's cooking" reports that list the current status of various development topics to the mailing list. The discussion following them give a good reference for project status, development direction and remaining tasks.

The name "git" was given by Linus Torvalds when he wrote the very first version. He described the tool as "the stupid content tracker" and the name as (depending on your mood):

  • random three-letter combination that is pronounceable, and not actually used by any common UNIX command. The fact that it is a mispronunciation of "get" may or may not be relevant.
  • stupid. contemptible and despicable. simple. Take your pick from the dictionary of slang.
  • "global information tracker": you're in a good mood, and it actually works for you. Angels sing, and a light suddenly fills the room.
  • "goddamn idiotic truckload of sh*t": when it breaks