A compatibility wrapper for writev(3p) has been reintroduced,
including fixes for CMake build and 'MAX_IO_SIZE' limits on NonStop.
Calls to write(3p) in send_sideband() and cat_blob() have been
refactored to use writev(3p) wrappers to reduce syscall overhead.
* ps/writev:
fast-import: use writev(3p) to send cat-blob responses
sideband: use writev(3p) to send pktlines
wrapper: properly handle MAX_IO_SIZE in writev(3p)
wrapper: introduce writev(3p) wrappers
compat/posix: introduce writev(3p) wrapper
The known limitations of the ref format migration in 'git refs' have
been moved to be displayed as a warning admonition directly under the
description of the 'migrate' subcommand, improving visibility. A
reference to 'git-maintenance' has also been corrected to use the
'linkgit' macro.
* kh/doc-refs-migrate-limitations:
doc: refs: linkgit to git-maintenance(1)
doc: refs: put ref migration warning under the command
'git add' has been taught a new '--resolved' option to stage
conflict-resolved paths, while leaving unrelated local changes
unstaged. It scans the unmerged paths for leftover conflict
markers and aborts if any are found.
* jc/add-resolved:
add: introduce '--resolved' option
read-cache: add remove_file_from_index_with_flags()
merge-ll: consolidate conflict marker scanning logic
read-cache: reindent
'git worktree' add did not prevent DWIM behavior when '-b' or '-B'
was specified, which has been corrected.
* yn/worktree-add-no-dwim-with-b:
worktree add: shouldn't dwim if -b or -B is given
The 'ssh-agent' tests in 't7528' have been fixed to work when the
user's login shell is csh-like, by explicitly passing '-s' to
'ssh-agent' to force Bourne shell syntax.
* kl/t7528-ssh-agent-for-csh-users:
t7528: fix failure under csh
Concurrent downloads of packfiles via packfile URIs and dumb HTTP are
safer by avoiding concurrent appends to the staging file. Opening in
read-write mode with separate file offsets prevents corruption and
preserves resumability. 'fetch-pack' now tolerates pre-existing
'.keep' files.
* tn/packfile-uri-concurrency:
fetch-pack: accept "pack" output for packfile URIs
http: permit unlinking partial packs on Windows
http: avoid concurrent appends to partial packs
http: accept HTTP 416 for complete partial packs
http: avoid closing index-pack input twice
http-fetch: correct --index-pack-arg documentation
The 'TRACE2_ANCESTRY' prerequisite in the 't0213' test script has been
refined to avoid failures under user-mode emulation by verifying that
the ancestry collector reports the expected process names rather than
the emulator binary name.
* jm/t0213-skip-emulated-ancestry-tests:
t0213: skip ancestry tests under user-mode emulation
A crash in the 'sparse-index' collapse code when encountering an
invalidated cache-tree node (due to an intent-to-add path) has been
fixed by avoiding collapsing such subtrees.
* ds/sparse-index-ita-crash:
sparse-index: avoid crash on intent-to-add entry outside the cone
A boundary case check in reachability bitmap traversal has been
corrected to properly handle the object at position zero, which was
previously skipped, leading to redundant bitmap loading.
* dl/pack-bitmap-position-zero:
pack-bitmap: handle objects at bitmap position zero
A memory leak in 'git merge' when run without arguments (which
triggers the default-to-upstream path) has been fixed. A test has
been added to cover this case.
* tc/merge-default-to-upstream-leakfix:
merge: fix leak with merge.defaultToUpstream
'git cat-file --batch-command' that asked for 'contents' without
'type' segfaults, which has been corrected.
* jk/cat-file-batch-wo-type-fix:
cat-file: handle content request for --batch-command without type
The get_commit_action() function has been refactored to be a pure
predicate by moving the side-effecting line-level log range folding to
simplify_commit(). This ensures that evaluating a commit's action
before the walk reaches it does not prematurely mutate its tracked
line ranges, making it safer for potential lookahead evaluations.
* mm/revision-pure-get-commit-action:
revision: make get_commit_action() a pure predicate
The code path that deals with relative paths in the 'diff-lib' has
been cleaned up.
* jk/diff-relative-cached-unmerged-more:
diff-lib: skip paths outside prefix in oneway_diff()
diff-lib: drop stale comment about advancing o->pos
When answering a `cat-blob` command, `cat_blob()` issues three separate
calls to write(3p) on the cat-blob fd: one for the header line, one for
the full blob payload, and one for the trailing newline. Frontends like
git-filter-repo issue these commands in bulk, once per rewritten blob,
so the syscall overhead adds up.
Use `writev_in_full()` to send all three parts with a single syscall.
This can be benchmarked with the following setup:
$ git cat-file --unordered --filter=object:type=blob
--batch-check='cat-blob %(objectname)' --batch-all-objects >request
$ git fast-import --cat-blob-fd=3 <request
Executing this with 100,000 objects in linux.git:
Benchmark 1: HEAD~
Time (mean ± σ): 1.320 s ± 0.003 s [User: 1.154 s, System: 0.161 s]
Range (min … max): 1.314 s … 1.324 s 10 runs
Benchmark 2: HEAD
Time (mean ± σ): 1.270 s ± 0.022 s [User: 1.133 s, System: 0.132 s]
Range (min … max): 1.209 s … 1.282 s 10 runs
Summary
HEAD ran
1.04 ± 0.02 times faster than HEAD~
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Every pktline that we send out via `send_sideband()` currently requires
two syscalls: one to write the pktline's length, and one to send its
data. This typically isn't all that much of a problem, but under extreme
load the syscalls may cause contention in the kernel.
Refactor the code to instead use the newly introduced writev(3p) infra
so that we can send out the data with a single syscall. This reduces the
number of syscalls from around 133,000 calls to write(3p) to around
67,000 calls to writev(3p).
This change leads to a performance improvement for git-upload-pack(1),
but we have to cheat a bit to really make it measurable. Usually, the
time is strongly dominated by generating the packfile itself. But if we
precompute the pack and serve it via the pack-objects hook then we can
essentially eliminate that overhead. The following setup is executed in
the Git repository:
$ cat >request <<-EOF
0048want 5ce91c059e side-band no-progress
00000009done
EOF
$ echo 5ce91c059e | git pack-objects --revs --stdout >pack
$ cat >hook <<-EOF
#!/bin/sh
cat >/dev/null
cat "$(pwd)"/pack
EOF
$ chmod u+x hook
$ git -c uploadpack.packObjectsHook="$(pwd)"/hook upload-pack . <request
Benchmarking the last command leads to the following results:
Benchmark 1: HEAD~
Time (mean ± σ): 192.9 ms ± 0.6 ms [User: 106.5 ms, System: 95.3 ms]
Range (min … max): 191.7 ms … 194.1 ms 50 runs
Benchmark 2: HEAD
Time (mean ± σ): 141.1 ms ± 0.7 ms [User: 63.2 ms, System: 86.6 ms]
Range (min … max): 139.8 ms … 142.7 ms 50 runs
Summary
HEAD ran
1.37 ± 0.01 times faster than HEAD~
This might not be impressive in absolute numbers when you also take into
account the time it takes to generate the packfile itself. But GitLab
(and supposedly other forges) have caching mechanisms in place that work
exactly like the above setup, where repeated incoming requests can be
served from the same cached packfile. And in those cases, the impact is
sizeable.
More importantly though, as hinted at above, GitLab has observed in the
past that with enough cache hits we eventually start to saturate a
semaphore in the Linux kernel itself in the pipe write path. This
bottleneck is being moved a bit by having to do less syscalls.
Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some systems like NonStop set a comparatively small `MAX_IO_SIZE`, which
limits the maximum number of bytes we're allowed to write in a single
call. We already handle this limit properly in `xwrite()`, but we have
recently introduced wrappers for writev(3p) where we don't. This will
cause the syscall to return EINVAL in case somebody passes an iovec
entry to writev(3p) that is larger than `MAX_IO_SIZE`.
Introduce a new function `xwritev()` that is similar to `xwrite()` in
that it handles such platform-specific nuances:
- We only pass the leading iovec entries to writev(3p) that fit into
`MAX_IO_SIZE`, pretending that the underlying syscall performed a
short write. This mirrors how `xwrite()` chomps overly large
requests before handing them to write(3p). As a consequence, callers
will never see writev(3p)'s EINVAL error for requests whose summed
length would overflow an ssize_t, but observe a short write instead.
- If already the first iovec entry exceeds the limit we instead punt
to `xwrite()`, which knows to handle this case for us.
- We restart the underlying syscall on EINTR and EAGAIN, just like
`xwrite()` does for write(3p).
Adapt `writev_in_full()` to use this new wrapper. With the retry logic
now living in `xwritev()`, the calling loop becomes the exact mirror
image of `write_in_full()`, which also retains the responsibility of
translating a zero-length write into ENOSPC.
Reported-by: Randall Becker <randall.becker@nexbridge.ca>
Helped-by: Jeff King <peff@peff.net>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the preceding commit we have added a compatibility wrapper for the
writev(3p) syscall. Introduce some generic wrappers for this function
that we nowadays take for granted in the Git codebase.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In a subsequent commit we're going to add the first caller to
writev(3p). Introduce a compatibility wrapper for this syscall that we
can use on systems that don't have this syscall.
The syscall exists on modern Unixes like Linux and macOS, and seemingly
even for NonStop according to [1]. It doesn't seem to exist on Windows
though.
[1]: http://nonstoptools.com/manuals/OSS-SystemCalls.pdf
[2]: https://www.gnu.org/software/gnulib/manual/html_node/writev.html
Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
I have to scroll down at least three screens in man(1) from the
`migrate` description in order to see the “known limitations” for
it. This is important information since the text says that concurrent
writes can lead to an inconsistent migrated state. Let’s move that text
up to the command description and put it inside a Warning admonition.
This section made sense when it was added in 25a0023f (builtin/refs:
new command to migrate ref storage formats, 2024-06-06); `migrate` was
the only subcommand, and this section was visible from the command
description. A one-page man page. But that is not the case anymore
now that the command has nine subcommands to describe.
Acked-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git for Windows has been updated to avoid auto-detecting the symlink
type if the target path starts with a slash, preventing NTLM
credential leaks when checking out repositories with crafted
symbolic links pointing to network shares.
* js/mingw-symlink-net-share-leak:
mingw: skip symlink type auto-detection for network share targets
The alias tests in 't/t0014-alias.sh' have been updated to dynamically
query the list of deprecated commands using 'git
--list-cmds=deprecated' to avoid test failures when running with
'WITH_BREAKING_CHANGES' in a build directory that contains stale
executables of formerly deprecated commands.
* jk/t0014-dynamic-deprecated-cmds:
t0014: generate deprecated command names dynamically
t0014: factor out choice of deprecated commands
The image version used by the static-analysis CI job has been bumped
to ubuntu-latest (Ubuntu 24.04), which brings in a newer Coccinelle
version that resolves a severe performance regression. A false
positive warning from the 'CHECK_ASSERTION_SIDE_EFFECTS' build with
GCC 15 in the Bloom filter code has also been silenced to facilitate
the image upgrade.
* jk/ci-static-analysis-image-bump:
ci: bump ubuntu image version for static-analysis job
bloom: silence CHECK_ASSERTION_SIDE_EFFECTS false positive
'git branch -d' has been taught to report when a branch cannot be
deleted because it is being used in an active bisect run.
* rs/branch-delete-bisect-warning:
branch: report active bisect run when rejecting delete
Object database housekeeping in 'git gc' and 'git maintenance' has
been refactored to be pluggable. The files-backend-specific logic,
including incremental and geometric repacking as well as object
pruning, has been moved out of the command implementation and into the
files object database source, enabling future alternative object
database backends to implement their own housekeeping services.
* ps/odb-pluggable-housekeeping:
odb: make optimizations pluggable
builtin/gc: fix signedness issues in ODB-related functionality
builtin/gc: refactor ODB optimizations to operate on "files" source
builtin/gc: introduce `odb_optimize_required()`
builtin/gc: move geometric repacking into `odb_optimize()`
builtin/gc: introduce object database optimization options
builtin/gc: inline config values specific to the "files" backend
builtin/gc: make repack arguments self-contained
builtin/gc: extract object database optimizations into separate function
builtin/gc: move worktree and rerere tasks before object optimizations
odb: run "pre-auto-gc" hook for all maintenance tasks
t7900: simplify how we check for maintenance tasks
Two bugs in how 'git rebase' handles skipped 'fixup' and 'squash'
commands have been fixed. One bug caused an incorrect commit count to
be shown in the template message when multiple commands were skipped,
and another prevented the editor from opening when the final command
in a chain containing 'fixup -c' was skipped.
* pw/rebase-fixup-fixes:
rebase: remember fixup -c after skipping fixup/squash
rebase -i: fix counting of fixups after rebase --skip
'git diff --relative' running with '--cached' has been corrected to
avoid a segfault when encountering unmerged paths outside the
prefix.
* jk/diff-relative-cached-unmerged:
diff-lib: add idx/tree sanity check to oneway_diff
diff: ignore unmerged paths outside prefix with --relative --cached
'git worktree add <path> <branch>' DWIMs <branch> to a
remote-tracking branch when neither -b, -B, nor --detach
is given.
However, 'git worktree add -b <new-branch> <path> <branch>' can
still DWIM <branch>, causing <new-branch> to be ignored. This is a
regression introduced by 128e5496b3 (worktree add: extend DWIM to
infer --orphan, 2023-05-17), which appeared in Git 2.42.
Signed-off-by: Yoichi NAKAYAMA <yoichi.nakayama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Ben uses the +github GMail trick to identify emails sent to him by folks
that found his GitHub profile. At the time, that also meant he had to
commit under the same email for GitHub to recognize his commits. He has
since found out that GitHub can be configured with more than one email
for identification, and he would prefer his canonical email to omit
mention of GitHub (where it's not relevant) going forward.
Signed-off-by: D. Ben Knoble <ben.knoble@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `chriscool@tuxfamily.org` address is an old one that I don't use
anymore, while `christian.couder@gmail.com` is the address I have been
sending patches from for a long time.
Let's swap the two addresses in the existing entry, so that the Gmail
address becomes the primary one and the old tuxfamily.org address is
mapped to it. This way both addresses still resolve to the same person,
and the address I actually use is the canonical one.
Signed-off-by: Christian Couder <christian.couder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Tests for 'git merge-base --is-ancestor' have been added to cover
exit codes (0 for success, 1 for non-ancestor, 128 for errors) and
to ensure it cannot be combined with '--all'.
* ns/merge-base-is-ancestor-tests:
merge-base: add tests for --is-ancestor
'git mv' has been updated to check for a missing destination
leading directory during the checking phase, allowing 'git mv -n'
to report the failure. The error message when the rename(2)
syscall fails has also been improved to name both the source and
the destination.
* lo/mv-missing-dest-dir-check:
mv: reject a destination whose leading path is missing or a symlink
mv: name both source and destination when rename fails
Documentation for 'git replay' has been updated to refer to its
configuration variables.
* kh/doc-replay-config:
doc: replay: move “default” to the right-hand side
doc: replay: use a nested description list
doc: replay: improve config description
doc: link to config for git-replay(1)
A new test helper commit_body() has been introduced to print the
message body of a commit, and various tests have been updated to use
it instead of spelling out the command pipeline manually and losing
the exit status of the 'git cat-file' command on the upstream of the
pipe.
* sk/test-commit-body-helper:
t: use commit_body to extract commit message bodies
test-lib-functions: add commit_body helper
Synopsis and options in the documentation for 'git format-patch',
'git imap-send', 'git send-email', and 'git request-pull' have been
updated to the modern style.
* ja/doc-synopsis-style-yet-more:
doc: convert git-request-pull synopsis and options to new style
doc: convert git-send-email synopsis and options to new style
doc: convert git-format-patch synopsis and options to new style
doc: convert git-imap-send synopsis and options to new style
The remote-matching logic for submodules has been corrected to resolve
'url.*.insteadOf' aliases before comparing the inventoried URL from
'.gitmodules' with the URLs of configured remotes.
* en/submodule-insteadof-remote-match:
submodule: resolve insteadOf aliases when matching remote
rewrites_release() in 'remote.c' has been updated to free 'struct
rewrite' instances, their '.instead_of' arrays, and their contents.
* jc/remote-insteadof-leakfix:
remote: plug memory leaks
A segfault when 'git clone --revision' talks to a server that does not
support protocol v2 (falling back to protocol v0) has been corrected.
* af/clone-revision-v0-segfault-fix:
builtin/clone: fix segfault when using --revision with protocol v0
Traversals with '--exclude-first-parent-only' have been corrected
to properly stop after the first parent even when it has already
been marked as 'SEEN'.
* jc/exclude-first-parent-seen:
revision: honor --exclude-first-parent-only with SEEN first parent
When the push remote is specified as a URL, the fetch refspec of a
uniquely matching configured remote is now used to find and update
the remote-tracking branch (e.g., '@{push}').
* hn/url-push-tracking:
remote: find tracking branches for URL push destinations
remote: pass repository to push tracking helper
Explicitly set sh mode for ssh-agent (ssh-agent -s) to prevent
failure when user's login shell is csh-like. The failure is
caused by propagation of the $SHELL value from the user's original
shell despite the test and test harness explictly using sh, which
makes ssh-agent emit initialization code for the wrong shell:
> cd t
> echo $SHELL
/bin/tcsh
> ./t7528-signed-commit-ssh.sh --verbose --debug
[...]
expecting success of 7528.2 'sign commits using literal public keys with ssh-agent':
[...]
./t7528-signed-commit-ssh.sh: 1: eval: setenv: not found
./t7528-signed-commit-ssh.sh: 1: eval: setenv: not found
[...]
Signed-off-by: Kenneth Lorber <keni@his.com>
Acked-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A crash in the 'sparse-index' collapse code when encountering an
invalidated cache-tree node (due to an intent-to-add path) has been
fixed by avoiding collapsing such subtrees.
* ds/sparse-index-ita-crash:
sparse-index: avoid crash on intent-to-add entry outside the cone
A boundary case check in reachability bitmap traversal has been
corrected to properly handle the object at position zero, which was
previously skipped, leading to redundant bitmap loading.
* dl/pack-bitmap-position-zero:
pack-bitmap: handle objects at bitmap position zero
Junio C Hamano