The creation of the on-disk data structures for the object database
has been made pluggable, allowing future backends to customize their
setup. As part of this, the initialization of the object database
has been deferred, and the loading of the loose-object map has been
detangled from repository initialization.
* ps/odb-make-creation-pluggable:
odb: make creation of on-disk structures pluggable
odb/source: introduce function to map source type to name
setup: defer object database creation
setup: handle ODB-related environment variables in `odb_new()`
setup: detangle loading of loose object maps
loose: load loose object map for the correct source
The 'git branch' command has been taught the '--delete-merged' option
to remove local branches that are already merged into their tracked
remote-tracking branches.
* hn/branch-delete-merged:
branch: add --dry-run for --delete-merged
branch: add branch.<name>.deleteMerged opt-out
branch: add --delete-merged <pattern>
branch: prepare delete_branches for a bulk caller
branch: let delete_branches skip unmerged branches on bulk refusal
branch: convert delete_branches() to a flags argument
branch: add --forked filter for --list mode
The 'git bisect' command has been taught a
'--reset-when-found[=<where>]' option that tells the command to
automatically run 'git bisect reset' to jump back to the original
state or to the found culprit.
* hn/bisect-reset-when-found:
bisect: add --reset-when-found to leave when done
bisect: let bisect_reset() optionally check out quietly
A compatibility wrapper for writev(3p) has been reintroduced,
including fixes for CMake build and 'MAX_IO_SIZE' limits on NonStop.
Calls to write(3p) in send_sideband() and cat_blob() have been
refactored to use writev(3p) wrappers to reduce syscall overhead.
* ps/writev:
fast-import: use writev(3p) to send cat-blob responses
sideband: use writev(3p) to send pktlines
wrapper: properly handle MAX_IO_SIZE in writev(3p)
wrapper: introduce writev(3p) wrappers
compat/posix: introduce writev(3p) wrapper
The known limitations of the ref format migration in 'git refs' have
been moved to be displayed as a warning admonition directly under the
description of the 'migrate' subcommand, improving visibility. A
reference to 'git-maintenance' has also been corrected to use the
'linkgit' macro.
* kh/doc-refs-migrate-limitations:
doc: refs: linkgit to git-maintenance(1)
doc: refs: put ref migration warning under the command
'git add' has been taught a new '--resolved' option to stage
conflict-resolved paths, while leaving unrelated local changes
unstaged. It scans the unmerged paths for leftover conflict
markers and aborts if any are found.
* jc/add-resolved:
add: introduce '--resolved' option
read-cache: add remove_file_from_index_with_flags()
merge-ll: consolidate conflict marker scanning logic
read-cache: reindent
The 'ssh-agent' tests in 't7528' have been fixed to work when the
user's login shell is csh-like, by explicitly passing '-s' to
'ssh-agent' to force Bourne shell syntax.
* kl/t7528-ssh-agent-for-csh-users:
t7528: fix failure under csh
Concurrent downloads of packfiles via packfile URIs and dumb HTTP are
safer by avoiding concurrent appends to the staging file. Opening in
read-write mode with separate file offsets prevents corruption and
preserves resumability. 'fetch-pack' now tolerates pre-existing
'.keep' files.
* tn/packfile-uri-concurrency:
fetch-pack: accept "pack" output for packfile URIs
http: permit unlinking partial packs on Windows
http: avoid concurrent appends to partial packs
http: accept HTTP 416 for complete partial packs
http: avoid closing index-pack input twice
http-fetch: correct --index-pack-arg documentation
The 'TRACE2_ANCESTRY' prerequisite in the 't0213' test script has been
refined to avoid failures under user-mode emulation by verifying that
the ancestry collector reports the expected process names rather than
the emulator binary name.
* jm/t0213-skip-emulated-ancestry-tests:
t0213: skip ancestry tests under user-mode emulation
Tests for 'git merge-base --is-ancestor' have been added to cover
exit codes (0 for success, 1 for non-ancestor, 128 for errors) and
to ensure it cannot be combined with '--all'.
* ns/merge-base-is-ancestor-tests:
merge-base: add tests for --is-ancestor
'git mv' has been updated to check for a missing destination
leading directory during the checking phase, allowing 'git mv -n'
to report the failure. The error message when the rename(2)
syscall fails has also been improved to name both the source and
the destination.
* lo/mv-missing-dest-dir-check:
mv: reject a destination whose leading path is missing or a symlink
mv: name both source and destination when rename fails
Documentation for 'git replay' has been updated to refer to its
configuration variables.
* kh/doc-replay-config:
doc: replay: move “default” to the right-hand side
doc: replay: use a nested description list
doc: replay: improve config description
doc: link to config for git-replay(1)
A new test helper commit_body() has been introduced to print the
message body of a commit, and various tests have been updated to use
it instead of spelling out the command pipeline manually and losing
the exit status of the 'git cat-file' command on the upstream of the
pipe.
* sk/test-commit-body-helper:
t: use commit_body to extract commit message bodies
test-lib-functions: add commit_body helper
Synopsis and options in the documentation for 'git format-patch',
'git imap-send', 'git send-email', and 'git request-pull' have been
updated to the modern style.
* ja/doc-synopsis-style-yet-more:
doc: convert git-request-pull synopsis and options to new style
doc: convert git-send-email synopsis and options to new style
doc: convert git-format-patch synopsis and options to new style
doc: convert git-imap-send synopsis and options to new style
A crash in the 'sparse-index' collapse code when encountering an
invalidated cache-tree node (due to an intent-to-add path) has been
fixed by avoiding collapsing such subtrees.
* ds/sparse-index-ita-crash:
sparse-index: avoid crash on intent-to-add entry outside the cone
A boundary case check in reachability bitmap traversal has been
corrected to properly handle the object at position zero, which was
previously skipped, leading to redundant bitmap loading.
* dl/pack-bitmap-position-zero:
pack-bitmap: handle objects at bitmap position zero
A memory leak in 'git merge' when run without arguments (which
triggers the default-to-upstream path) has been fixed. A test has
been added to cover this case.
* tc/merge-default-to-upstream-leakfix:
merge: fix leak with merge.defaultToUpstream
'git cat-file --batch-command' that asked for 'contents' without
'type' segfaults, which has been corrected.
* jk/cat-file-batch-wo-type-fix:
cat-file: handle content request for --batch-command without type
The get_commit_action() function has been refactored to be a pure
predicate by moving the side-effecting line-level log range folding to
simplify_commit(). This ensures that evaluating a commit's action
before the walk reaches it does not prematurely mutate its tracked
line ranges, making it safer for potential lookahead evaluations.
* mm/revision-pure-get-commit-action:
revision: make get_commit_action() a pure predicate
The code path that deals with relative paths in the 'diff-lib' has
been cleaned up.
* jk/diff-relative-cached-unmerged-more:
diff-lib: skip paths outside prefix in oneway_diff()
diff-lib: drop stale comment about advancing o->pos
When answering a `cat-blob` command, `cat_blob()` issues three separate
calls to write(3p) on the cat-blob fd: one for the header line, one for
the full blob payload, and one for the trailing newline. Frontends like
git-filter-repo issue these commands in bulk, once per rewritten blob,
so the syscall overhead adds up.
Use `writev_in_full()` to send all three parts with a single syscall.
This can be benchmarked with the following setup:
$ git cat-file --unordered --filter=object:type=blob
--batch-check='cat-blob %(objectname)' --batch-all-objects >request
$ git fast-import --cat-blob-fd=3 <request
Executing this with 100,000 objects in linux.git:
Benchmark 1: HEAD~
Time (mean ± σ): 1.320 s ± 0.003 s [User: 1.154 s, System: 0.161 s]
Range (min … max): 1.314 s … 1.324 s 10 runs
Benchmark 2: HEAD
Time (mean ± σ): 1.270 s ± 0.022 s [User: 1.133 s, System: 0.132 s]
Range (min … max): 1.209 s … 1.282 s 10 runs
Summary
HEAD ran
1.04 ± 0.02 times faster than HEAD~
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Every pktline that we send out via `send_sideband()` currently requires
two syscalls: one to write the pktline's length, and one to send its
data. This typically isn't all that much of a problem, but under extreme
load the syscalls may cause contention in the kernel.
Refactor the code to instead use the newly introduced writev(3p) infra
so that we can send out the data with a single syscall. This reduces the
number of syscalls from around 133,000 calls to write(3p) to around
67,000 calls to writev(3p).
This change leads to a performance improvement for git-upload-pack(1),
but we have to cheat a bit to really make it measurable. Usually, the
time is strongly dominated by generating the packfile itself. But if we
precompute the pack and serve it via the pack-objects hook then we can
essentially eliminate that overhead. The following setup is executed in
the Git repository:
$ cat >request <<-EOF
0048want 5ce91c059e side-band no-progress
00000009done
EOF
$ echo 5ce91c059e | git pack-objects --revs --stdout >pack
$ cat >hook <<-EOF
#!/bin/sh
cat >/dev/null
cat "$(pwd)"/pack
EOF
$ chmod u+x hook
$ git -c uploadpack.packObjectsHook="$(pwd)"/hook upload-pack . <request
Benchmarking the last command leads to the following results:
Benchmark 1: HEAD~
Time (mean ± σ): 192.9 ms ± 0.6 ms [User: 106.5 ms, System: 95.3 ms]
Range (min … max): 191.7 ms … 194.1 ms 50 runs
Benchmark 2: HEAD
Time (mean ± σ): 141.1 ms ± 0.7 ms [User: 63.2 ms, System: 86.6 ms]
Range (min … max): 139.8 ms … 142.7 ms 50 runs
Summary
HEAD ran
1.37 ± 0.01 times faster than HEAD~
This might not be impressive in absolute numbers when you also take into
account the time it takes to generate the packfile itself. But GitLab
(and supposedly other forges) have caching mechanisms in place that work
exactly like the above setup, where repeated incoming requests can be
served from the same cached packfile. And in those cases, the impact is
sizeable.
More importantly though, as hinted at above, GitLab has observed in the
past that with enough cache hits we eventually start to saturate a
semaphore in the Linux kernel itself in the pipe write path. This
bottleneck is being moved a bit by having to do less syscalls.
Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some systems like NonStop set a comparatively small `MAX_IO_SIZE`, which
limits the maximum number of bytes we're allowed to write in a single
call. We already handle this limit properly in `xwrite()`, but we have
recently introduced wrappers for writev(3p) where we don't. This will
cause the syscall to return EINVAL in case somebody passes an iovec
entry to writev(3p) that is larger than `MAX_IO_SIZE`.
Introduce a new function `xwritev()` that is similar to `xwrite()` in
that it handles such platform-specific nuances:
- We only pass the leading iovec entries to writev(3p) that fit into
`MAX_IO_SIZE`, pretending that the underlying syscall performed a
short write. This mirrors how `xwrite()` chomps overly large
requests before handing them to write(3p). As a consequence, callers
will never see writev(3p)'s EINVAL error for requests whose summed
length would overflow an ssize_t, but observe a short write instead.
- If already the first iovec entry exceeds the limit we instead punt
to `xwrite()`, which knows to handle this case for us.
- We restart the underlying syscall on EINTR and EAGAIN, just like
`xwrite()` does for write(3p).
Adapt `writev_in_full()` to use this new wrapper. With the retry logic
now living in `xwritev()`, the calling loop becomes the exact mirror
image of `write_in_full()`, which also retains the responsibility of
translating a zero-length write into ENOSPC.
Reported-by: Randall Becker <randall.becker@nexbridge.ca>
Helped-by: Jeff King <peff@peff.net>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the preceding commit we have added a compatibility wrapper for the
writev(3p) syscall. Introduce some generic wrappers for this function
that we nowadays take for granted in the Git codebase.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In a subsequent commit we're going to add the first caller to
writev(3p). Introduce a compatibility wrapper for this syscall that we
can use on systems that don't have this syscall.
The syscall exists on modern Unixes like Linux and macOS, and seemingly
even for NonStop according to [1]. It doesn't seem to exist on Windows
though.
[1]: http://nonstoptools.com/manuals/OSS-SystemCalls.pdf
[2]: https://www.gnu.org/software/gnulib/manual/html_node/writev.html
Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.
Make the creation of on-disk structures pluggable to accommodate for
this.
Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.
Lift the call to `odb_new()` out of `apply_repository_format()` so that
callers have more wiggle room with when exactly they call it, and adapt
them accordingly. The only exception is `init_db()`, where we now defer
creating the object database until we call `create_object_database()`.
With this change, initializing and creating the object database on disk
is now neatly encapsulated in a single function, which will make it
easier for a subsequent commit to move creation of the on-disk data
structures into the `struct odb_source` backends.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When initializing a repository's object database we have to respect the
GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment
variables, which can be set by the user to override the default location
of where we write objects to and read objects from.
This is handled in `apply_repository_format()`, which is fine. But in a
subsequent commit we'll have to defer constructing the object database
to a later point in some cases, and that will require a second site
where we call `odb_new()`. And of course, that second site would have to
handle those environment variables, as well.
It would be somewhat awkward to duplicate the logic though. But there's
a better alternative: instead of handling this logic in "setup.c", we
can easily handle environment variables in `odb_new()` itself. This
ensures that object database creation is neatly self-contained, and we
don't have to duplicate any of the logic.
Another benefit is that in a future patch series we plan to move
handling of alternates into the backends themselves [1], and that will
require us to also handle those environment variables in the "files"
backend itself. So moving the logic into the ODB level already gets us
one step closer to that goal.
Refactor the logic accordingly.
[1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.
Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:
- We assume that the primary object database is the "files" object
database and unconditionally downcast it. This will cause us to BUG
in case a different object database type was used together with a
compat hash algorithm.
- We require the object database to already have been initialized when
configuring the object database. This means that we must intermix
configuration of the repository and initialization of its
sub-structures in a weird way.
Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source. The consequence is that we'll ignore loose object maps of
any configured alternates.
Fix this bug by instead loading the map via the loose source's path.
Helped-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
I have to scroll down at least three screens in man(1) from the
`migrate` description in order to see the “known limitations” for
it. This is important information since the text says that concurrent
writes can lead to an inconsistent migrated state. Let’s move that text
up to the command description and put it inside a Warning admonition.
This section made sense when it was added in 25a0023f (builtin/refs:
new command to migrate ref storage formats, 2024-06-06); `migrate` was
the only subcommand, and this section was visible from the command
description. A one-page man page. But that is not the case anymore
now that the command has nine subcommands to describe.
Acked-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git for Windows has been updated to avoid auto-detecting the symlink
type if the target path starts with a slash, preventing NTLM
credential leaks when checking out repositories with crafted
symbolic links pointing to network shares.
* js/mingw-symlink-net-share-leak:
mingw: skip symlink type auto-detection for network share targets
The alias tests in 't/t0014-alias.sh' have been updated to dynamically
query the list of deprecated commands using 'git
--list-cmds=deprecated' to avoid test failures when running with
'WITH_BREAKING_CHANGES' in a build directory that contains stale
executables of formerly deprecated commands.
* jk/t0014-dynamic-deprecated-cmds:
t0014: generate deprecated command names dynamically
t0014: factor out choice of deprecated commands
The image version used by the static-analysis CI job has been bumped
to ubuntu-latest (Ubuntu 24.04), which brings in a newer Coccinelle
version that resolves a severe performance regression. A false
positive warning from the 'CHECK_ASSERTION_SIDE_EFFECTS' build with
GCC 15 in the Bloom filter code has also been silenced to facilitate
the image upgrade.
* jk/ci-static-analysis-image-bump:
ci: bump ubuntu image version for static-analysis job
bloom: silence CHECK_ASSERTION_SIDE_EFFECTS false positive
'git branch -d' has been taught to report when a branch cannot be
deleted because it is being used in an active bisect run.
* rs/branch-delete-bisect-warning:
branch: report active bisect run when rejecting delete
Object database housekeeping in 'git gc' and 'git maintenance' has
been refactored to be pluggable. The files-backend-specific logic,
including incremental and geometric repacking as well as object
pruning, has been moved out of the command implementation and into the
files object database source, enabling future alternative object
database backends to implement their own housekeeping services.
* ps/odb-pluggable-housekeeping:
odb: make optimizations pluggable
builtin/gc: fix signedness issues in ODB-related functionality
builtin/gc: refactor ODB optimizations to operate on "files" source
builtin/gc: introduce `odb_optimize_required()`
builtin/gc: move geometric repacking into `odb_optimize()`
builtin/gc: introduce object database optimization options
builtin/gc: inline config values specific to the "files" backend
builtin/gc: make repack arguments self-contained
builtin/gc: extract object database optimizations into separate function
builtin/gc: move worktree and rerere tasks before object optimizations
odb: run "pre-auto-gc" hook for all maintenance tasks
t7900: simplify how we check for maintenance tasks
Two bugs in how 'git rebase' handles skipped 'fixup' and 'squash'
commands have been fixed. One bug caused an incorrect commit count to
be shown in the template message when multiple commands were skipped,
and another prevented the editor from opening when the final command
in a chain containing 'fixup -c' was skipped.
* pw/rebase-fixup-fixes:
rebase: remember fixup -c after skipping fixup/squash
rebase -i: fix counting of fixups after rebase --skip
'git diff --relative' running with '--cached' has been corrected to
avoid a segfault when encountering unmerged paths outside the
prefix.
* jk/diff-relative-cached-unmerged:
diff-lib: add idx/tree sanity check to oneway_diff
diff: ignore unmerged paths outside prefix with --relative --cached
"git branch --dry-run --delete-merged ..." prints one line per ref that
would be deleted without modifying refs or branch configuration.
--dry-run is only meaningful together with --delete-merged and is
rejected otherwise.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Setting branch.<name>.deleteMerged=false exempts that branch from
"git branch --delete-merged", which is useful for a topic you want
to keep developing after an early round of it has been merged
upstream. Unless --quiet is given, each skip is reported so the
user knows why their topic was kept.
Explicit deletion with "git branch -d" still uses the normal merge
check and ignores this setting.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git branch (--delete-merged <pattern>)... [<branch-pattern>...]
deletes local branches matching the optional branch patterns when their
configured upstream matches one of the --delete-merged arguments and
their tip is reachable from that upstream. The work has already landed
on the upstream they track, so the local copy is no longer needed.
Each <pattern> may name a ref, a remote, or a shell glob. The option can
be repeated to widen the upstream match. Keeping the candidate patterns
as positional arguments lets users bound the set of local branches that
may be deleted independently of the upstream selection.
A branch is not deleted when:
* it is checked out in any worktree
* its configured upstream ref no longer exists, since a missing
upstream is not by itself a sign of integration
* pushing it to the remote configured by branch.<name>.remote would
update its upstream, as determined by that remote's configured push
and fetch refspecs. For example, a local "main" that tracks
"origin/main" is kept even when remote.pushDefault names a fork.
Right after a pull it merely looks fully merged.
* it is the local upstream of a branch that is not being deleted, so
no branch is deleted out from under stacked work.
A branch whose work is not yet merged into its upstream is silently
skipped, so one unmerged topic does not abort the whole sweep.
Collect protected local upstreams without changing the candidate set
during ref iteration, then remove them after iteration. This makes the
result independent of ref iteration order. If a protected branch's own
upstream is deleted by the same sweep, clear its upstream configuration.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach delete_branches() a new mode for the upcoming --delete-merged
caller that checks whether a branch is merged into its upstream without
falling back to HEAD when there is no upstream. Existing callers keep
their current behavior.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano