The reftable code has been optimized to avoid an unnecessary
stat/reload of the stack when an addition already holds the
list_file lock, reducing the number of newfstatat syscalls from
linear to constant when writing refs.
* kn/reftable-optimize-reloading:
reftable/stack: avoid reloading the stack when already locked
reftable/stack: move list lock to `struct reftable_stack`
reftable/stack: rename reftable_stack_new_addition()
reftable/stack: remove `REFTABLE_STACK_NEW_ADDITION_RELOAD`
A heap-use-after-free bug in the object name parsing code when
reporting failures with a relative path to a sparse directory has
been corrected.
* sk/object-name-use-after-free:
object-name: avoid use-after-free in get_oid_with_context_1()
The documentation for 'git format-rev' has been updated to use the
[synopsis] block definition on code blocks to properly highlight
placeholders, and a quoting inconsistency in the running text has
been fixed.
* kh/format-rev-doc-synopsis:
doc: format-rev: use [synopsis] on code block
doc: format-rev: quote subject placeholder before and after
The global variable 'fetch_if_missing' has been moved to a member in
'struct repository', continuing the libification process and
allowing per-repository control (such as for submodules).
* ty/repository-fetch-if-missing:
repository: move fetch_if_missing into struct repository
'git -C <dir> checkout fi<TAB>' did not complete, which has been
corrected.
* jc/complete-checkout:
completion: 'git checkout' completes untracked paths as a last resort
completion: complete tracked paths for "git checkout"
completion: no-op refactoring of checkout completion
'git -C <dir> diff fi<TAB>' did not complete 'file', which has been
corrected.
* jc/complete-diff-tracked-paths:
completion: 'git diff' completes untracked paths as a last resort
completion: complete tracked paths for 'git diff'
completion: no-op refactoring of diff completion
The unused name parameter in 'struct chdir_notify_entry' has been
removed from chdir_notify_register(), chdir_notify_unregister(), and
related callback signatures across several subsystems, simplifying the
API now that trace output no longer uses it.
* ch/chdir-notify-drop-name:
chdir-notify.h: Removed unused param 'name'
The performance of adding numerous new packfiles has been improved
by introducing a fast path for known-new packfiles to skip an
unnecessary traversal in packfile_list_append(), avoiding a
quadratic complexity regression on load.
* js/packfile-fast-append:
packfile: fix perf regression with many packs
'git repack' has been taught '--drop-filtered' to delete local
promisor blobs exceeding a limit (currently 'blob:limit=') in partial
clones, reclaiming space. Guards prevent running during other
operations or if referenced by the index.
* ss/repack-drop-filtered:
builtin/repack: add guards for --drop-filtered
builtin/repack: actually drop filtered promisor blobs
builtin/repack: enumerate promisor blobs for --drop-filtered
repack-promisor: allow excluding objects from the rebuilt promisor pack
list-objects-filter: add list_objects_filter__filter_oidset()
builtin/repack: add --drop-filtered and --dry-run options
The pack-objects command has been updated to record the total bytes
written to pack files in trace2 output, allowing performance
analysis of different compression settings by comparing the
resulting pack sizes.
* fr/pack-objects-trace-pack-bytes:
pack-objects: trace pack bytes written
The mechanism to generate a packfile corresponding to the result of
a fetch/push has been made pluggable through a set of object
database callback functions, removing hardcoded references to
'pack-objects' and enabling alternative ODBs to serve packfiles
themselves.
* ps/odb-pluggable-pack-generation:
bundle: generate packfiles via the object database
bundle: get (mostly) rid of `the_repository`
builtin/bundle: refactor option handling for progress meter
send-pack: generate packfiles via the object database
upload-pack: generate packfiles via the object database
odb: introduce interface to generate packfiles
The 'git receive-pack' command has been updated to use a new ODB
transaction interface for writing incoming packfiles, making it more
backend-agnostic.
* jt/receive-pack-pluggable-writes:
odb/transaction: add transaction interface to write packfiles
odb: return temporary ODB source when set
builtin/receive-pack: explicitly pass packfile fd
builtin/receive-pack: report unpack errors via strbuf
builtin/receive-pack: lift global state out of unpack()
builtin/receive-pack: read unpack limit config lazily
builtin/receive-pack: pass shallow file explicitly
odb/transaction: add transaction finalize interface
builtin/receive-pack: properly clean up keep files
The threshold for geometric repacking to trigger based on loose
object count has been adjusted to match that of 'git gc --auto',
preventing over-aggressive repacking during concurrent writes.
* ps/odb-geometric-repack-loose-threshold:
odb/files: be less aggressive with geometric repacking
The trailer parsing machinery has been updated to avoid mistaking
lines that begin with a URL (e.g., 'https://...') as trailer lines.
This prevents intended textual URLs from being mangled or mistakenly
treated as metadata keys.
* kh/trailers-no-urls:
trailers: stop recognizing URLs as trailers
The object database layer has been simplified by eagerly loading
alternate object directories upon initialization, instead of
deferring it to the first object lookup. This eliminates the need
for scattered lazy-loading calls throughout the codebase and paves
the way for integrating alternates with the pluggable backends.
* ps/odb-eagerly-load-alternates:
odb: drop `alternates_db` field
odb: drop `loaded_alternates` field
odb: eagerly initialize alternates
odb: decouple source path comparisons from `the_repository`
setup: create ref and object databases after config is written
The command line completion (in contrib/) has been taught to handle
the experimental 'git history' command.
* vm/complete-history:
completion: complete 'git history split' pathspecs
completion: complete 'git history --update-refs' values
completion: complete 'git history --empty' values
completion: add 'git history' subcommands
Various tests in 't7900-maintenance.sh' have been updated to use a
throwaway repository, and auto-detaching of maintenance tasks is now
disabled for these tests to fix flaky races with concurrent background
maintenance jobs.
* ps/t7900-deflake-maintenance:
t7900: fix flaky "maintenance.strategy" test
t7900: adapt some tests to use a throwaway repository
A client requesting the promisor-remote capability without a value
caused a null pointer dereference, which has been corrected by
rejecting a request without an argument.
* en/serve-promisor-remote-fix:
serve: reject valueless promisor-remote capability
The 'pack-objects' and delta-encoding code paths have been updated to
use 'size_t' instead of 'unsigned long' for object sizes and offset
limits, avoiding potential truncation issues on 64-bit Windows.
* js/pack-objects-delta-size-t:
packfile: widen `unpack_object_header_buffer()` to `size_t`
git-zlib: widen `git_deflate_bound()` to `size_t`
t/helper/test-pack-deltas: widen `do_compress()`'s maxsize local to `size_t`
http-push: widen `start_put()`'s size local from `ssize_t` to `size_t`
diff: widen `deflate_it()`'s bound local from int to `size_t`
archive-zip: widen `zlib_deflate_raw()`'s maxsize local to `size_t`
packfile, git-zlib: widen `use_pack()` and zstream avail fields to `size_t`
delta: widen `create_delta()` and `diff_delta()` to `size_t`
pack-objects: widen `mem_usage` and `try_delta()`'s out-param to `size_t`
pack-objects: widen `free_unpacked()` return to `size_t`
pack-objects: widen delta-cache accounting to `size_t`
delta: widen `create_delta_index()` parameter to `size_t`
diff-delta: widen `struct delta_index`' size fields to `size_t`
The '--shallow-file' option of 'git' command requires a value, but the
code did not check the presence of a value and instead segfaulted
without one, which has been corrected.
* cc/git-shallow-file-wo-value:
git: avoid segfault on "git --shallow-file" without a value
The setting of a now-unused member '.pretty_given' in the sequencer
machinery has been removed.
* en/sequencer-lose-pretty-given:
sequencer: remove unnecessary variable setting
A handful of code paths have been corrected to check return values
from functions like curl_easy_duphandle(), deflateInit(), lseek(),
dup(), and strbuf_getline_lf(), resolving several Coverity warnings
about unchecked returns.
* js/coverity-unchecked-returns-fix:
bisect: handle dup() failure when redirecting stdout
bisect: check get_terms return at all call sites
bisect: check strbuf_getline_lf return when reading terms
transport-helper: warn when export-marks file cannot be finalized
transport-helper: check dup() return in get_exporter
compat/pread: check initial lseek for errors
last-modified: handle repo_parse_commit() failures
reftable tests: check reftable_table_init_ref_iterator() return
reftable/block: check deflateInit() return value
reftable: handle block-writer initialization errors
config: propagate launch_editor() failure in show_editor()
http: die on curl_easy_duphandle failure in get_active_slot
When making modifications to the reftable stack, the stack obtains a
lock to the list file and removes the lock after the commit phase. Since
most operations reload the stack to ensure we have the latest state, any
branched operation during the locked phase could trigger a state reload.
To prevent data loss due to concurrent writes, state reload is necessary
right after obtaining the lock. But any reloads after that are just a
no-op. Now that the struct has access to the lock file status, simply
skip reloading if the lock is present.
Benchmarking with a fixed, non-symbolic target OID in the 'refs/tags/'
namespace (since it triggers a stack reload when checking if reflog
exists for the given tag name), shows a consistent 15-20% improvement
with these patches:
refcount master patch speedup
-------- ------- ------- -------
2,000 18.5 ms 16.6 ms 1.11x
20,000 120.7 ms 102.8 ms 1.17x
50,000 296.5 ms 247.1 ms 1.20x
We can also see the improvements in the number of syscall counts. On
master, the number of calls to `newfstatat()` grows linearly with the
number of refs created. With this patch, the number is now a constant:
refcount master patch
-------- ------ ------
1,000 1,059 55
5,000 5,059 55
10,000 10,059 55
20,000 20,059 55
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The struct `reftable_addition` is used to modify a given stack, as such,
it also includes a `struct reftable_flock` used to obtain the lock to
the list file. While the scope of the field lies within this struct, it
doesn't allow for optimizations to be made on `struct reftable_stack`
itself.
Move the field to `struct reftable_stack`, allowing us to make a simple
optimization around avoiding a stack reload when we have already
obtained a lock. While this is currently possible in the write path, the
write path also contains multiple branches to reads which only work
on top of `struct reftable_stack`, and we would miss the optimization in
such paths.
Since the lock is now shared across all additions on the same stack, a
second `reftable_addition` that fails to acquire the already held lock
would still call `reftable_addition_close()`, which will release the
`stack->list_lock` which is still held by the first addition. To avoid
this, add a new bit field `locked` to `reftable_addition` that tracks
whether a particular addition is the one holding the lock, and only
release it in that case. Add a unit test to validate this behavior.
While here, remove an unused header file from 'reftable/stack.h'.
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rename the function `reftable_stack_new_addition()` to
`reftable_stack_addition_new()` to be more inline with our naming
scheme.
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 80e7342ea8 (reftable/stack: allow locking of outdated stacks,
2024-09-24), the `REFTABLE_STACK_NEW_ADDITION_RELOAD` was introduced so
that callers of `reftable_stack_init_addition()` can also reload the
stack if there was a concurrent update made before the lock was
obtained.
Then 16684b6fae (refs/reftable: always reload stacks when creating
lock, 2025-08-12) updated all of the remaining call-sites to propagate
this flag to ensure that we always reload the stack whenever there was a
concurrent update.
As all calls to `reftable_stack_init_addition()` inevitably propagate
the flag, it is safe to remove the flag and its associated code and make
the reloading of the stack the default flow. This makes it easier to
follow the flow and simplifies the logic.
The only exceptions are:
1. Unit tests, where we explicitly do not propagate the flag. These
tests are now modified with the new status quo.
2. `reftable_stack_clean()`, which was propagating 0 to
`reftable_stack_new_addition()` but was then manually reloading the
stack after. Here the new flow will achieve the same, while also
allowing us to remove the manual reload.
This also makes two checks for 'REFTABLE_OUTDATED_ERROR' redundant, so
remove them also.
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The object database (odb) API has been refactored to distinguish
between missing objects and corrupt ones by returning more
descriptive error statuses. Both the packed and loose backends now
faithfully propagate error details using a generic strbuf error
mechanism, removing backend-specific leakage from central lookup
paths.
* ps/odb-generic-corrupt-objects:
odb: handle `OBJECT_INFO_DIE_IF_CORRUPT` generically
odb/source: allow `read_object_info()` to bubble up error messages
odb/source: let callers discern missing and corrupt objects
odb/source: introduce error status when reading objects
odb/source-packed: flag known-bad objects as corrupt and not missing
The DWIM logic in 'git worktree add' sometimes tried to infer a
remote-tracking branch when an explicit '-b' or '-B' option was
given to create a new branch, causing the explicit branch name to
be ignored, which has been corrected.
* yn/worktree-add-no-dwim-with-b:
worktree add: shouldn't dwim if -b or -B is given
The merge-base computation has been optimized by stopping the walk
early when one side's exclusive commits in the queue are exhausted,
yielding significant speedups for queries with one-sided histories.
* kk/merge-base-exhaustion:
commit-reach: remove commit-date ordering fallback
commit-reach: move min_generation check into paint_queue_get()
commit-reach: terminate merge-base walk when one paint side is exhausted
commit-reach: introduce struct paint_state with per-side counters
t6600: add clock-skew topologies and step counts for edge cases
commit-reach: add trace2 instrumentation to paint_down_to_common()
t6099: add side-exhaustion regression test
t6600: add test cases for side-exhaustion edge cases
test-lib-functions: improve diagnostic output for trace2 data assertions
Documentation/technical: add paint-down-to-common doc
The sequencer has been updated to release the object database before
spawning 'git commit'. This prevents open file handles from
blocking auto-maintenance tasks, such as repacking, on systems like
Windows where open files cannot be easily unlinked.
* js/sequencer-release-odb-before-commit:
sequencer: release the ODB before spawning git commit
The error message given by 'git send-email' when a message file is
missing a 'Subject:' header has been clarified, and the error string
is now terminated with a newline so that Perl avoids appending its
internal source location data.
* hn/send-email-missing-subject-error:
send-email: clarify missing subject error
The 'struct odb_read_stream' and 'struct odb_write_stream'
structures have been consolidated into a single unified 'struct
odb_stream' structure, simplifying object database streaming APIs
and enabling streaming of arbitrary object types.
* ps/odb-streams:
odb/streaming: unify function names to create new streams
odb/streaming: rename `struct input_zstream_data`
odb/streaming: rename `struct read_object_fd_data`
odb/streaming: consolidate read and write streams
odb/streaming: rename `struct odb_read_stream`
odb/streaming: support streaming arbitrary object types
odb/streaming: drop `is_finished` field
odb/streaming: track write stream size in the structure
The usage string of 'git fast-import' has been updated to use the
parse_options() API for displaying help, and its SYNOPSIS in the
documentation has been standardized to match.
* cc/fast-import-usage:
fast-import: remove useless from_stream argument
fast-import: use parse_options() for command line options
fast-import: use callbacks to parse some options
fast-import: use struct option for usage string
fast-import: move command state globals into 'struct fast_import_state'
fast-import: introduce 'struct fast_import_state'
fast-import: factor out option_*() functions
fast-import: use int for some bool flags
fast-import: localize 'i' into the 'for' loops using it
api-parse-options.adoc: document hidden and OPT_*_F option macros
api-parse-options.adoc: document per-option flags
parse-options: introduce OPT_HIDDEN_GROUP
The 'remote-object-info' command for 'git cat-file --batch-command'
has been extended to support the '%(objecttype)' placeholder.
* ps/cat-file-remote-object-info-type:
cat-file: unify default format
serve: advertise type capability
fetch-object-info: parse type from server response
protocol-caps: add type support to object-info
transport: drop remote object-info fields from transport struct
fetch-object-info: die() on the remaining error path
fetch-object-info: use dedicated struct for the results
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: detect malformed server responses
t5701: use test_file_size() to get the size of a file
When performing auto-maintenance with geometric repacking we have two
conditions that may trigger a repack:
- Either the geometric sequence of packfiles is invalidated.
- Or we have too many loose objects.
The first condition shouldn't trigger all that often: it may be hit when
we fetch a new packfile, but users tend to not do that all the time. The
second condition is what typically triggers more regularly though, as
every command that ends up writing new objects may cause us to cross the
threshold of loose objects. It is thus preferable to not be too
aggressive here, as otherwise we may end up repacking objects quite
often.
For the geometric-repacking strategy though we have a default of 100
objects, only. As we're approximating the count of objects by only
reading the "objects/17/" shared, we'd only need 2 objects in there
before we perform a repack by default, which is quite aggressive.
git-gc(1) on the other hand has a default of 6700, so it is quite a bit
more conservative here.
Being this aggressive is also causing problems as reported by our users.
When running lots of concurrent writers, those writes will constantly
end up spawning maintenance jobs that end up repacking objects. As we
also prune objects, a concurrently running process that tries to write
an object may see that the sharding directories get removed under their
feet. While we try re-creating such leading directories, we only do so a
single time, and it may happen that the directory vanishes again before
we had the chance to create the loose object. This is not a new problem,
but it is exacerbated by us running maintenance this aggressively.
Improve the status quo by reducing the frequency at which we pack loose
objects to the same frequency that git-gc(1) uses.
Reported-by: Stefan Haller <lists@haller-berlin.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-bundle(1) spawns git-pack-objects(1) directly to generate the pack
data that gets appended to the bundle header. While bundles are not
part of the wire protocol, they are a transfer mechanism for packs all
the same, so convert them to use the pack generation interface of the
object database as well.
This makes the pack generator the single spawn point for all pack
streams that leave the repository, leaving only local maintenance tasks
like git-repack(1) with direct knowledge of git-pack-objects(1).
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Refactor "bundle.c" so that we don't depend on `the_repository` anymore.
This conversion is trivial for most of the part, as we already have a
repository available in all calling conexts.
The only exception is that we use `get_log_output_encoding()`, which
implicitly depends on `the_repository`. Add an `extern` declaration for
this function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not
accidentally introduce more uses of `the_repository`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The git-bundle(1) command has a couple of command line options that
relate to whether or not progress should be reported. These options
match the options that git-pack-objects(1) expects, and consequently
they mostly get passed through to it directly.
This results in somewhat of a confusing interface: there are four
different options that relate to whether or not progress should be
displayed and how verbose it should be. But in reality, there's really
only two modes:
- "--progress" and "--all-progress" result in the same outcome, which
is also documented as such.
- "--all-progress-implied" does nothing as we pass that argument to
git-pack-objects(1) unconditionally anyway.
So in the end, the options only control whether or not progress should
be displayed at all, nothing else.
Refactor the interface to instead use a simple `progress` boolean. This
makes argument handling a lot more straight-forward and it prepares us
for the next commit, where we're migrating git-bundle(1) to the generic
interface for generating a packfile.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano