The object database layer has been simplified by eagerly loading
alternate object directories upon initialization, instead of
deferring it to the first object lookup. This eliminates the need
for scattered lazy-loading calls throughout the codebase and paves
the way for integrating alternates with the pluggable backends.
* ps/odb-eagerly-load-alternates:
odb: drop `alternates_db` field
odb: drop `loaded_alternates` field
odb: eagerly initialize alternates
odb: decouple source path comparisons from `the_repository`
setup: create ref and object databases after config is written
The command line completion (in contrib/) has been taught to handle
the experimental 'git history' command.
* vm/complete-history:
completion: complete 'git history split' pathspecs
completion: complete 'git history --update-refs' values
completion: complete 'git history --empty' values
completion: add 'git history' subcommands
The object database (odb) API has been refactored to distinguish
between missing objects and corrupt ones by returning more
descriptive error statuses. Both the packed and loose backends now
faithfully propagate error details using a generic strbuf error
mechanism, removing backend-specific leakage from central lookup
paths.
* ps/odb-generic-corrupt-objects:
odb: handle `OBJECT_INFO_DIE_IF_CORRUPT` generically
odb/source: allow `read_object_info()` to bubble up error messages
odb/source: let callers discern missing and corrupt objects
odb/source: introduce error status when reading objects
odb/source-packed: flag known-bad objects as corrupt and not missing
The DWIM logic in 'git worktree add' sometimes tried to infer a
remote-tracking branch when an explicit '-b' or '-B' option was
given to create a new branch, causing the explicit branch name to
be ignored, which has been corrected.
* yn/worktree-add-no-dwim-with-b:
worktree add: shouldn't dwim if -b or -B is given
The merge-base computation has been optimized by stopping the walk
early when one side's exclusive commits in the queue are exhausted,
yielding significant speedups for queries with one-sided histories.
* kk/merge-base-exhaustion:
commit-reach: remove commit-date ordering fallback
commit-reach: move min_generation check into paint_queue_get()
commit-reach: terminate merge-base walk when one paint side is exhausted
commit-reach: introduce struct paint_state with per-side counters
t6600: add clock-skew topologies and step counts for edge cases
commit-reach: add trace2 instrumentation to paint_down_to_common()
t6099: add side-exhaustion regression test
t6600: add test cases for side-exhaustion edge cases
test-lib-functions: improve diagnostic output for trace2 data assertions
Documentation/technical: add paint-down-to-common doc
The sequencer has been updated to release the object database before
spawning 'git commit'. This prevents open file handles from
blocking auto-maintenance tasks, such as repacking, on systems like
Windows where open files cannot be easily unlinked.
* js/sequencer-release-odb-before-commit:
sequencer: release the ODB before spawning git commit
The error message given by 'git send-email' when a message file is
missing a 'Subject:' header has been clarified, and the error string
is now terminated with a newline so that Perl avoids appending its
internal source location data.
* hn/send-email-missing-subject-error:
send-email: clarify missing subject error
The 'struct odb_read_stream' and 'struct odb_write_stream'
structures have been consolidated into a single unified 'struct
odb_stream' structure, simplifying object database streaming APIs
and enabling streaming of arbitrary object types.
* ps/odb-streams:
odb/streaming: unify function names to create new streams
odb/streaming: rename `struct input_zstream_data`
odb/streaming: rename `struct read_object_fd_data`
odb/streaming: consolidate read and write streams
odb/streaming: rename `struct odb_read_stream`
odb/streaming: support streaming arbitrary object types
odb/streaming: drop `is_finished` field
odb/streaming: track write stream size in the structure
The usage string of 'git fast-import' has been updated to use the
parse_options() API for displaying help, and its SYNOPSIS in the
documentation has been standardized to match.
* cc/fast-import-usage:
fast-import: remove useless from_stream argument
fast-import: use parse_options() for command line options
fast-import: use callbacks to parse some options
fast-import: use struct option for usage string
fast-import: move command state globals into 'struct fast_import_state'
fast-import: introduce 'struct fast_import_state'
fast-import: factor out option_*() functions
fast-import: use int for some bool flags
fast-import: localize 'i' into the 'for' loops using it
api-parse-options.adoc: document hidden and OPT_*_F option macros
api-parse-options.adoc: document per-option flags
parse-options: introduce OPT_HIDDEN_GROUP
The 'remote-object-info' command for 'git cat-file --batch-command'
has been extended to support the '%(objecttype)' placeholder.
* ps/cat-file-remote-object-info-type:
cat-file: unify default format
serve: advertise type capability
fetch-object-info: parse type from server response
protocol-caps: add type support to object-info
transport: drop remote object-info fields from transport struct
fetch-object-info: die() on the remaining error path
fetch-object-info: use dedicated struct for the results
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: detect malformed server responses
t5701: use test_file_size() to get the size of a file
'git worktree add <path> <branch>' DWIMs <branch> to a
remote-tracking branch when neither -b, -B, nor --detach
is given.
However, 'git worktree add -b <new-branch> <path> <branch>' can
still DWIM <branch>, causing <new-branch> to be ignored. This is a
regression introduced by 128e5496b3 (worktree add: extend DWIM to
infer --orphan, 2023-05-17), which appeared in Git 2.42.
Signed-off-by: Yoichi NAKAYAMA <yoichi.nakayama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A heap-use-after-free bug in the object name parsing code when
reporting failures with a relative path to a sparse directory has
been corrected.
* sk/object-name-use-after-free:
object-name: avoid use-after-free in get_oid_with_context_1()
The documentation for 'git format-rev' has been updated to use the
[synopsis] block definition on code blocks to properly highlight
placeholders, and a quoting inconsistency in the running text has
been fixed.
* kh/format-rev-doc-synopsis:
doc: format-rev: use [synopsis] on code block
doc: format-rev: quote subject placeholder before and after
Documentation for 'git interpret-trailers' has been updated to explain
the format of trailer keys (alphanumeric characters and hyphens),
replace outdated terminology, define key terms upfront, and document
how comment lines in the input are treated.
* kh/doc-trailers:
doc: interpret-trailers: document comment line treatment
doc: interpret-trailers: rewrite new-trailers paragraphs
doc: interpret-trailers: commit to “trailer block” term
doc: interpret-trailers: join new-trailers again
doc: interpret-trailers: add key format example
doc: interpret-trailers: explain key format
doc: interpret-trailers: explain the format after the intro
doc: interpret-trailers: not just for commit messages
doc: interpret-trailers: use “metadata” in Name as well
doc: interpret-trailers: replace “lines” with “metadata”
doc: interpret-trailers: stop fixating on RFC 822
The creation of the on-disk data structures for the object database
has been made pluggable, allowing future backends to customize their
setup. As part of this, the initialization of the object database
has been deferred, and the loading of the loose-object map has been
detangled from repository initialization.
* ps/odb-make-creation-pluggable:
odb: make creation of on-disk structures pluggable
odb/source: introduce function to map source type to name
setup: defer object database creation
setup: handle ODB-related environment variables in `odb_new()`
setup: detangle loading of loose object maps
loose: load loose object map for the correct source
The 'git branch' command has been taught the '--delete-merged' option
to remove local branches that are already merged into their tracked
remote-tracking branches.
* hn/branch-delete-merged:
branch: add --dry-run for --delete-merged
branch: add branch.<name>.deleteMerged opt-out
branch: add --delete-merged <pattern>
branch: prepare delete_branches for a bulk caller
branch: let delete_branches skip unmerged branches on bulk refusal
branch: convert delete_branches() to a flags argument
branch: add --forked filter for --list mode
* jc/complete-checkout:
completion: 'git checkout' completes untracked paths as a last resort
completion: complete tracked paths for "git checkout"
completion: no-op refactoring of checkout completion
'git -C <dir> diff fi<TAB>' did not complete 'dir/file', which has
been corrected.
* jc/complete-diff-tracked-paths:
completion: 'git diff' completes untracked paths as a last resort
completion: complete tracked paths for 'git diff'
completion: no-op refactoring of diff completion
When a lookup with `OBJECT_INFO_DIE_IF_CORRUPT` fails we want to die in
case the object exists, but cannot be read. This flag is handled in two
different spots right now:
- `do_oid_object_info_extended()` calls `has_packed_and_bad()` to
check whether the object is known to be corrupt in any packfile.
This function reaches into the internals of the packed source and
thus breaks the abstraction provided by our object sources.
- The loose source handles the flag itself and dies directly in
`read_object_info_from_path()`, which means that we die even in
cases where another source may still have a good copy of the
object.
Besides being inconsistent, it also ties us to the specific backend used
by the database sources because `has_packed_and_bad()` assumes that they
use the "files" backend. Any other backend will instead cause us to die
when calling `odb_source_files_downcast()`, even if the object was
simply nonexistent.
In the preceding commits we've carved out the infrastructure to make
this mechanism fully generic. On the one hand, all backends now tell us
whether the object is missing or corrupt via their return values. And
on the other hand, they have been taught to provide a readable error
message to the caller.
Adapt `do_oid_object_info_extended()` to use those new mechanisms. This
means that we won't die immediately anymore when a loose object is
corrupt, and we properly handle backends other than the "files" backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When reading an object fails even though it exists, the sources know
best what exactly went wrong and where the corrupt object is located.
This information is lost though when bubbling up the error to the object
database layer, which forces that layer to reconstruct it after the
fact. This is exactly what `do_oid_object_info_extended()` does via
`has_packed_and_bad()`, but that function only really knows to handle
the "files" backend by reaching into its internals.
Introduce a new `errmsg` parameter for the `read_object_info()` callback
that sources are expected to populate with a human-readable message in
case reading the object has failed. Adapt the packed and loose sources
to populate the buffer with the messages that we ultimately want to
surface to the user.
For now, all callers are adapted to pass a `NULL` pointer. We will add a
user of this new infrastructure in a subsequent commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As explained in the preceding commits, reading objects can either fail
because the object truly does not exist or because it exists, but its
data is corrupt. Some callers do care about this distinction, but there
is no way to tell these two cases apart right now.
Introduce a new `ODB_READ_NOT_FOUND` value that ought to be returned by
the backends in case the object truly does not exist and adapt backends
to use it.
Note that we don't yet return this error from `odb_read_object_info()`
itself. This will be fixed in a subsequent commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `read_object_info()` callback of `struct odb_source` is documented
to return a negative error code in case reading the object has failed,
and zero otherwise. This is overly broad though, as there are two very
different kinds of failures:
- The object may not exist in the source at all.
- The object exists, but reading it has failed, for example because
its on-disk state is corrupt.
This distinction matters to callers: when an object is corrupt in one
source we may still find a good copy of it in another source, so we may
still be able to proceed with a given operation.
The "packed" source already distinguishes these cases by returning a
positive value for missing objects and a negative value in case reading
the object has failed. But it is the only such source that distinguishes
those cases, and the returned value is translated into a negative error
code by the "files" backend anyway.
Introduce a new error status that is specific to reading objects and
adapt the infrastructure to return it. For now, we only discern
successful reads from generic failures, which mostly matches the status
quo. In subsequent commits though we're about to add an error that
explicitly tells the caller that an object does not exist.
Note that we keep the "packed" backend as-is with its positive return
code for missing objects. This will be fixed in the next commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When reading packed objects we know to tell apart missing objects and
corrupt objects by returning a positive error code in the former case,
and a negative one in the latter case. We do that by distinguishing
between errors returned by `find_pack_entry()`, which yields the offset
of the object, and `packed_object_info()`, which reads the object
contents.
But even though we already distinguish those cases when reading packed
objects, the logic is broken in case a caller tries to read an object
that has been marked as corrupt. In that case, `find_pack_entry()` will
tell us that the object in question does not exist, and consequently
we'll not flag the object as corrupt but as missing.
Fix this issue by bubbling up whether the object is corrupt and, if so,
which packfile contains the corrupted object.
Note that we don't yet need the information about the specific packfile,
so we could've just as well made this a `bool *corrupted` pointer. But
we'll need information about the containing packfile in a subsequent
commit so that we can generate a proper error message telling the user
which packfile contains the broken object.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The unused name parameter in 'struct chdir_notify_entry' has been
removed from chdir_notify_register(), chdir_notify_unregister(), and
related callback signatures across several subsystems, simplifying the
API now that trace output no longer uses it.
* ch/chdir-notify-drop-name:
chdir-notify.h: Removed unused param 'name'
The performance of adding numerous new packfiles has been improved
by introducing a fast path for known-new packfiles to skip an
unnecessary traversal in packfile_list_append(), avoiding a
quadratic complexity regression on load.
* js/packfile-fast-append:
packfile: fix perf regression with many packs
'git repack' has been taught '--drop-filtered' to delete local
promisor blobs exceeding a limit (currently 'blob:limit=') in partial
clones, reclaiming space. Guards prevent running during other
operations or if referenced by the index.
* ss/repack-drop-filtered:
builtin/repack: add guards for --drop-filtered
builtin/repack: actually drop filtered promisor blobs
builtin/repack: enumerate promisor blobs for --drop-filtered
repack-promisor: allow excluding objects from the rebuilt promisor pack
list-objects-filter: add list_objects_filter__filter_oidset()
builtin/repack: add --drop-filtered and --dry-run options
The 'git bisect' command has been taught a
'--reset-when-found[=<where>]' option that tells the command to
automatically run 'git bisect reset' to jump back to the original
state or to the found culprit.
* hn/bisect-reset-when-found:
bisect: add --reset-when-found to leave when done
bisect: let bisect_reset() optionally check out quietly
A compatibility wrapper for writev(3p) has been reintroduced,
including fixes for CMake build and 'MAX_IO_SIZE' limits on NonStop.
Calls to write(3p) in send_sideband() and cat_blob() have been
refactored to use writev(3p) wrappers to reduce syscall overhead.
* ps/writev:
fast-import: use writev(3p) to send cat-blob responses
sideband: use writev(3p) to send pktlines
wrapper: properly handle MAX_IO_SIZE in writev(3p)
wrapper: introduce writev(3p) wrappers
compat/posix: introduce writev(3p) wrapper
The known limitations of the ref format migration in 'git refs' have
been moved to be displayed as a warning admonition directly under the
description of the 'migrate' subcommand, improving visibility. A
reference to 'git-maintenance' has also been corrected to use the
'linkgit' macro.
* kh/doc-refs-migrate-limitations:
doc: refs: linkgit to git-maintenance(1)
doc: refs: put ref migration warning under the command
This code block uses the placeholder `<subject>`. Let’s highlight this
placeholder properly by using the `synopsis` open block definition which
was introduced in a34d1d53 (doc: convert git-show to synopsis style,
2026-02-06). This renders the block like a code block but with emphasis
styling on placeholders, just like inline-verbatim (`) in running text.
Yes, note that open blocks since commit a34d1d53 can, on synopsis-style
docs like this one, be immediately preceded by `[synopsis]`, just like
the command synopsis is:
[synopsis]
(EXPERIMENTAL!) git format-rev - [...]
Cf. verse-style:
[verse]
'git name-rev' [...]
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We first talk about just `%s`, but then show the result with
quotes. That is inconsistent. Let’s use quotes both in the format
as well as in the result.
The implied input here, which is not spelled out for brevity, is:
Did we not fix this in <commit object name>?
Which is then supposed to be formatted to `"<subject>"`.
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Various tests in 't7900-maintenance.sh' have been updated to use a
throwaway repository, and auto-detaching of maintenance tasks is now
disabled for these tests to fix flaky races with concurrent background
maintenance jobs.
* ps/t7900-deflake-maintenance:
t7900: fix flaky "maintenance.strategy" test
t7900: adapt some tests to use a throwaway repository
A client requesting the promisor-remote capability without a value
caused a null pointer dereference, which has been corrected by
rejecting a request without an argument.
* en/serve-promisor-remote-fix:
serve: reject valueless promisor-remote capability
The 'pack-objects' and delta-encoding code paths have been updated to
use 'size_t' instead of 'unsigned long' for object sizes and offset
limits, avoiding potential truncation issues on 64-bit Windows.
* js/pack-objects-delta-size-t:
packfile: widen `unpack_object_header_buffer()` to `size_t`
git-zlib: widen `git_deflate_bound()` to `size_t`
t/helper/test-pack-deltas: widen `do_compress()`'s maxsize local to `size_t`
http-push: widen `start_put()`'s size local from `ssize_t` to `size_t`
diff: widen `deflate_it()`'s bound local from int to `size_t`
archive-zip: widen `zlib_deflate_raw()`'s maxsize local to `size_t`
packfile, git-zlib: widen `use_pack()` and zstream avail fields to `size_t`
delta: widen `create_delta()` and `diff_delta()` to `size_t`
pack-objects: widen `mem_usage` and `try_delta()`'s out-param to `size_t`
pack-objects: widen `free_unpacked()` return to `size_t`
pack-objects: widen delta-cache accounting to `size_t`
delta: widen `create_delta_index()` parameter to `size_t`
diff-delta: widen `struct delta_index`' size fields to `size_t`
The '--shallow-file' option of 'git' command requires a value, but the
code did not check the presence of a value and instead segfaulted
without one, which has been corrected.
* cc/git-shallow-file-wo-value:
git: avoid segfault on "git --shallow-file" without a value
The setting of a now-unused member '.pretty_given' in the sequencer
machinery has been removed.
* en/sequencer-lose-pretty-given:
sequencer: remove unnecessary variable setting
A handful of code paths have been corrected to check return values
from functions like curl_easy_duphandle(), deflateInit(), lseek(),
dup(), and strbuf_getline_lf(), resolving several Coverity warnings
about unchecked returns.
* js/coverity-unchecked-returns-fix:
bisect: handle dup() failure when redirecting stdout
bisect: check get_terms return at all call sites
bisect: check strbuf_getline_lf return when reading terms
transport-helper: warn when export-marks file cannot be finalized
transport-helper: check dup() return in get_exporter
compat/pread: check initial lseek for errors
last-modified: handle repo_parse_commit() failures
reftable tests: check reftable_table_init_ref_iterator() return
reftable/block: check deflateInit() return value
reftable: handle block-writer initialization errors
config: propagate launch_editor() failure in show_editor()
http: die on curl_easy_duphandle failure in get_active_slot
The `struct object_database::alternates_db` field tracks the value of
the "GIT_ALTERNATE_OBJECT_DIRECTORIES" environment variable and is
used in `odb_prepare_alternates()`. It's not necessary to store it as a
separate field anymore though, as we stopped lazy-loading alternates.
Consequently, we can simply pass it to `odb_prepare_alternates()` via
`odb_new()` now.
Do so and remove the field.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `struct object_database::loaded_alternates` field tells us whether
or not alternates have been loaded already. This field was useful before
the preceding commit as we were indeed lazy-loading alternates. But now
that we started to eagerly load them we can assume them to be loaded
after `odb_new()`, and hence the field does not serve any purpose
anymore.
Remove it.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When creating the object database we initialize the main object database
source, but we don't yet initialize its alternates. Instead, we have
many calls to `odb_prepare_alternates()` cluttered around the code base
whenever we are about to iterate through the sources.
This lazy loading doesn't really add much value: the moment where we
read any object we _have_ to load the alternates anyway. So given that
most of our commands would access the object database this optimization
is not really buying us much in the first place. Quite on the contrary,
it makes the code harder to understand and is a potential source of bugs
in case any callsite forgot to prepare alternates before we iterate
through the sources.
Historically though there was a reason why we deferred lazy-loading: it
may happen that the repository has "core.ignoreCase" configured, and we
use that to deduplicate the list of alternates in case we had the same
alternate configured multiple times, but with different casing. We used
to initialize the object database before we had fully configured the
owning repository though, and consequently we couldn't access that
configuration yet. This has changed in the preceding commit though where
we started to parse "core.ignoreCase" manually.
Eagerly prepare alternates both when creating the object database and
when flushing its caches. Drop the now-unneeded calls to prepare the
alternates that are scattered across the code base.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When registering alternates we deduplicate object database sources by
their path so that the same source won't be added twice. Ever since
cf2dc1c238 (speed up alt_odb_usable() with many alternates, 2021-07-07)
this duplicate check is backed by a map keyed by the source's path,
using `fspathhash()` and `fspatheq()` as hash and equality functions,
respectively.
These functions are problematic in this context for two reasons:
- They implicitly depend on `the_repository` instead of the
repository that owns the object database.
- They derive case-sensitivity from `repo_ignore_case()`, which
returns a default value in case the repository's configuration has
not been parsed yet. Object database sources may be registered
before that is the case, so the answer may flip depending on when a
source gets registered.
Fix this by making the comparison self-contained in the object
database. Instead of using `fspathhash()` and `fspatheq()` we resolve
"core.ignoreCase" manually and then use the correct comparison function
based on the result. This requires us to migrate to a `struct hashmap`,
as the khash interface does not give us the ability to pass an arbitrary
payload to these functions, and hence we'd have to use global state to
decide which of those to use.
Note that we can unconditionally use `strihash()` to compute entry
hashes regardless of case sensitivity: a hash function only needs to
guarantee that equal keys have equal hashes, and a case-insensitive
hash satisfies this requirement for both case-sensitive and
case-insensitive equality.
Overall it's quite debatable whether all of this complexity really is
worth it, out of two reasons:
- We could linearly search through all sources to find duplicates. But
the mentioned commit cares about cases with thousands of alternates,
and a linear search would of course regress performance quite a bit.
This doesn't really feel like a reasonable case to care about, but I
don't feel comfortable regressing it anyway.
- It's dubious whether we should handle "core.ignoreCase" in the first
place. The downside would be that we might add the same alternate
multiple times with different casing. But this is an edge case, and
it's not even fully fixed because we don't resolve symlinks or
mountpoints, either.
So for now, keep this infrastructure in-place while removing the global
dependency on `the_repository`. We may want to revisit this in the
future though.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When creating a new repository we create both the reference and object
databases after we have finalized the repository. This ensures that
those subsystems find a fully-configured repository at the time where
they are asked to create their own on-disk data structures.
There is one exception though: while we have already fully configured
the repository at this point, we haven't yet written both
"core.sharedRepository" and "receive.denyNonFastforwards". The latter
configuration doesn't really matter to us, but the first one does as the
"files" object database source reads it.
This doesn't cause any problems right now, but it will in a subsequent
patch where we will start to read "core.ignoreCase" when creating the
object database. Move the initialization of both of these data
structures towards the end of `init_db()`. The only thing that now comes
after is status reporting, but that's it.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano