The consistency checks for the object database (fsck) have been
decoupled from the generic builtin implementation and moved into the
backend-specific object source layers, making them pluggable for
different object storage formats.
* ps/odb-pluggable-fsck:
builtin/fsck: move loose object verification into the loose source
builtin/fsck: move multi-pack index verification into the packed source
builtin/fsck: move bitmap verification into the packed source
builtin/fsck: move reverse index verification into the packed source
builtin/fsck: move packfile verification into the packed source
odb: provide infrastructure for pluggable fsck checks
builtin/fsck: don't check alternates with "--no-full"
builtin/fsck: de-globalize option handling
builtin/fsck: merge `fsck_obj_buffer()` and `fsck_obj()`
builtin/fsck: use `fsck_obj_buffer()` when checking loose objects
The setup of alternates has been deferred to object database
creation time during clone, which drops the unused ad-hoc alternate
writing API, simplifying the object database backend interface.
* ps/odb-alternates-at-creation:
odb/source: remove the ability to write alternates
builtin/clone: write alternates via `odb_create_on_disk()`
odb/source: support writing alternates when creating the database
builtin/clone: move setup of alternates for non-shared local clones
builtin/clone: move setup of alternates for shared local clones
builtin/clone: refactor handling of "--reference{,-if-able}"
builtin/clone: move around `setup_reference()`
builtin/clone: defer setup of the object database
The codebase has been updated to use the newer utimensat() POSIX
function instead of the obsolescent utime(), allowing
high-precision timestamps while preserving fallback compatibility.
* as/utimensat-utimes:
compat/posix: drop legacy <utime.h> header and shims
treewide: use utimensat(2) instead of legacy utime(3p)
compat/posix: introduce utimensat(2) wrapper
The mechanism to generate a packfile corresponding to the result of
a fetch/push has been made pluggable through a set of object
database callback functions, removing hardcoded references to
'pack-objects' and enabling alternative ODBs to serve packfiles
themselves.
* ps/odb-pluggable-pack-generation:
bundle: generate packfiles via the object database
bundle: get (mostly) rid of `the_repository`
builtin/bundle: refactor option handling for progress meter
send-pack: generate packfiles via the object database
upload-pack: generate packfiles via the object database
odb: introduce interface to generate packfiles
The 'git receive-pack' command has been updated to use a new ODB
transaction interface for writing incoming packfiles, making it more
backend-agnostic.
* jt/receive-pack-pluggable-writes:
odb/transaction: add transaction interface to write packfiles
odb: return temporary ODB source when set
builtin/receive-pack: explicitly pass packfile fd
builtin/receive-pack: report unpack errors via strbuf
builtin/receive-pack: lift global state out of unpack()
builtin/receive-pack: read unpack limit config lazily
builtin/receive-pack: pass shallow file explicitly
odb/transaction: add transaction finalize interface
builtin/receive-pack: properly clean up keep files
The threshold for geometric repacking to trigger based on loose
object count has been adjusted to match that of 'git gc --auto',
preventing over-aggressive repacking during concurrent writes.
* ps/odb-geometric-repack-loose-threshold:
odb/files: be less aggressive with geometric repacking
The object database layer has been simplified by eagerly loading
alternate object directories upon initialization, instead of
deferring it to the first object lookup. This eliminates the need
for scattered lazy-loading calls throughout the codebase and paves
the way for integrating alternates with the pluggable backends.
* ps/odb-eagerly-load-alternates:
odb: drop `alternates_db` field
odb: drop `loaded_alternates` field
odb: eagerly initialize alternates
odb: decouple source path comparisons from `the_repository`
setup: create ref and object databases after config is written
The object database (odb) API has been refactored to distinguish
between missing objects and corrupt ones by returning more
descriptive error statuses. Both the packed and loose backends now
faithfully propagate error details using a generic strbuf error
mechanism, removing backend-specific leakage from central lookup
paths.
* ps/odb-generic-corrupt-objects:
odb: handle `OBJECT_INFO_DIE_IF_CORRUPT` generically
odb/source: allow `read_object_info()` to bubble up error messages
odb/source: let callers discern missing and corrupt objects
odb/source: introduce error status when reading objects
odb/source-packed: flag known-bad objects as corrupt and not missing
The consistency checks for loose objects are hosted by "builtin/fsck.c".
These checks are obviously specific to the "loose" backend.
Move the logic into `odb_source_loose_fsck()`. Introduce a new "verbose"
flag so that we can properly retain semantics around whether or not we
want to print some status messages.
Note that this fixes a bug as a side effect: the progress meter was
captured in the callback data before `start_progress()` was even called,
so the per-subdirectory progress updates always operated on a NULL
pointer and the meter jumped straight from 0 to 256 upon completion. The
new code only sets up the callback data's progress meter after it has
been created, so the progress display now advances incrementally again.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The checks for multi-pack indexes are hosted in `cmd_fsck()` directly.
These checks are obviously specific to the "packed" backend.
Move the logic into `odb_source_packed_fsck()`. As in preceding commits,
this means that we now properly honor both "--connectivity-only" and
"--no-full". Furthermore, we drop the dedicated `ERROR_MULTI_PACK_INDEX`
bit and instead use the generic `ERROR_OBJECT` bit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The checks for bitmaps live in `verify_bitmap_files()`, which is called
by "builtin/fsck.c". These checks are obviously specific to the "packed"
backend.
Move the logic into `odb_source_packed_fsck()`. As in preceding commits,
this means that we now properly honor both "--connectivity-only" and
"--no-full". Furthermore, we drop the dedicated `ERROR_BITMAP` bit and
instead use the generic `ERROR_OBJECT` bit.
Note that this change also adapts `verify_bitmap_files()` to be
focused on a single "packed" source instead of verifying bitmaps from
all sources. This change is required as we already know to loop around
the sources in `odb_fsck()` itself.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The checks for reverse indexes live in `check_pack_rev_indexes()`, which
is hosted in "builtin/fsck.c". These checks are obviously specific to
the "packed" backend.
Move the logic into `odb_source_packed_fsck()`. As in the preceding
commit, drop the dedicated `ERROR_PACK_REV_INDEX` bit and instead use
the generic `ERROR_OBJECT` bit.
Note that this changes behaviour in two ways:
- The checks are now skipped when "--connectivity-only" was passed.
This is because we don't even run `odb_fsck()` at all when that
flag has been passed by the user, and not verifying data structures
of the object database matches the documented intent of that flag,
which is to only check the connectivity of reachable objects.
- The checks are now skipped for non-local sources when "--no-full"
was passed. This is, again, in line with the documented intent of
that flag.
Add a test to cast these semantics into stone.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move the packfile verification out of `cmd_fsck()` and into the "packed"
source. While doing so, thread the progress meter and object callback
through the newly introduced `struct odb_fsck_options` so that the
caller's preferences are honoured without exposing those details at the
"builtin/fsck.c" level.
Note that the old code reported failures when verifying packfiles with
the `ERROR_PACK` bit, which gets returned to the caller via the exit
code. This bit is neither exercised in our test suite nor is it
documented anywhere in our codebase. Furthermore, this bit is highly
specific to the object storage backend, which makes it a bad fit for the
new pluggable infrastructure. So instead of retaining these semantics,
we drop them and return the generic `ERROR_OBJECT` bit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The on-disk consistency checks in git-fsck(1) are conceptually
backend-specific: while connectivity checks and object-level parsing
checks are generic, verifying the physical integrity of packfiles and
loose objects is meaningful only to backends that use these formats:
Having these checks live in "builtin/fsck.c" violates that layering,
because it forces the command to reach directly into format-specific
internals.
Provide new infrastructure to make these format-specific checks
pluggable and implement stubs for the different source types we already
have. In subsequent commits we'll move functionality over piece by
piece.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There are no users of `odb_source_write_alternates()` in our tree
anymore. Remove that function and its supporting infrastructure.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add the ability to write alternates when creating the object database.
This change allows us to remove the `write_alternates()` callback in a
subsequent patch.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ps/odb-eagerly-load-alternates:
odb: drop `alternates_db` field
odb: drop `loaded_alternates` field
odb: eagerly initialize alternates
odb: decouple source path comparisons from `the_repository`
setup: create ref and object databases after config is written
* ps/odb-pluggable-pack-generation:
bundle: generate packfiles via the object database
bundle: get (mostly) rid of `the_repository`
builtin/bundle: refactor option handling for progress meter
send-pack: generate packfiles via the object database
upload-pack: generate packfiles via the object database
odb: introduce interface to generate packfiles
The unused name parameter in 'struct chdir_notify_entry' has been
removed from chdir_notify_register(), chdir_notify_unregister(), and
related callback signatures across several subsystems, simplifying the
API now that trace output no longer uses it.
* ch/chdir-notify-drop-name:
chdir-notify.h: Removed unused param 'name'
* ps/odb-eagerly-load-alternates:
odb: drop `alternates_db` field
odb: drop `loaded_alternates` field
odb: eagerly initialize alternates
odb: decouple source path comparisons from `the_repository`
setup: create ref and object databases after config is written
The 'struct odb_read_stream' and 'struct odb_write_stream'
structures have been consolidated into a single unified 'struct
odb_stream' structure, simplifying object database streaming APIs
and enabling streaming of arbitrary object types.
* ps/odb-streams:
odb/streaming: unify function names to create new streams
odb/streaming: rename `struct input_zstream_data`
odb/streaming: rename `struct read_object_fd_data`
odb/streaming: consolidate read and write streams
odb/streaming: rename `struct odb_read_stream`
odb/streaming: support streaming arbitrary object types
odb/streaming: drop `is_finished` field
odb/streaming: track write stream size in the structure
Now that a compatibility wrapper for utimensat(2) has been introduced,
migrate all call sites across the codebase to use utimensat(2) instead of
the legacy utime(3p) interface:
- In `commit-graph.c`, use utimensat(2) with UTIME_OMIT and the computed
timestamp `now` to bump the commit-graph modification time consistently
across all files without needing an extra stat(2) call to preserve atime.
- In `copy.c`, use utimensat(2) to copy full sub-second access and
modification timestamps from the source file.
- In `odb/source-packed.c`, `odb/source-loose.c`, and `object-file.c`,
use utimensat(2) with `struct timespec` to freshen file timestamps.
- In `builtin/pack-objects.c`, update the pack timestamp with
utimensat(2).
- In `rerere.c`, touch the postimage file with utimensat(2) passing NULL
to set both atime and mtime to current time.
- In `t/helper/test-chmtime.c`, update file modification times using
utimensat(2).
Signed-off-by: Alexey Samsonov <vonosmas@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When performing auto-maintenance with geometric repacking we have two
conditions that may trigger a repack:
- Either the geometric sequence of packfiles is invalidated.
- Or we have too many loose objects.
The first condition shouldn't trigger all that often: it may be hit when
we fetch a new packfile, but users tend to not do that all the time. The
second condition is what typically triggers more regularly though, as
every command that ends up writing new objects may cause us to cross the
threshold of loose objects. It is thus preferable to not be too
aggressive here, as otherwise we may end up repacking objects quite
often.
For the geometric-repacking strategy though we have a default of 100
objects, only. As we're approximating the count of objects by only
reading the "objects/17/" shared, we'd only need 2 objects in there
before we perform a repack by default, which is quite aggressive.
git-gc(1) on the other hand has a default of 6700, so it is quite a bit
more conservative here.
Being this aggressive is also causing problems as reported by our users.
When running lots of concurrent writers, those writes will constantly
end up spawning maintenance jobs that end up repacking objects. As we
also prune objects, a concurrently running process that tries to write
an object may see that the sharding directories get removed under their
feet. While we try re-creating such leading directories, we only do so a
single time, and it may happen that the directory vanishes again before
we had the chance to create the loose object. This is not a new problem,
but it is exacerbated by us running maintenance this aggressively.
Improve the status quo by reducing the frequency at which we pack loose
objects to the same frequency that git-gc(1) uses.
Reported-by: Stefan Haller <lists@haller-berlin.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Packfiles have two primary use cases:
- They are used to store objects at rest in a Git repository.
- They are used on the transport layer to transfer objects between two
repositories.
The first class is closely tied to a given object database backend, and
as such this use is highly specific to how such a backend decides to
store its data. This shows in git-pack-objects(1), which is used by
git-repack(1) et al to optimize the object database, which supports lots
of options that are closely coupled with how data is stored.
But the second class is quite a lot more generic: we don't care about
specifics of how the object database stores its objects, but to generate
the packfiles we only care about the object graph itself. Still, this
use case is also coupled with git-pack-objects(1).
Unfortunately, because git-pack-objects(1) covers both classes, the
result is that it is very hard to port the whole command to properly
support pluggable object databases. There are simply way too many
options that an alternative implementation will have a very hard time to
support in the first place.
And despite being hard to implement, it's also quite unnecessary to
implement those backend-specific options. Optimizing the object database
has already been made pluggable, and an alternative implementation is
unlikely to care about cruft packs, unpacked objects, keep packs and the
like. But we still need to make at least _parts_ of the packfile
generation pluggable so that backends can generate packfiles for the
transport layer itself.
Introduce a new interface that lets backends generate a new packfile and
implement that interface for the "files" backend. The options supported
by the callback are exactly the set of options that are required for the
transport layer, but nothing more.
This means that git-pack-objects(1) itself cannot be ported over to this
new interface, but as explained above that's a hard feat to pull off due
to the backend-specific features. Ideally though, we should expose the
ability to generate arbitrary packfiles using this interface. The intent
of this is to eventually introduce a git-objects(1) subcommand (similar
to git-refs(1)) that exposes generic interfaces for accessing everything
related to the object database. In that case, we are able to expose only
those options that are generic.
Subsequent commits will convert git-upload-pack(1), git-send-pack(1) and
git-bundle(1) to use this interface.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In git-receive-pack(1), the incoming packfile is written to the ODB via
`unpack()`, which spawns git-index-pack(1) or git-unpack-objects(1)
directly. With pluggable object databases, an alternative backend may
need to handle writing packfile data differently though.
Introduce `odb_transaction_write_pack()` as a generic interface to
handle writing a packfile to a transaction and use the logic from
`unpack()` as the "files" backend implementation. Note that when storing
the objects as a packfile, git-index-pack(1) also writes a ".keep"
lockfile next to it to prevent a concurrent repack from removing the new
pack prior to reference updates being performed. The "files" transaction
backend is responsible for managing these ".keep" files and removes them
post-commit once the transaction is finalized.
Call sites in git-receive-pack(1) are updated accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When committing an ODB transaction via `odb_transaction_commit()`, the
staged objects are made visible and the underlying transaction is freed
at the same time. Coupling these two steps does not leave room for any
post-commit transaction operations to be introduced though. Such a
capability is useful if an ODB transaction backend needs to hold on to
lockfiles after transaction commit until references are updated, as is
the case with the existing "files" backend in git-receive-pack(1).
Stop freeing the transaction in `odb_transaction_commit()` and introduce
`odb_transaction_finalize()` to explicitly clean up the transaction
accordingly. Note that the finalize interface also provides an optional
callback for any backend-specific deferred cleanup. In a subsequent
commit, the "files" transaction backend will use this to remove ".keep"
files generated for packfiles received via git-receive-pack(1) after
references have been updated. In preparation for this, the
`odb_transaction_finalize()` call site in git-receive-pack(1) is made
after the reference updates are finished.
All other callers commit a transaction and immediately finalize it
without any work happening in between those two operations.
Consequently, they cannot meaningfully recover in case either of them
would fail, and spelling out these two separate steps with proper error
handling would be quite repetitive and pointless. Introduce a helper
`odb_transaction_commit_and_finalize_or_die()` for those call sites and
update them accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The creation of the on-disk data structures for the object database
has been made pluggable, allowing future backends to customize their
setup. As part of this, the initialization of the object database
has been deferred, and the loading of the loose-object map has been
detangled from repository initialization.
* ps/odb-make-creation-pluggable:
odb: make creation of on-disk structures pluggable
odb/source: introduce function to map source type to name
setup: defer object database creation
setup: handle ODB-related environment variables in `odb_new()`
setup: detangle loading of loose object maps
loose: load loose object map for the correct source
When a lookup with `OBJECT_INFO_DIE_IF_CORRUPT` fails we want to die in
case the object exists, but cannot be read. This flag is handled in two
different spots right now:
- `do_oid_object_info_extended()` calls `has_packed_and_bad()` to
check whether the object is known to be corrupt in any packfile.
This function reaches into the internals of the packed source and
thus breaks the abstraction provided by our object sources.
- The loose source handles the flag itself and dies directly in
`read_object_info_from_path()`, which means that we die even in
cases where another source may still have a good copy of the
object.
Besides being inconsistent, it also ties us to the specific backend used
by the database sources because `has_packed_and_bad()` assumes that they
use the "files" backend. Any other backend will instead cause us to die
when calling `odb_source_files_downcast()`, even if the object was
simply nonexistent.
In the preceding commits we've carved out the infrastructure to make
this mechanism fully generic. On the one hand, all backends now tell us
whether the object is missing or corrupt via their return values. And
on the other hand, they have been taught to provide a readable error
message to the caller.
Adapt `do_oid_object_info_extended()` to use those new mechanisms. This
means that we won't die immediately anymore when a loose object is
corrupt, and we properly handle backends other than the "files" backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When reading an object fails even though it exists, the sources know
best what exactly went wrong and where the corrupt object is located.
This information is lost though when bubbling up the error to the object
database layer, which forces that layer to reconstruct it after the
fact. This is exactly what `do_oid_object_info_extended()` does via
`has_packed_and_bad()`, but that function only really knows to handle
the "files" backend by reaching into its internals.
Introduce a new `errmsg` parameter for the `read_object_info()` callback
that sources are expected to populate with a human-readable message in
case reading the object has failed. Adapt the packed and loose sources
to populate the buffer with the messages that we ultimately want to
surface to the user.
For now, all callers are adapted to pass a `NULL` pointer. We will add a
user of this new infrastructure in a subsequent commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As explained in the preceding commits, reading objects can either fail
because the object truly does not exist or because it exists, but its
data is corrupt. Some callers do care about this distinction, but there
is no way to tell these two cases apart right now.
Introduce a new `ODB_READ_NOT_FOUND` value that ought to be returned by
the backends in case the object truly does not exist and adapt backends
to use it.
Note that we don't yet return this error from `odb_read_object_info()`
itself. This will be fixed in a subsequent commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `read_object_info()` callback of `struct odb_source` is documented
to return a negative error code in case reading the object has failed,
and zero otherwise. This is overly broad though, as there are two very
different kinds of failures:
- The object may not exist in the source at all.
- The object exists, but reading it has failed, for example because
its on-disk state is corrupt.
This distinction matters to callers: when an object is corrupt in one
source we may still find a good copy of it in another source, so we may
still be able to proceed with a given operation.
The "packed" source already distinguishes these cases by returning a
positive value for missing objects and a negative value in case reading
the object has failed. But it is the only such source that distinguishes
those cases, and the returned value is translated into a negative error
code by the "files" backend anyway.
Introduce a new error status that is specific to reading objects and
adapt the infrastructure to return it. For now, we only discern
successful reads from generic failures, which mostly matches the status
quo. In subsequent commits though we're about to add an error that
explicitly tells the caller that an object does not exist.
Note that we keep the "packed" backend as-is with its positive return
code for missing objects. This will be fixed in the next commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When reading packed objects we know to tell apart missing objects and
corrupt objects by returning a positive error code in the former case,
and a negative one in the latter case. We do that by distinguishing
between errors returned by `find_pack_entry()`, which yields the offset
of the object, and `packed_object_info()`, which reads the object
contents.
But even though we already distinguish those cases when reading packed
objects, the logic is broken in case a caller tries to read an object
that has been marked as corrupt. In that case, `find_pack_entry()` will
tell us that the object in question does not exist, and consequently
we'll not flag the object as corrupt but as missing.
Fix this issue by bubbling up whether the object is corrupt and, if so,
which packfile contains the corrupted object.
Note that we don't yet need the information about the specific packfile,
so we could've just as well made this a `bool *corrupted` pointer. But
we'll need information about the containing packfile in a subsequent
commit so that we can generate a proper error message telling the user
which packfile contains the broken object.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When creating the object database we initialize the main object database
source, but we don't yet initialize its alternates. Instead, we have
many calls to `odb_prepare_alternates()` cluttered around the code base
whenever we are about to iterate through the sources.
This lazy loading doesn't really add much value: the moment where we
read any object we _have_ to load the alternates anyway. So given that
most of our commands would access the object database this optimization
is not really buying us much in the first place. Quite on the contrary,
it makes the code harder to understand and is a potential source of bugs
in case any callsite forgot to prepare alternates before we iterate
through the sources.
Historically though there was a reason why we deferred lazy-loading: it
may happen that the repository has "core.ignoreCase" configured, and we
use that to deduplicate the list of alternates in case we had the same
alternate configured multiple times, but with different casing. We used
to initialize the object database before we had fully configured the
owning repository though, and consequently we couldn't access that
configuration yet. This has changed in the preceding commit though where
we started to parse "core.ignoreCase" manually.
Eagerly prepare alternates both when creating the object database and
when flushing its caches. Drop the now-unneeded calls to prepare the
alternates that are scattered across the code base.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When registering alternates we deduplicate object database sources by
their path so that the same source won't be added twice. Ever since
cf2dc1c238 (speed up alt_odb_usable() with many alternates, 2021-07-07)
this duplicate check is backed by a map keyed by the source's path,
using `fspathhash()` and `fspatheq()` as hash and equality functions,
respectively.
These functions are problematic in this context for two reasons:
- They implicitly depend on `the_repository` instead of the
repository that owns the object database.
- They derive case-sensitivity from `repo_ignore_case()`, which
returns a default value in case the repository's configuration has
not been parsed yet. Object database sources may be registered
before that is the case, so the answer may flip depending on when a
source gets registered.
Fix this by making the comparison self-contained in the object
database. Instead of using `fspathhash()` and `fspatheq()` we resolve
"core.ignoreCase" manually and then use the correct comparison function
based on the result. This requires us to migrate to a `struct hashmap`,
as the khash interface does not give us the ability to pass an arbitrary
payload to these functions, and hence we'd have to use global state to
decide which of those to use.
Note that we can unconditionally use `strihash()` to compute entry
hashes regardless of case sensitivity: a hash function only needs to
guarantee that equal keys have equal hashes, and a case-insensitive
hash satisfies this requirement for both case-sensitive and
case-insensitive equality.
Overall it's quite debatable whether all of this complexity really is
worth it, out of two reasons:
- We could linearly search through all sources to find duplicates. But
the mentioned commit cares about cases with thousands of alternates,
and a linear search would of course regress performance quite a bit.
This doesn't really feel like a reasonable case to care about, but I
don't feel comfortable regressing it anyway.
- It's dubious whether we should handle "core.ignoreCase" in the first
place. The downside would be that we might add the same alternate
multiple times with different casing. But this is an edge case, and
it's not even fully fixed because we don't resolve symlinks or
mountpoints, either.
So for now, keep this infrastructure in-place while removing the global
dependency on `the_repository`. We may want to revisit this in the
future though.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `name` parameter in `chdir_notify_entry` was only ever used by
chdir_notify_reparent() to produce trace output. That function was
removed in 5bf546755c (chdir-notify: drop unused
`chdir_notify_reparent()`, 2026-06-25), which left `name` with no
remaining consumers.
Prior to that removal, most callers had already stopped passing a
meaningful name, switching to NULL in 1f43ff2c7e (refs: unregister
reference stores from "chdir_notify", 2026-06-25) and 0de2467e6c
(odb/source-packed: start converting to a proper `struct odb_source`,
2026-06-17).
Since no caller has populated `name` with real data for some time,
and its last consumer is gone, drop it from chdir_notify_register(),
chdir_notify_unregister(), and the callback signature to simplify
the API.
Signed-off-by: Colin Hinton <colinlewishinton@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ps/odb-make-creation-pluggable:
odb: make creation of on-disk structures pluggable
odb/source: introduce function to map source type to name
setup: defer object database creation
setup: handle ODB-related environment variables in `odb_new()`
setup: detangle loading of loose object maps
loose: load loose object map for the correct source
When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.
Make the creation of on-disk structures pluggable to accommodate for
this.
Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.
Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:
- We assume that the primary object database is the "files" object
database and unconditionally downcast it. This will cause us to BUG
in case a different object database type was used together with a
compat hash algorithm.
- We require the object database to already have been initialized when
configuring the object database. This means that we must intermix
configuration of the repository and initialization of its
sub-structures in a weird way.
Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Object database housekeeping in 'git gc' and 'git maintenance' has
been refactored to be pluggable. The files-backend-specific logic,
including incremental and geometric repacking as well as object
pruning, has been moved out of the command implementation and into the
files object database source, enabling future alternative object
database backends to implement their own housekeeping services.
* ps/odb-pluggable-housekeeping:
odb: make optimizations pluggable
builtin/gc: fix signedness issues in ODB-related functionality
builtin/gc: refactor ODB optimizations to operate on "files" source
builtin/gc: introduce `odb_optimize_required()`
builtin/gc: move geometric repacking into `odb_optimize()`
builtin/gc: introduce object database optimization options
builtin/gc: inline config values specific to the "files" backend
builtin/gc: make repack arguments self-contained
builtin/gc: extract object database optimizations into separate function
builtin/gc: move worktree and rerere tasks before object optimizations
odb: run "pre-auto-gc" hook for all maintenance tasks
t7900: simplify how we check for maintenance tasks
Unify the function names to create new streams from different sources so
that they follow a common schema. While at it, document the ownership of
the file descriptor passed to `odb_stream_from_fd()`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
With the preceding refactorings the `struct read_object_fd_data` is now
somewhat misnamed, as it doesn't only contain the data anymore, but also
the stream itself. Rename the structure to `struct fd_stream` to better
match the new structure.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `struct odb_read_stream` and `struct odb_write_stream` both provide
the same functionality: they allow a caller to read object data from an
arbitrary source. Historically, the only difference was that the read
stream was used to read data out of the object database, whereas the
write stream was used to write data into the object database, but the
interfaces were mostly the same.
Over the preceding commits we have refactored the write stream to have
almost exactly the same interface as the read stream. With these
refactorings we can now easily merge those two streams into a single
interface that's used for both use cases.
While most of the changes are mechanical, there are two sites that need
special mention:
- "builtin/unpack-objects.c" creates a write stream from compressed
object data.
- "odb/streaming.c" creates a write stream from a file descriptor.
Adapting these sites to yield the new stream type requires a couple more
changes. Most importantly, instead of embedding the pointer to the data
in `struct odb_write_stream`, we now allocate a structure that wraps the
new `struct odb_stream` base. Other than that though, the changes are
rather straight forward.
Some of the structures and functions are now somewhat misnamed. These
will be fixed in subsequent commits.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rename `struct odb_read_stream` to just `struct odb_stream`. This
prepares for unification of the two different types of streams, as these
provide the same functionality with the preceding refactorings.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The object database supports the ability to write object streams into
it. This functionality is used when we encounter a blob that is larger
than "core.bigFileThreshold" so that we don't have to soak large files
into memory.
As we only ever write large files, the infrastructure doesn't support
specifying any other object type than "blob". This limitation is quite
artificial though: there is no reason why we shouldn't support writing
arbitrary large objects with a stream. While it's very unlikely that we
encounter a huge object other than a blob, users are known to be
creative and sometimes like to inflict pain on themselves by creating
commits or trees that are huge.
Extend the infrastructure to support streaming arbitrary object types.
For now we don't use this functionality anywhere, but it brings us a bit
closer to unify `struct odb_read_stream` and `struct odb_write_stream`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `is_finished` field is used to track whether a write stream is done
writing all of its data. Tracking this field as part of the stream
itself shouldn't be required though: callers will already know when the
stream is done when the stream's read function returns zero bytes, same
as when reading from a file descriptor.
There is one exception where it gets a bit more complicated: when
consuming data in "builtin/unpack-objects.c" it may happen that we don't
yield any new bytes after reading from the pipe. This is addressed by
looping until we have produced at least a single byte of output.
Drop the field from `struct odb_write_stream`. Again, same as in the
preceding commit, this brings the structure a bit closer to its sibling
`struct odb_read_stream`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When passing around a `struct odb_write_stream` we typically also have
to pass the number of bytes that the stream will yield. This is required
because the object header itself contains that size, and consequently we
cannot write the header without that information.
Move this information into the stream itself so that it becomes self-
describing. In addition to that, this also brings the `struct
odb_write_stream` a bit closer to the `struct odb_read_stream` so that
we can eventually merge both stream types.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The logic to write loose objects has been refactored and moved from
'object-file.c' to the loose backend source file 'odb/source-loose.c',
making the loose backend more self-contained. This is achieved by
first refactoring force_object_loose() to use generic ODB write
interfaces instead of loose-backend internals.
* ps/odb-move-loose-object-writing:
object-file: move logic to write loose objects
object-file: move `force_object_loose()`
object-file: force objects loose via generic interface
object-file: fix memory leak in `force_object_loose()`
odb: support setting mtime when writing objects
odb: lift object existence check out of the "loose" backend
odb: compute object hash in `odb_write_object_ext()`
t/u-odb-inmemory: implement wrapper for writing objects
odb: compute compat object ID in `odb_write_object_ext()`
The tempfile and lockfile APIs have been refactored to stop depending
on the 'the_repository' global variable, and their callers have been
updated to use the repository-aware variants.
* rs/tempfile-wo-the-repository:
use repo_hold_lock_file_for_update{,_mode,_timeout}() with custom repos
tempfile: stop using the_repository
lockfile: add repo_hold_lock_file_for_update{,_timeout}{,_mode}()
refs/packed: use repo_create_tempfile()
tempfile: add repo_create_tempfile{,_mode}()
The object database enumeration interface odb_for_each_object() has
been taught to accept object filters, allowing the underlying backends
to optimize the traversal by using reachability bitmaps when
available. 'git cat-file --batch-all-objects' has been updated to use
this generic interface, simplifying its code and avoiding direct
access to ODB backend internals.
* ps/odb-for-each-object-filter:
builtin/cat-file: filter objects via object database
odb: introduce object filters to `odb_for_each_object()`
pack-bitmap: introduce function to open bitmap for a single source
pack-bitmap: drop `_1` suffix from functions that open bitmaps
pack-bitmap: iterate object sources when opening bitmaps
pack-bitmap: allow aborting iteration of bitmapped objects
pack-objects: drop unused return value from add_object_entry()
pack-bitmap: mark object filter as `const`
odb/source-packed: improve lookup when enumerating objects
Junio C Hamano