Objects received by git-receive-pack(1) are quarantined in a temporary
"incoming" directory and migrated into the object database prior to the
reference updates. The quarantine is currently managed through
`tmp_objdir` directly. In a pluggable ODB future, how exactly an object
gets written to a transaction may vary for a given ODB source. Refactor
git-receive-pack(1) to use the ODB transaction interfaces to manage the
object staging area in a more agnostic manner accordingly.
Note that the ODB transaction is now responsible for managing the
primary and alternate ODBs for the repository. One small change as a
result is that the temporary directory is now applied as the primary ODB
in the main process instead of an alternate. This does not change
anything for git-receive-pack(1) though because it only needs access to
the newly written objects and doesn't care how exactly it is set up.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When performing the connectivity checks for a shallow ref in
`update_shallow_ref()`, the child process environment variables are
populated via `tmp_objdir_env()`. This is unnecessary though as
`update_shallow_ref()` is only reached after `tmp_objdir_migrate()` has
been performed which means there is no longer a temporary directory that
needs to be shared with child processes.
Drop the call to `tmp_objdir_env()` accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The temporary directory used by git-receive-pack(1) to write objects is
managed slightly differently than how it is done via ODB transactions:
- The temporary directory is eagerly created upfront, instead of
waiting for the first object write.
- The prefix name of the temporary directory is "incoming" instead of
"bulk-fsync".
In a subsequent commit, git-receive-pack(1) will use ODB transactions
instead of `tmp_objdir` directly. To provide a means to configure the
same transaction behavior, introduce `enum odb_transaction_flags` and
the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
transactions using the "files" backend to be set up for
git-receive-pack(1). Transaction call sites are updated accordingly to
provide the required flag parameter.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The ODB transaction backend is responsible for creating/managing its own
staging area for writing objects. Other child processes spawned by Git
may need access to uncommitted objects or write new objects in the
staging area though.
Introduce `odb_transaction_env()` which is expected to provide the set
of environment variables needed by a child process to access the
transaction's staging area.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When `odb_transaction_commit()` is invoked, the return value of the
backend commit callback is silently discarded. A backend has no way
to signal that committing failed, such as when the "files" backend
cannot migrate its temporary object directory into the permanent
ODB.
In a subsequent commit, git-receive-pack(1) starts using ODB transaction
to stage objects and consequently cares about such failures so it can
handle the error appropriately. Change the commit callback signature to
return an int error code and have `odb_transaction_commit()` forward it
accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When `odb_transaction_begin()` is invoked, the function returns the
transaction pointer directly. There is no way for the backend to
signal that it failed to set up its state, such as when creating the
temporary object directory backing the transaction.
In a subsequent commit, git-receive-pack(1) starts using ODB
transactions and needs to be able to report such failures rather
than silently ignore them. Refactor `odb_transaction_begin()` to
return an int error code and write the resulting transaction into an
out parameter. Also introduce `odb_transaction_begin_or_die()` as a
convenience for callsites that do not need to handle errors
explicitly.
Note that `odb_transaction_begin()` now returns an error when the ODB
already has an inflight transaction pending. ODB transaction call sites
that may encounter an inflight transaction are updated to explicitly
handle this case.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "files" transaction backend may encounter errors related to managing
the temporary directory used to stage objects, but silently ignores
these errors. Instead return errors encountered in the
`odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
callers to handle them as needed.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
ODB transactions are started via `odb_transaction_begin()` and contain
validation to avoid starting multiple transactions at the same time. The
"files" backend also has the same logic, but is redundant due to the
generic layer already handling it. Drop this validation from the "files"
backend accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When a "files" transaction is committed,
`flush_loose_object_transaction()` is invoked to handle performing a
hardware flush along with migrating the temporary object directory into
the primary and configuring the repository ODB source accordingly. The
function name here is a bit misleading because the helper is doing a bit
more than just "flushing" the transaction contents. Also, in a
subsequent commit, the transaction temporary directory is used to stage
packfiles and not just loose objects anymore.
Lift the helper function logic into `odb_transaction_files_commit()` to
more accurately signal to readers the operation being performed.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When writing an object to a "files" ODB transaction, a full hardware
flush is not initially performed during the fsync in
`fsync_loose_object_transaction()` and instead delayed until the
transaction is later committed.
To be more consistent with other "files" ODB transaction helpers, rename
the function to `odb_transaction_files_fsync()` accordingly. The
conditional in the helper is also slightly restructured to improve
clarity to readers.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "files" ODB transaction backend lazily creates a temporary object
directory when the first loose object is written to the transaction via
`prepare_loose_object_transaction()`. In a subsequent commit, the
temporary directory is used to also write packfiles to.
Rename the function to `odb_transaction_files_prepare()` accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A linker warning on macOS when building with Xcode 16.3 or newer has
been avoided by passing -fno-common to the compiler when a
sufficiently new linker is detected.
* hn/macos-linker-warning:
config.mak.uname: avoid macOS linker warning on Xcode 16.3+
In t3070-wildmatch, "via ls-files" test variants with patterns
containing backslash escapes are now skipped on Windows, avoiding 36
test failures caused by pathspec separator conversion.
* kk/wildmatch-windows-ls-files-prereq:
t3070: skip ls-files tests with backslash patterns on Windows
The documentation for "--word-diff" has been extended with a bit of
implementation detail of where these different words come from.
* mm/doc-word-diff:
doc: clarify that --word-diff operates on line-level hunks
A memory leak in `fetch_and_setup_pack_index()` when verification of
the downloaded pack index fails has been plugged. Also an obsolete
`unlink()` call on parse failure has been cleaned up.
* lp/http-fetch-pack-index-leak-fix:
http: fix memory leak in fetch_and_setup_pack_index()
http: cleanup function fetch_and_setup_pack_index()
The loose object source has been refactored into a proper `struct
odb_source`.
* ps/odb-source-loose:
odb/source-loose: drop pointer to the "files" source
odb/source-loose: stub out remaining callbacks
odb/source-loose: wire up `write_object_stream()` callback
object-file: refactor writing objects to use loose source
odb/source-loose: wire up `write_object()` callback
loose: refactor object map to operate on `struct odb_source_loose`
odb/source-loose: wire up `freshen_object()` callback
odb/source-loose: drop `odb_source_loose_has_object()`
odb/source-loose: wire up `count_objects()` callback
odb/source-loose: wire up `find_abbrev_len()` callback
odb/source-loose: wire up `for_each_object()` callback
odb/source-loose: wire up `read_object_stream()` callback
odb/source-loose: wire up `read_object_info()` callback
odb/source-loose: wire up `close()` callback
odb/source-loose: wire up `reprepare()` callback
odb/source-loose: start converting to a proper `struct odb_source`
odb/source-loose: store pointer to "files" instead of generic source
odb/source-loose: move loose source into "odb/" subsystem
The `git log -L` implementation has been refactored to use the
standard diff output pipeline, enabling pickaxe and diff-filter to
work as expected. Additionally, metadata-only diff formats like
--raw and --name-only are now supported with -L.
* mm/line-log-cleanup:
line-log: allow non-patch diff formats with -L
line-log: integrate -L output with the standard log-tree pipeline
revision: move -L setup before output_format-to-diff derivation
Correct use of sockaddr API in "git daemon".
* st/daemon-sockaddr-fixes:
daemon: guard NULL REMOTE_PORT in execute() logging
daemon: fix IPv6 address truncation in ip2str()
daemon: fix IPv6 address corruption in lookup_hostname()
The documentation for `push.default = simple` has been clarified to
better explain its behavior, making it clear that it pushes the
current branch to a same-named branch on the remote, and detailing
the upstream requirements for centralized workflows.
* ib/doc-push-default-simple:
doc: clarify push.default=simple behavior
The 'git-jump' command (in contrib/) has been taught to automatically
pick a mode (merge, diff, or ws) when invoked without arguments.
* gh/jump-auto-mode:
git-jump: pick a mode automatically when invoked without arguments
Formatting object name in full hexadecimal form has been optimized
by using a new strbuf_add_oid_hex() helper function.
* rs/strbuf-add-oid-hex:
hex: add and use strbuf_add_oid_hex()
Adding a decimal integer with strbuf_addf("%u") appears commonly;
they have been optimized by using a custom formatter.
* rs/strbuf-add-uint:
ls-tree: use strbuf_add_uint()
ls-files: use strbuf_add_uint()
cat-file: use strbuf_add_uint()
strbuf: add strbuf_add_uint()
"git push" learned to take a "remote group" name to push to, which
causes pushes to multiple places, just like "git fetch" would do.
* ua/push-remote-group:
push: support pushing to a remote group
remote: move remote group resolution to remote.c
remote: fix sign-compare warnings in push_cas_option
The "promisor.quiet" configuration variable was not used from
relevant submodules when commands like "grep --recurse-submodules"
triggered a lazy fetch, which has been corrected.
* th/promisor-quiet-per-repo:
promisor-remote: fix promisor.quiet to use the correct repository
Reachability bitmap generation has been significantly optimized. By
reordering tree traversal, caching object positions, and refining how
pseudo-merge bitmaps are constructed, the performance of "git repack
--write-midx-bitmaps" is improved, especially for large repositories
and when using pseudo-merges.
* tb/bitmap-build-performance:
pack-bitmap: build pseudo-merge bitmaps after regular bitmaps
pack-bitmap: remember pseudo-merge parents
pack-bitmap: sort bitmaps before XORing
pack-bitmap: cache object positions during fill
pack-bitmap: consolidate `find_object_pos()` success path
pack-bitmap: reuse stored selected bitmaps
pack-bitmap: check subtree bits before recursing
pack-bitmap: pass object position to `fill_bitmap_tree()`
A batch of documentation pages has been updated to use the modern
synopsis style.
* ja/doc-synopsis-style-again:
doc: convert git-imap-send synopsis and options to new style
doc: convert git-apply synopsis and options to new style
doc: convert git-am synopsis and options to new style
doc: convert git-grep synopsis and options to new style
doc: git bisect: clarify the usage of the synopsis vs actual command
doc: convert git-bisect to synopsis style
The check for non-stale commits in the priority queue used by
`paint_down_to_common` and `ahead_behind` has been optimized by
replacing an O(N) scan with an O(1) counter, yielding performance
improvements in repositories with wide histories.
* kk/commit-reach-optim:
commit-reach: replace queue_has_nonstale() scan with O(1) tracking
commit-reach: deduplicate queue entries in paint_down_to_common
object.h: fix stale entries in object flag allocation table
"git stash -p" has been optimized by reusing cached index
entries in its temporary index, avoiding unnecessary lstat()
calls on unchanged files.
* aj/stash-patch-optimize-temporary-index:
stash: reuse cached index entries in --patch temporary index
'git restore --staged' has been optimized to avoid unnecessarily expanding
the sparse index when operating on paths within the sparse checkout
definition, by handling sparse directory entries at the tree level.
* ds/restore-sparse-index:
restore: avoid sparse index expansion
t1092: test 'git restore' with sparse index
The GIT_WORK_TREE variable prepared to invoke the push-to-checkout
hook was leaking into the environment even when there was no hook
used and broke the default push-to-deploy (i.e., let "git checkout"
update the working tree only when the working tree is clean).
* ar/receive-pack-worktree-env:
receive-pack: fix updateInstead with core.worktree
Building on macOS with Xcode 16.3 or newer emits:
ld: warning: reducing alignment of section __DATA,__common
from 0x8000 to 0x4000 because it exceeds segment maximum
alignment
Pass -fno-common when "ld -v" reports ld-1167 or newer, so tentative
definitions of large arrays go into BSS instead of __DATA,__common.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "git pack-objects --path-walk" traversal has been integrated
with several object filters, including blobless and sparse filters.
* ds/path-walk-filters:
path-walk: support `combine` filter
path-walk: support `object:type` filter
path-walk: support `tree:0` filter
t6601: tag otherwise-unreachable trees
pack-objects: support sparse:oid filter with path-walk
path-walk: add pl_sparse_trees to control tree pruning
path-walk: support blob size limit filter
backfill: die on incompatible filter options
path-walk: support blobless filter
path-walk: always emit directly-requested objects
t/perf: add pack-objects filter and path-walk benchmark
pack-objects: pass --objects with --path-walk
t5620: make test work with path-walk var
"Friday noon" asked in the morning on Sunday was parsed to be one
day before the specified time, which has been corrected.
* ta/approxidate-noon-fix:
approxidate: use deferred mday adjustments for "specials"
approxidate: make "specials" respect fixed day-of-month
t0006: add support for approxidate test date adjustment
approxidate: make "today" wrap to midnight
The "name" argument in git_connect() and related functions has been
converted to a "service" enum to improve type safety and clarify its
purpose.
* jk/connect-service-enum:
transport-helper: fix typo in BUG() message
connect: use "service" enum for "name" argument
Inside the function `fetch_and_setup_pack_index()`, when the pack
obtained using `parse_pack_index()` fails to be verified by
`verify_pack_index()`, the function returns without closing and freeing
said pack.
Fix this by calling `close_pack_index()` to munmap the index file for
the leaking pack (which might have been mmapped by `fetch_pack_index()`
or `verify_pack_index()`), and then free it, when the verification
fails.
Signed-off-by: LorenzoPegorari <lorenzo.pegorari2002@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Cleanup the function `fetch_and_setup_pack_index()` by removing the
useless call to the function `unlink()`.
This is not necessary anymore since 63aca3f7f1 (dumb-http: store
downloaded pack idx as tempfile, 2024-10-25), when `fetch_pack_index()`
started registering its return value (in this case `tmp_idx`) as a
tempfile to be deleted at process exit.
Signed-off-by: LorenzoPegorari <lorenzo.pegorari2002@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Now that all callbacks of the loose source operate on `struct
odb_source_loose` directly we no longer have to reach into the "files"
source at all.
Drop this field and update `odb_source_loose_new()` to instead accept
all parameters required to initialize itself. This ensures that the
"loose" backend is a fully standalone source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Stub out remaining callback functions for the "loose" backend.
Note that we also stub out transactions for loose objects. In fact, we
already have the infrastructure in place for those, and we could in
theory implement those, as well. But there are separate efforts ongoing
to polish up transactional interfaces, and doing so now would likely
result in some messiness. This omission will thus be worked on in a
subsequent patch series, once the dust has settled.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Wire up the `write_object_stream()` callback.
Note that we don't move the implementation into "odb/source-loose.c".
This is because most of the logic to write loose objects is still
contained in "object-file.c", and detangling that requires us to do some
refactorings as explained in the preceding commit. So for now, the
implementation of writing an object stream is still located in
"object-file.c".
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "object-file" subsystem still hosts the majority of logic used to
write loose objects. Eventually, we'll want to move this logic into
"odb/source-loose.c", but this isn't yet easily possible because a lot
of the writing logic is still being shared with `force_object_loose()`.
We will eventually detangle this logic so that we can indeed move all of
it into the "loose" source. Meanwhile though, refactor the code so that
it operates on a `struct odb_source_loose` directly to already make the
dependency explicit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move `odb_source_loose_write_object()` from "object-file.c" into
"odb/source-loose.c" and wire it up as the `write_object()` callback of
the loose source.
As in preceding commits, this requires us to expose a couple of generic
functions from "object-file.c" as they are used in both subsystems now.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
While the loose object map functions in "loose.c" accept a generic
`struct odb_source *`, they always expect this to be the "files"
backend. Furthermore, the subsystem doesn't even care about the "files"
backend, but only uses it as a stepping stone to get to the "loose"
backend.
This assumption is implicit and thus not immediately obvious. Refactor
the interfaces to instead operate on a `struct odb_source_loose`
instead, which eliminates the implicit dependency and unnecessary detour
via the "files" source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move `odb_source_loose_freshen_object()` from "object-file.c" into
"odb/source-loose.c" and wire it up as the `freshen_object()` callback
of the loose source.
As part of the move, `check_and_freshen_source()` is inlined into the
callback function, as it has no other callers anymore.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Justin Tobler