A new 'diff.<driver>.process' configuration has been introduced to
allow a long-running external process to act as a hunk provider,
enabling external tools to control which lines Git considers changed
while leaving all output formatting (word diff, color, blame, etc.) to
Git's standard pipeline.
* mm/diff-process-hunks:
fixup! diff: consult oid-only hunk providers via diff.<driver>.process
diff: consult oid-only hunk providers via diff.<driver>.process
userdiff: add diff.<driver>.process config
sub-process: add a gentle status read
sub-process: separate process lifecycle from hashmap management
blame: read precomputed hunks
diff: read precomputed hunks for stat output
diff: record precomputed hunks during stat output
diff-hunks: add the store format, library, and command
diff: introduce a hunk provider interface
gitattributes: document how external diff drivers relate to diff features
The build-time knob 'USE_NSEC' for nanosecond stat precision has been
converted to a runtime configuration 'core.useNanosec', allowing
distributions to bundle one binary that adapts to filesystem
capabilities dynamically.
* dk/use-nsec-runtime:
core: convert build-time USE_NSEC into runtime core.useNanosec
environment: align repo_config_values_init with struct declaration
meson: expose knob for xmlto relative links in manuals
Racy Git problems persist today, manifesting themselves in the
performance of commands like "git diff" in new worktrees [1]. We have
long had a build knob "USE_NSEC" to tell Git to use in-core nanosecond
precision when available, which mitigates most if not all racy issues,
but most builds we know about don't use it. In part, that's because
someone distributing Git can't safely enable it at compile-time if they
don't know exactly what platforms their distribution will be used on.
[1]: https://lore.kernel.org/git/CALnO6CADMJSixqYvL1Yo8qKX5rWhKQ+2OoSEuPUh-yoeK9TseQ@mail.gmail.com
These days, most platforms are likely to be safe for the USE_NSEC code.
Regardless, we want to give users the ability to benefit from it. This
requires exposing the compile-time gated code as a runtime option.
In addition, update the Racy Git documentation and other mentions of
USE_NSEC in the code.
Due to the conversion from #ifdef to runtime check, using the flag
"--ignore-space-change" may be particularly helpful when viewing changes
from this patch.
Signed-off-by: D. Ben Knoble <ben.knoble@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The codebase has been updated to use the newer utimensat() POSIX
function instead of the obsolescent utime(), allowing
high-precision timestamps while preserving fallback compatibility.
* as/utimensat-utimes:
compat/posix: drop legacy <utime.h> header and shims
treewide: use utimensat(2) instead of legacy utime(3p)
compat/posix: introduce utimensat(2) wrapper
A compatibility workaround has been introduced for macOS to address
a memory leak in the system regex engine when it encounters invalid
multibyte sequences. The workaround segments the input buffer at
invalid byte boundaries and searches each valid segment separately
using regexec(), avoiding the leaking path.
* cl/regexec-macos-leak:
SQUASH???
regexec: work around macOS TRE leak on invalid UTF-8
In POSIX.1-2008, utime(3p) was marked as obsolescent in favor of
utimensat(2) and futimens(2). In the recent POSIX.1-2024 (Issue 8)
specification, <utime.h> and utime(3p) were officially removed.
utimensat(2) operates on `struct timespec` rather than the second-only
`struct utimbuf`, allowing sub-second timestamp updates while also
providing support for UTIME_NOW and UTIME_OMIT flags to selectively
update or preserve individual access and modification timestamps.
Introduce a compatibility layer for utimensat(2):
- Provide fallback definitions for AT_FDCWD, UTIME_NOW, and UTIME_OMIT
in case the system headers lack them.
- Introduce `ST_ATIME_NSEC(st)` to complement `ST_MTIME_NSEC(st)` and
`ST_CTIME_NSEC(st)`.
- Implement `git_utimensat()` in `compat/utimensat.c` as a fallback using
utimes(2) on platforms that define NO_UTIMENSAT.
- Implement `mingw_utimensat()` in `compat/mingw.c` converting `struct
timespec` to Windows FILETIME with 100ns precision.
- Wire up NO_UTIMENSAT support in Makefile, meson.build,
contrib/buildsystems/CMakeLists.txt, and configure.ac.
Subsequent commits will migrate callers across the codebase to
utimensat(2) and drop the legacy <utime.h> header.
Signed-off-by: Alexey Samsonov <vonosmas@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A compatibility wrapper for writev(3p) has been reintroduced,
including fixes for CMake build and 'MAX_IO_SIZE' limits on NonStop.
Calls to write(3p) in send_sideband() and cat_blob() have been
refactored to use writev(3p) wrappers to reduce syscall overhead.
* ps/writev:
fast-import: use writev(3p) to send cat-blob responses
sideband: use writev(3p) to send pktlines
wrapper: properly handle MAX_IO_SIZE in writev(3p)
wrapper: introduce writev(3p) wrappers
compat/posix: introduce writev(3p) wrapper
In a subsequent commit we're going to add the first caller to
writev(3p). Introduce a compatibility wrapper for this syscall that we
can use on systems that don't have this syscall.
The syscall exists on modern Unixes like Linux and macOS, and seemingly
even for NonStop according to [1]. It doesn't seem to exist on Windows
though.
[1]: http://nonstoptools.com/manuals/OSS-SystemCalls.pdf
[2]: https://www.gnu.org/software/gnulib/manual/html_node/writev.html
Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The provider chain so far holds the diff-hunks store in front of the
terminal builtin computation. Open it to external processes: a pair on
a path whose driver configures diff.<driver>.process is answered by a
long-running process speaking a pkt-line protocol (following the filter
process protocol), registered at the head of the chain and consulted
before the store and before any blob is loaded.
The protocol starts with the smallest request that can carry an answer:
object names alone. A request is the pathname and the pair's
old-oid/new-oid, with no content. The process answers with hunk lines,
with a zero-hunk success that asserts the blobs equivalent (trailing
newlines included), or with status=need-content, on which the pair
falls through to the builtin answer. This serves the two shapes that
need no content pushed to them: a cache keyed on the blob pair, and a
process that fetches the blobs itself (for example over "git cat-file
--batch"). A pair whose side is not a stored blob carries a NULL id;
the provider sends no request and passes it. Because Git holds no
content for the exchange, the answer is used as sent: hunks are
validated for order, overlap, lockstep alignment, and magnitude, then
replayed without the normalization xdiff applies to diffs it computes
itself. The magnitude bound is the blobs' sizes, read from the object
database without loading content: a blob of N bytes holds at most N
lines.
Because the process's answer is authoritative, it outranks the store,
and its head-of-chain position says so. A pair the process answers
never reaches the store and is never recorded, so nothing it produces
enters the store, which holds the builtin answer only. A request it
does not answer, whether need-content, a missing capability, or a
missing id, passes down the chain to the builtin answer, which is what
the store serves, so the store may serve such a pair and a warming run
may record it. Entries recorded before a process was configured are
not purged; a pair the process answers ignores them, and "git diff-hunks
clear" discards them.
The provider gates itself per request. The driver is looked up by the
old-side path, so a renamed file resolves to the same driver, and by
the repository-relative path, so a diff.relative run from a
subdirectory names the pair the same way. Options the process is never
told about select no process: the whitespace-ignoring options, -I,
--anchored, and an algorithm forced by option or configuration (blame
routes its algorithm through xdl_opts, so --histogram is covered). The
request gains its last field, the path; the consumers change only by
filling it, and neither names the process.
The provider's state is its repository's pool of running processes,
keyed by the configured command, so drivers sharing a command share a
process, a submodule speaks to its own, and releasing the provider
(from repo_clear()) stops them. The pool owns a copy of each command
string, so an entry outlives a config re-read. A command that fails
stays as an entry that is not retried: its request and every later one
pass, so the store may serve the path for the rest of the command.
A protocol error in a response never kills the command. The response
is read through a packet reader gentle about framing, so an error takes
one path: a single warning, the process stopped and marked failed, and
the builtin diff for the rest of the command. That covers garbage
bytes, a truncated response, an empty packet, a bare status, and an
unrecognized status. Semantically invalid coordinates cost only their
pair: the response is drained, the pair is computed, and the process
stays alive. A path the protocol cannot carry (an embedded newline, or
one too long for a packet) falls back per path rather than costing the
command its process. The handshake keeps one fatal check: a process
that announces a capability Git did not request aborts the command, as
the long-running filter protocol does.
Consulting is allowed per command, following the allow_textconv
precedent. "git diff", "git log" and "git show", and "git blame" set
allow_diff_process; the plumbing diff commands and the interactive-patch
machinery never set it, so scripted and staging output stays builtin.
The options adjust the flag:
- --no-ext-diff clears it and --ext-diff sets it;
- --diff-process and --no-diff-process set and clear it alone, leaving
external diff drivers as they were;
- format-patch clears it unconditionally, so a generated patch applies
for recipients without the process;
- range-diff passes --no-ext-diff to the "git log" it compares.
git blame and the summary formats consult the process. For blame, a
pair reported equivalent emits no hunks, so the whole commit passes to
its parent. In the stat formats such a pair sums to a zero-count entry,
which the "nothing changed" rule omits, as under -w. The subprocess is
long-running: one startup cost across a traversal, one round-trip per
consulted pair. Answers travel in struct xdl_hunk, new in
xdiff-interface.h, holding xdiff's 1-based coordinates; nothing feeds
them back to xdiff, since only coordinate consumers consult.
A content-carrying request is the natural extension: it would serve
sides that are not stored blobs and processes that want content pushed
to them, and bring patch output and log -L's range tracking to the same
answer. As it stands, a process's answers show in blame and the summary
formats while patch output stays builtin.
t4080 exercises the protocol, the per-command gate, and the error paths:
- each adversarial response shape warns and falls back to builtin, the
request log proving which failures disable the process and which keep
it alive (a malformed hunk line, coordinates past the blob size, a
count overflowing strtol(), overlapping or misaligned hunks, an
unrecognized status, a bare status, an empty packet, a mid-response
crash, and raw garbage);
- a capability-less process and status=abort degrade without noise, and
a failed start warns once and returns the path to the store;
- a trailing token on a hunk line is ignored, pinning field
appendability;
- positive consults for git diff, git show, and diff-tree under
--ext-diff and --diff-process; textconv output and gitlink sides are
never identified; a diff.relative run consults by the repo-relative
path;
- the equivalence answer is pinned from both consumers, and a warming
run past a deferring process records the pair for a later read.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Blame and "git log --stat" recover hunk coordinates by diffing blob
pairs, and recompute them on every run. Add a cache of those
coordinates at $GIT_DIR/objects/info/diff-hunks, beside the
commit-graph, so a later run can look them up instead of decompressing
the blobs and running xdiff again.
The store is a single chunk-format file (see gitformat-chunk(5)): an
8-byte header, a DHIX index of fixed-size entries sorted by key, a DHDT
segment of hunk records, and a trailing hash checksum. An entry is
keyed by the two blob object ids and the xdl_opts the pair was diffed
under, so a stored result is served only where that exact key recurs,
independent of path. A zero-context diff trims unchanged lines from
hunk edges and can pick a different but equally valid set of hunks than
an untrimmed diff, so a recording caller stores a pair only when its
trimmed and untrimmed diffs are identical; such an entry answers any
consumer at any context, and the rare divergent pair is always
computed. Identical hunk blocks are interned once and shared across
keys.
The library provides a reader (repo_diff_hunks_store and _replay, gated
by core.diffHunks), loaded once and cached on the object database as the
commit-graph is, and a writer that accumulates entries and flushes them
in one atomic pass. An absent, corrupt, or disabled store reads as all
misses. A record with no hunks is invalid too: replaying it would claim
the pair equivalent, which the store never asserts, so it reads as a
miss.
Ordinary reads are diagnostic-free. Loading parses the chunk table
through read_table_of_contents_quiet(), new in chunk-format, which
prints nothing on a malformed table and takes the repository's hash
algorithm rather than the_hash_algo, so the file is bounds-checked under
the algorithm it is keyed by.
The flush closes the repository's mmapped store and forgets that loading
was attempted before committing the lockfile. A warming run that also
reads may hold the file it is replacing mapped, and the rename must not
land on a live mapping, which Windows refuses; a read after the flush
then observes the committed file. commit-graph closes its graph before
committing for the same reason.
Writing is off by default, enabled per run by GIT_DIFF_HUNKS_WRITE or
persistently by diffHunks.write, the environment winning. A writer
seeds from the existing store, so a flush merges rather than replaces.
The seed's checksum is verified first: a corrupt store is discarded, not
rewritten with a fresh checksum verify could no longer catch. An entry
that fails the shared diff_provider_check_hunk() or names no blob is
dropped with a warning, since it would only ever read as a miss. A seed
that discarded or dropped anything forces the flush even when the
warming run computed nothing new. The writer fsyncs through a new
diff-hunks core.fsync component.
"git diff-hunks" inspects and manages the file: "verify" checks the
checksum, chunk table, sort order, entry bounds, and every entry's hunk
sequence against that shared check, so a store whose entries could only
read as misses fails verify; "clear" removes the file. Later patches
wire the readers and the writer into the diff and blame paths.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
To learn which line ranges changed between two blobs, every consumer in
the diff machinery loads both blobs and runs xdiff. There is no other
way to supply that answer, even when it is known elsewhere: a cache may
hold the ranges from the last time the pair was diffed, and a
format-aware process may have its own idea of which lines changed.
Either could answer from the blob object ids alone, but the loading and
computing are hard-wired into each consumer, so such an answer has no
place to enter.
Introduce the hunk provider interface, diff-provider.h, between asking
the question and computing the answer. A provider answers a request
made of the pair's identity, its blob object ids and the parameters
that determine the diff. A provider is either authoritative, so its
answer may deliberately differ from the builtin diff, or not, so its
answer must reproduce the builtin result exactly. Every answer served
from identity passes diff_provider_check_hunk() before a consumer sees
it: coordinates fit int32, hunks are ordered and non-overlapping, and
the unchanged runs between them match on both sides. A failing answer
is discarded and the pair falls through as unanswered.
Providers are repository-lifecycle objects. Each repository owns a
chain of them, built on first consultation and released from
repo_clear(), so a submodule gets its own providers and no provider
state outlives the repository it serves. The chain has a fixed
composition, and each provider gates itself per request, passing when
it does not apply. Chain order is the authority: the first answer
wins. A provider may instead refuse a pair whose request is shaped by
parameters its recording key cannot express. After a refusal, no later
provider answers the pair from identity, and the consumer must not
record what it computes for it. The last provider is the builtin
computation, the only one that computes rather than answering from
identity, so a walk given a fill callback always ends in an answer,
refusal or not.
The walk in diff-provider.c maps a provider's four dispositions
(answer, pass, fail, refuse) onto the consumer-facing outcomes, and
checks with BUG() that only the computing provider fails and that it
passes on a walk with no fill callback. The implementor contract, the
provider struct, its dispositions, and the shared check, lives in
diff-provider-internal.h, as refs/refs-internal.h is to refs.h;
consumers see only diff-provider.h.
The consumer surface is two types. struct diff_provider_request names
what is diffed and under which parameters; each later commit that
consults on more state adds the field it keys on (the object ids and
diff options, then the path). enum diff_provider_outcome flattens two
dependent axes into four points: the response state (answered,
unanswered, failed) and, only when unanswered, whether the caller may
record what it computes. The record rule rides in the outcome, not a
separate flag, so -Wswitch forces every consumer to place the no-record
arm. A provider added later maps onto these values inside the walk, so
consumer code is written once.
diff_provider_emit_hunks() is the consumer entry: the caller states the
request, a hunk callback, and a content-loading callback that reaches
the terminal provider only when the ranges are computed. Blame's
pass_blame_to_parent() is the first consumer, since it knows both blob
ids before reading either blob; its loads move into the fill callback.
With only the terminal provider registered, every request still
computes, so behavior is unchanged. (Blame's -C/-M split detection
diffs partial buffers with no blob identity and stays on xdi_diff().)
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The 'remote-object-info' command has been added to 'git cat-file
--batch-command', allowing clients to request object metadata
(currently size) from a remote server via protocol v2 without
downloading the entire object. Format placeholders are dynamically
filtered on the client based on server-advertised capabilities,
returning empty strings for inapplicable or unsupported fields.
* ps/cat-file-remote-object-info:
cat-file: make remote-object-info allow-list adapt to the server
cat-file: add remote-object-info to batch-command
transport: add client support for object-info
serve: advertise object-info feature
protocol-caps: check object existence regardless of the attributes requested
fetch-pack: move fetch initialization
connect: make write_fetch_command_and_capabilities() more generic
fetch-pack: move write_fetch_command_and_capabilities() to connect.c
fetch-pack: use unsigned int for hash_algo variable
fetch-pack: drop the static advertise_sid variable
t1006: extract helper functions into new 'lib-cat-file.sh'
cat-file: declare loop counter inside for()
transport-helper: fix memory leak of helper on disconnect
On macOS, the system regex engine leaks an internal buffer when
regexec() encounters an invalid multibyte sequence in a UTF-8 locale.
The line-by-line path can call regexec_buf() for each pattern on every
line, so "git grep" can leak repeatedly on a file containing invalid
UTF-8. The total leak grows with the number of calls, and the per-call
allocation grows with the pattern's automaton. In one case, grepping a
repository containing PDFs exhausted memory and caused the machine to
restart.
ce025ae4f6 (grep: disable lookahead on error, 2024-10-20) made "git
grep" fall back to line-by-line matching when regexec() reports an error
on invalid UTF-8. That fallback cannot prevent this leak: the allocation
has already leaked when regexec() returns REG_ILLSEQ.
Avoid the leaking path by providing a Darwin-specific regexec_buf().
Walk the input with mbrtowc(), split it at bytes that cannot form a
complete multibyte character, and search each valid segment separately.
This preserves matches in valid text on either side of an invalid byte.
Search each segment with REG_STARTEND so match offsets remain relative to
the original buffer. Set REG_NOTBOL and REG_NOTEOL for internal segment
boundaries so "^" and "$" do not match there. Keep the flags clear at
the true beginning and end of the buffer.
Use the normal regexec_buf() path in single-byte locales, where no byte
can form an invalid multibyte sequence. Use the bundled regex
implementation unchanged when NO_REGEX is enabled.
Declare the Darwin override in compat/darwin.h and map regexec_buf() to
darwin_regexec_buf(). This follows the platform override pattern used
by the other compatibility headers and leaves the common inline
implementation as the default.
There is no reliable way to detect a future macOS version in which the
system regex implementation has been fixed. Even after a fix, Git will
need the workaround while it supports affected macOS releases, so treat
it as an indefinite compatibility workaround.
Add tests for matches before, after, and between invalid bytes, including
an offset check after an invalid byte. Also check incomplete trailing
input and anchors at true and internal line boundaries.
Signed-off-by: Chungmin Lee <chungmin@chungminlee.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Sometimes, it is beneficial to retrieve information about an object
without downloading it entirely. The server-side logic for this
functionality was implemented in commit "a2ba162cda (object-info:
support for retrieving object info, 2021-04-20)." And the wire
format is documented at
https://git-scm.com/docs/protocol-v2#_object_info.
Introduce client-side support for the object-info capability.
Add its own function for object-info separate from existing fetch
infrastructure.
Currently, the client supports requesting a list of OIDs with the size
attribute from a v2 server. If the server does not advertise this
feature (i.e., transfer.advertiseobjectinfo is set to false), the client
returns an error and exits.
Note that:
1. The entire request is written into req_buf before being sent to the
remote. This approach follows the pattern used in the
send_fetch_request() logic within 'fetch-pack.c'. Streaming the
request is not addressed in this patch.
2. A new field 'unrecognized' has been added to object_info. This new
field is set at fetch_object_info() when the object is unrecognized
by the server.
Helped-by: Jonathan Tan <jonathantanmy@google.com>
Helped-by: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Calvin Wan <calvinwan@google.com>
Signed-off-by: Eric Ju <eric.peijian@gmail.com>
Signed-off-by: Pablo Sabater <pabloosabaterr@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The build system has been updated to support building universal macOS
binaries when 'Rust' is enabled, by compiling separate static archives
for each target triple listed in 'RUST_TARGETS' and combining them
using the macOS 'lipo' tool. The 'git-credential-osxkeychain' helper
has been updated to link against '$(RUST_LIB)' when 'Rust' is enabled.
* sn/osxkeychain-rust-universal:
contrib: wire up osxkeychain in contrib/Makefile on macOS
Makefile: support universal macOS builds via RUST_TARGETS
Makefile: add $(RUST_LIB) prerequisite to osxkeychain
The 'reftable' code has been hardened against corrupted tables by
fixing out-of-bounds writes, out-of-bounds reads, and abort calls
during parsing.
* ps/reftable-hardening:
reftable/table: fix OOB read on truncated table
reftable/table: fix NULL pointer access when seeking to bogus offsets
reftable/block: fix OOB read with bogus restart offset
reftable/block: fix use of uninitialized memory when binsearch fails
reftable/block: fix OOB read with bogus restart count
reftable/block: fix OOB read with bogus block size
reftable/block: fix OOB write with bogus inflated log size
t/unit-tests: introduce test helper to write reftable blocks
reftable/record: don't abort when decoding invalid ref value type
reftable/basics: fix OOB read on binary search of empty range
oss-fuzz: add fuzzer for parsing reftables
meson: support building fuzzers with libFuzzer
On macOS, Universal Binaries contain native executable code for
multiple architectures (such as Intel x86_64 and Apple Silicon arm64)
bundled into a single file. This is standard practice for macOS
distribution and CI packaging (such as internal distribution packages
or tooling like Burrito/Homebrew), allowing a single build artifact
to run natively across all Macs without Rosetta emulation or
maintaining separate packages.
When building Git C code for multiple architectures on macOS, the
Apple toolchain (clang) natively supports universal builds via
CFLAGS/LDFLAGS. When "-arch x86_64 -arch arm64" is passed, clang
automatically compiles and links universal binaries for all C object
files and executables out of the box.
Cargo and rustc, however, do not support multiple "-arch" flags or
emitting universal binaries in a single invocation. Instead, Cargo
requires invoking each target triple independently (e.g., passing
"--target x86_64-apple-darwin" and "--target aarch64-apple-darwin").
To bridge this gap when Rust is enabled:
1. Allow specifying space-separated target triples in RUST_TARGETS.
2. Introduce declarative pattern rules (target/%/...) to compile
each target-specific library slice via Cargo.
3. On macOS, if multiple targets are specified, use "lipo" (part of
the mandatory Xcode Command Line Tools) to combine the resulting
static libraries into target/release/libgitcore.a.
Once $(RUST_LIB) is compiled into a universal static archive, the
standard C linker seamlessly links it with the C object files to
produce universal Git executables.
Signed-off-by: Shardul Natu <snatu@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When Rust is enabled, the git-credential-osxkeychain helper depends on
Rust symbols compiled into $(RUST_LIB). While commit 522ea8ef7d
("osxkeychain: fix build with Rust") updated the linker command line to
use $(LIBS), it omitted $(RUST_LIB) from the target prerequisite list.
Without this prerequisite, running a parallel build ("make -j") from a
clean working tree can fail because Make does not know to invoke Cargo
to build libgitcore.a before linking git-credential-osxkeychain.
Note that we depend explicitly on $(LIB_FILE) and $(RUST_LIB) rather
than $(GITLIBS). Unlike standard Git builtins and programs like scalar
(which define cmd_main() and rely on common-main.o to supply main()),
git-credential-osxkeychain.c defines its own standalone int main().
If $(GITLIBS) were used, $(filter %.o,$^) in the link recipe would
match both git-credential-osxkeychain.o and common-main.o, causing a
duplicate symbol linking error for _main on macOS.
Additionally, wrap the definitions of $(RUST_LIB) and the "rust" build
target in "ifndef NO_RUST". This ensures that when NO_RUST=1 is
specified, $(RUST_LIB) evaluates to empty, making the Rust dependency a
clean no-op without needing intermediate variables.
Signed-off-by: Shardul Natu <snatu@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The packed object source has been refactored into a proper struct
odb_source.
* ps/odb-source-packed:
odb/source-packed: drop pointer to "files" parent source
midx: refactor interfaces to work on "packed" source
odb/source-packed: stub out remaining functions
odb/source-packed: wire up `freshen_object()` callback
odb/source-packed: wire up `find_abbrev_len()` callback
odb/source-packed: wire up `count_objects()` callback
odb/source-packed: wire up `for_each_object()` callback
odb/source-packed: wire up `read_object_stream()` callback
odb/source-packed: wire up `read_object_info()` callback
packfile: use higher-level interface to implement `has_object_pack()`
odb/source-packed: wire up `reprepare()` callback
odb/source-packed: wire up `close()` callback
odb/source-packed: start converting to a proper `struct odb_source`
odb/source-packed: store pointer to "files" instead of generic source
packfile: move packed source into "odb/" subsystem
packfile: split out packfile list logic
packfile: rename `struct packfile_store` to `odb_source_packed`
Add a new fuzzer that exercises our parsing of reftables. Fallout from
this fuzzer will be fixed over subsequent commits.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In subsequent patches we'll be turning `struct odb_source_packed` into a
proper `struct odb_source`. As a first step towards this goal, move its
struct out of "packfile.{c,h}" and into "odb/source-packed.{c,h}".
This detaches the implementation of the packfile object source from the
generic packfile code, following the same convention already used by the
"files" and "in-memory" sources.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the next commit we're about to introduce the "packed" object database
source. This source will embed a packfile list, and consequently we'll
have to include "packfile.h" to make the struct definition available.
This will unfortunately lead to a cyclic dependency that we cannot
resolve with a forward declaration.
Split out the code that relates to the packfile list into a separate
compilation unit so that both "packfile.h" and "odb/source-packed.h" can
include it.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Without NO_RUST defined, the varint encoder/decoder lives in the
RUST_LIB, which needs to be linked. Symptom:
cc [... -o contrib/credential/osxkeychain/git-credential-osxkeychain [...]
Undefined symbols for architecture x86_64:
"_decode_varint", referenced from:
_read_untracked_extension in libgit.a[x86_64][63](dir.o)
_read_untracked_extension in libgit.a[x86_64][63](dir.o)
_read_one_dir in libgit.a[x86_64][63](dir.o)
_read_one_dir in libgit.a[x86_64][63](dir.o)
_load_cache_entry_block in libgit.a[x86_64][174](read-cache.o)
"_encode_varint", referenced from:
_write_untracked_extension in libgit.a[x86_64][63](dir.o)
_write_untracked_extension in libgit.a[x86_64][63](dir.o)
_write_untracked_extension in libgit.a[x86_64][63](dir.o)
_write_one_dir in libgit.a[x86_64][63](dir.o)
_write_one_dir in libgit.a[x86_64][63](dir.o)
_do_write_index in libgit.a[x86_64][174](read-cache.o)
ld: symbol(s) not found for architecture x86_64
While it is curious why these functions are needed at all (osxkeychain
does not read or write the index), the compile error is a real problem.
Instead of trying to play games to add `GITLIBS` while filtering out
`common-main.o`, replace the `$(LIB_FILE) $(EXTLIBS)` construct with the
much shorter `$(LIBS)` construct that _already_ filters out
`common-main.o` and adds the Rust library when needed.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The loose object source has been refactored into a proper `struct
odb_source`.
* ps/odb-source-loose:
odb/source-loose: drop pointer to the "files" source
odb/source-loose: stub out remaining callbacks
odb/source-loose: wire up `write_object_stream()` callback
object-file: refactor writing objects to use loose source
odb/source-loose: wire up `write_object()` callback
loose: refactor object map to operate on `struct odb_source_loose`
odb/source-loose: wire up `freshen_object()` callback
odb/source-loose: drop `odb_source_loose_has_object()`
odb/source-loose: wire up `count_objects()` callback
odb/source-loose: wire up `find_abbrev_len()` callback
odb/source-loose: wire up `for_each_object()` callback
odb/source-loose: wire up `read_object_stream()` callback
odb/source-loose: wire up `read_object_info()` callback
odb/source-loose: wire up `close()` callback
odb/source-loose: wire up `reprepare()` callback
odb/source-loose: start converting to a proper `struct odb_source`
odb/source-loose: store pointer to "files" instead of generic source
odb/source-loose: move loose source into "odb/" subsystem
* ps/odb-source-loose:
odb/source-loose: drop pointer to the "files" source
odb/source-loose: stub out remaining callbacks
odb/source-loose: wire up `write_object_stream()` callback
object-file: refactor writing objects to use loose source
odb/source-loose: wire up `write_object()` callback
loose: refactor object map to operate on `struct odb_source_loose`
odb/source-loose: wire up `freshen_object()` callback
odb/source-loose: drop `odb_source_loose_has_object()`
odb/source-loose: wire up `count_objects()` callback
odb/source-loose: wire up `find_abbrev_len()` callback
odb/source-loose: wire up `for_each_object()` callback
odb/source-loose: wire up `read_object_stream()` callback
odb/source-loose: wire up `read_object_info()` callback
odb/source-loose: wire up `close()` callback
odb/source-loose: wire up `reprepare()` callback
odb/source-loose: start converting to a proper `struct odb_source`
odb/source-loose: store pointer to "files" instead of generic source
odb/source-loose: move loose source into "odb/" subsystem
In subsequent patches we'll be turning `struct odb_source_loose` into a
proper `struct odb_source`. As a first step towards this goal, move its
struct out of "object-file.c" and into "odb/source-loose.c".
This detaches the implementation of the loose object source from the
generic object file code, following the same convention already used by
the "files" and "in-memory" sources.
No functional changes are intended.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The fsmonitor daemon has been implemented for Linux.
* pt/fsmonitor-linux:
fsmonitor: convert shown khash to strset in do_handle_client
fsmonitor: add tests for Linux
fsmonitor: add timeout to daemon stop command
fsmonitor: close inherited file descriptors and detach in daemon
run-command: add close_fd_above_stderr option
fsmonitor: implement filesystem change listener for Linux
fsmonitor: rename fsm-settings-darwin.c to fsm-settings-unix.c
fsmonitor: rename fsm-ipc-darwin.c to fsm-ipc-unix.c
fsmonitor: use pthread_cond_timedwait for cookie wait
compat/win32: add pthread_cond_timedwait
fsmonitor: fix hashmap memory leak in fsmonitor_run_daemon
fsmonitor: fix khash memory leak in do_handle_client
t9210, t9211: disable GIT_TEST_SPLIT_INDEX for scalar clone tests
Add a new odb "in-memory" source that is meant to only hold
tentative objects (like the virtual blob object that represents the
working tree file used by "git blame").
* ps/odb-in-memory:
t/unit-tests: add tests for the in-memory object source
odb: generic in-memory source
odb/source-inmemory: stub out remaining functions
odb/source-inmemory: implement `freshen_object()` callback
odb/source-inmemory: implement `count_objects()` callback
odb/source-inmemory: implement `find_abbrev_len()` callback
odb/source-inmemory: implement `for_each_object()` callback
odb/source-inmemory: convert to use oidtree
oidtree: add ability to store data
cbtree: allow using arbitrary wrapper structures for nodes
odb/source-inmemory: implement `write_object_stream()` callback
odb/source-inmemory: implement `write_object()` callback
odb/source-inmemory: implement `read_object_stream()` callback
odb/source-inmemory: implement `read_object_info()` callback
odb: fix unnecessary call to `find_cached_object()`
odb/source-inmemory: implement `free()` callback
odb: introduce "in-memory" source
The internal URL parsing logic has been made accessible via a new
subcommand "git url-parse".
* mm/git-url-parse:
t9904: add tests for the new url-parse builtin
doc: describe the url-parse builtin
builtin: create url-parse command
urlmatch: define url_parse function
url: return URL_SCHEME_UNKNOWN instead of dying
url: move scheme detection to URL header/source
url: move url_is_local_not_ssh to url.h
connect: rename enum protocol to url_scheme
Stop using unmaintained custom allocator in Windows build which was
the last user of the code.
* js/mingw-no-nedmalloc:
mingw: remove the vendored compat/nedmalloc/ subtree
mingw: drop the build-system plumbing for nedmalloc
mingw: stop using nedmalloc
Update code paths that assumed "unsigned long" was long enough for
"size_t".
* js/objects-larger-than-4gb-on-windows:
ci: run expensive tests on push builds to integration branches
t5608: mark >4GB tests as EXPENSIVE
test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1
test-tool synthesize: precompute pack for 4 GiB + 1
test-tool synthesize: use the unsafe hash for speed
t5608: add regression test for >4GB object clone
test-tool: add a helper to synthesize large packfiles
delta, packfile: use size_t for delta header sizes
odb, packfile: use size_t for streaming object sizes
git-zlib: handle data streams larger than 4GB
index-pack, unpack-objects: use size_t for object size
A new builtin "git format-rev" is introduced for pretty formatting
one revision expression per line or commit object names found in
running text.
* kh/name-rev-custom-format:
format-rev: introduce builtin for on-demand pretty formatting
name-rev: make dedicated --annotate-stdin --name-only test
name-rev: factor code for sharing with a new command
name-rev: run clang-format before factoring code
name-rev: wrap both blocks in braces
While the in-memory object source is a full-fledged source, our code
base only exercises parts of its functionality because we only use it in
git-blame(1). Implement unit tests to verify that the yet-unused
functionality of the backend works as expected.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Next to our typical object database sources, each object database also
has an implicit source of "cached" objects. These cached objects only
exist in memory and some use cases:
- They contain evergreen objects that we expect to always exist, like
for example the empty tree.
- They can be used to store temporary objects that we don't want to
persist to disk, which is used by git-blame(1) to create a fake
worktree commit.
Overall, their use is somewhat restricted though. For example, we don't
provide the ability to use it as a temporary object database source that
allows the user to write objects, but discard them after Git exists. So
while these cached objects behave almost like a source, they aren't used
as one.
This is about to change over the following commits, where we will turn
cached objects into a new "in-memory" source. This will allow us to use
it exactly the same as any other source by providing the same common
interface as the "files" source.
For now, the in-memory source only hosts the cached objects and doesn't
provide any logic yet. This will change with subsequent commits, where
we move respective functionality into the source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current ODB transaction interface is colocated with other ODB
interfaces in "odb.{c,h}". Subsequent commits will expand `struct
odb_transaction` to support write operations on the transaction
directly. To keep things organized and prevent "odb.{c,h}" from becoming
more unwieldy, split out `struct odb_transaction` into a separate
header.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new builtin for pretty formatting one revision expression
per line or commit object names found in running text.
Sometimes you want to format commits. Most of the time you’re
walking the graph, e.g. getting a range of commits like
`master..topic`. That’s a job for git-log(1).
But there are times when you want to format commits that you encounter
on demand:
• Full hashes in running text that you might want to pretty-print
• git-last-modified(1) outputs full hashes that you can do the same
with
• git-cherry(1) has `-v` for commit subject, but maybe you want
something else?
But now you can’t use git-log(1), git-show(1), or git-rev-list(1):
• You can’t feed commits piecemeal to these commands, one input
for one output; they block until standard in is closed
• You can’t feed a list of possibly duplicate commits, like the output
of git-last-modified(1); they effectively deduplicate the output
Beyond these two points there’s also the input massage problem: you
cannot feed mixed input (revisions mixed with arbitrary text).
One might hope that git-cat-file(1) can save us. But it doesn’t
support pretty formats.
But there is one command that already both handles revisions as
arguments, revisions on standard input, and even revisions mixed in
with arbitrary text. Namely git-name-rev(1): the command for outputting
symbolic names for commits.
We made some room in `builtin/name-rev.c` two commits ago. Let’s
now add this new git-format-rev(1) command. Taking inspiration from
git-name-rev(1), there are two modes:
• revs: like git-name-rev(1) in argv mode, but one revision per line
on standard in
• text: like git-name-rev(1) with `--annotate-stdin`
***
We need to add this command to the exception list in
`t/t1517-outside-repo.sh` because it uses “EXPERIMENTAL!”
in the usage line.
Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rust support is enabled by default (but still allows opting out) in
some future version of Git.
* bc/rust-by-default:
Enable Rust by default
Linux: link against libdl
ci: install cargo on Alpine
docs: update version with default Rust support
Hook scripts defined via the configuration system can now be
configured to run in parallel.
* ar/parallel-hooks:
t1800: test SIGPIPE with parallel hooks
hook: allow hook.jobs=-1 to use all available CPU cores
hook: add hook.<event>.enabled switch
hook: move is_known_hook() to hook.c for wider use
hook: warn when hook.<friendly-name>.jobs is set
hook: add per-event jobs config
hook: add -j/--jobs option to git hook run
hook: mark non-parallelizable hooks
hook: allow pre-push parallel execution
hook: allow parallel hook execution
hook: parse the hook.jobs config
config: add a repo_config_get_uint() helper
repository: fix repo_init() memleak due to missing _clear()
To test Git's behavior with very large pack files, we need a way to
generate such files quickly.
A naive approach using only readily-available Git commands would take
over 10 hours for a 4GB pack file, which is prohibitive.
Side-stepping Git's machinery and actual zlib compression by writing
uncompressed content with the appropriate zlib header makes things
much faster. The fastest method using this approach generates many
small, unreachable blob objects and takes about 1.5 minutes for 4GB.
However, this cannot be used because we need to test git clone, which
requires a reachable commit history.
Generating many reachable commits with small, uncompressed blobs takes
about 4 minutes for 4GB. But this approach 1) does not reproduce the
issues we want to fix (which require individual objects larger than
4GB) and 2) is comparatively slow because of the many SHA-1
calculations.
The approach taken here generates a single large blob (filled with NUL
bytes), along with the trees and commits needed to make it reachable.
This takes about 2.5 minutes for 4.5GB, which is the fastest option
that produces a valid, clonable repository with an object large enough
to trigger the bugs we want to test.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
With the previous commit removing every opt-in, the build-system
plumbing for nedmalloc has nothing left to switch on. Remove it so
that the upcoming deletion of the compat/nedmalloc/ tree is a pure
file removal.
Assisted-by: Opus 4.7
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git commands can accept a rather wide variety of URLs syntaxes.
The range of accepted inputs might expand even more in the future.
This makes the parsing of URL components difficult since standard URL
parsers cannot be used. Extracting the components of a git URL would
require implementing all the schemes that git itself supports, not to
mention tracking its development continuously in case new URL schemes
are added.
The url-parse builtin command is designed to solve this problem
by exposing git's native URL parsing facilities as a plumbing command.
Other programs can then call upon git itself to parse the git URLs
and extract their components. This should be quite useful for scripts.
Signed-off-by: Matheus Afonso Martins Moreira <matheus@matheusmoreira.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The fsmonitor IPC path logic in fsm-ipc-darwin.c is not
Darwin-specific and will be reused by the upcoming Linux
implementation. Rename it to fsm-ipc-unix.c to reflect that it
is shared by all Unix platforms.
Introduce FSMONITOR_OS_SETTINGS (set to "unix" for non-Windows, "win32"
for Windows) as a separate variable from FSMONITOR_DAEMON_BACKEND so
that the build files can distinguish between platform-specific files
(listen, health, path-utils) and shared Unix files (ipc, settings).
Move fsm-ipc to the FSMONITOR_OS_SETTINGS section in the Makefile, and
switch fsm-path-utils to use FSMONITOR_DAEMON_BACKEND since path-utils
is platform-specific (there will be separate darwin and linux versions).
Based-on-patch-by: Eric DeCosta <edecosta@mathworks.com>
Based-on-patch-by: Marziyeh Esipreh <marziyeh.esipreh@gmail.com>
Signed-off-by: Paul Tarjan <github@paulisageek.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move is_known_hook() from builtin/hook.c (static) into hook.c and
export it via hook.h so it can be reused.
Make it return bool and the iterator `h` for clarity (iterate hooks).
Both meson.build and the Makefile are updated to reflect that the
header is now used by libgit, not the builtin sources.
The next commit will use this to reject hook friendly-names that
collide with known event names.
Co-authored-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Our breaking changes document says that we'll enable Rust by default in
Git 2.54. Adjust the Makefile to switch the option from WITH_RUST to
NO_RUST to enable it by default and update the help text accordingly.
Similarly, for Meson, enable the option by default and do not
automatically disable it if Cargo is missing, since the goal is to help
users find where they are likely to have problems in the future.
Update our CI tests to swap out the single Linux job with Rust to a
single job without, both for Makefile and Meson. Similarly, update the
Windows Makefile job to not use Rust, while the Meson job (which does
not build with ci/lib.sh) will default to having it enabled.
Move the check for Cargo in the Meson build because it is no longer
needed in the main script.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano