Blame and "git log --stat" recover hunk coordinates by diffing blob
pairs, and recompute them on every run. Add a cache of those
coordinates at $GIT_DIR/objects/info/diff-hunks, beside the
commit-graph, so a later run can look them up instead of decompressing
the blobs and running xdiff again.
The store is a single chunk-format file (see gitformat-chunk(5)): an
8-byte header, a DHIX index of fixed-size entries sorted by key, a DHDT
segment of hunk records, and a trailing hash checksum. An entry is
keyed by the two blob object ids and the xdl_opts the pair was diffed
under, so a stored result is served only where that exact key recurs,
independent of path. A zero-context diff trims unchanged lines from
hunk edges and can pick a different but equally valid set of hunks than
an untrimmed diff, so a recording caller stores a pair only when its
trimmed and untrimmed diffs are identical; such an entry answers any
consumer at any context, and the rare divergent pair is always
computed. Identical hunk blocks are interned once and shared across
keys.
The library provides a reader (repo_diff_hunks_store and _replay, gated
by core.diffHunks), loaded once and cached on the object database as the
commit-graph is, and a writer that accumulates entries and flushes them
in one atomic pass. An absent, corrupt, or disabled store reads as all
misses. A record with no hunks is invalid too: replaying it would claim
the pair equivalent, which the store never asserts, so it reads as a
miss.
Ordinary reads are diagnostic-free. Loading parses the chunk table
through read_table_of_contents_quiet(), new in chunk-format, which
prints nothing on a malformed table and takes the repository's hash
algorithm rather than the_hash_algo, so the file is bounds-checked under
the algorithm it is keyed by.
The flush closes the repository's mmapped store and forgets that loading
was attempted before committing the lockfile. A warming run that also
reads may hold the file it is replacing mapped, and the rename must not
land on a live mapping, which Windows refuses; a read after the flush
then observes the committed file. commit-graph closes its graph before
committing for the same reason.
Writing is off by default, enabled per run by GIT_DIFF_HUNKS_WRITE or
persistently by diffHunks.write, the environment winning. A writer
seeds from the existing store, so a flush merges rather than replaces.
The seed's checksum is verified first: a corrupt store is discarded, not
rewritten with a fresh checksum verify could no longer catch. An entry
that fails the shared diff_provider_check_hunk() or names no blob is
dropped with a warning, since it would only ever read as a miss. A seed
that discarded or dropped anything forces the flush even when the
warming run computed nothing new. The writer fsyncs through a new
diff-hunks core.fsync component.
"git diff-hunks" inspects and manages the file: "verify" checks the
checksum, chunk table, sort order, entry bounds, and every entry's hunk
sequence against that shared check, so a store whose entries could only
read as misses fails verify; "clear" removes the file. Later patches
wire the readers and the writer into the diff and blame paths.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
To learn which line ranges changed between two blobs, every consumer in
the diff machinery loads both blobs and runs xdiff. There is no other
way to supply that answer, even when it is known elsewhere: a cache may
hold the ranges from the last time the pair was diffed, and a
format-aware process may have its own idea of which lines changed.
Either could answer from the blob object ids alone, but the loading and
computing are hard-wired into each consumer, so such an answer has no
place to enter.
Introduce the hunk provider interface, diff-provider.h, between asking
the question and computing the answer. A provider answers a request
made of the pair's identity, its blob object ids and the parameters
that determine the diff. A provider is either authoritative, so its
answer may deliberately differ from the builtin diff, or not, so its
answer must reproduce the builtin result exactly. Every answer served
from identity passes diff_provider_check_hunk() before a consumer sees
it: coordinates fit int32, hunks are ordered and non-overlapping, and
the unchanged runs between them match on both sides. A failing answer
is discarded and the pair falls through as unanswered.
Providers are repository-lifecycle objects. Each repository owns a
chain of them, built on first consultation and released from
repo_clear(), so a submodule gets its own providers and no provider
state outlives the repository it serves. The chain has a fixed
composition, and each provider gates itself per request, passing when
it does not apply. Chain order is the authority: the first answer
wins. A provider may instead refuse a pair whose request is shaped by
parameters its recording key cannot express. After a refusal, no later
provider answers the pair from identity, and the consumer must not
record what it computes for it. The last provider is the builtin
computation, the only one that computes rather than answering from
identity, so a walk given a fill callback always ends in an answer,
refusal or not.
The walk in diff-provider.c maps a provider's four dispositions
(answer, pass, fail, refuse) onto the consumer-facing outcomes, and
checks with BUG() that only the computing provider fails and that it
passes on a walk with no fill callback. The implementor contract, the
provider struct, its dispositions, and the shared check, lives in
diff-provider-internal.h, as refs/refs-internal.h is to refs.h;
consumers see only diff-provider.h.
The consumer surface is two types. struct diff_provider_request names
what is diffed and under which parameters; each later commit that
consults on more state adds the field it keys on (the object ids and
diff options, then the path). enum diff_provider_outcome flattens two
dependent axes into four points: the response state (answered,
unanswered, failed) and, only when unanswered, whether the caller may
record what it computes. The record rule rides in the outcome, not a
separate flag, so -Wswitch forces every consumer to place the no-record
arm. A provider added later maps onto these values inside the walk, so
consumer code is written once.
diff_provider_emit_hunks() is the consumer entry: the caller states the
request, a hunk callback, and a content-loading callback that reaches
the terminal provider only when the ranges are computed. Blame's
pass_blame_to_parent() is the first consumer, since it knows both blob
ids before reading either blob; its loads move into the fill callback.
With only the terminal provider registered, every request still
computes, so behavior is unchanged. (Blame's -C/-M split detection
diffs partial buffers with no blob identity and stays on xdi_diff().)
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "Defining an external diff driver" section explains how to
configure diff.<driver>.command but not how the driver relates to the
rest of Git's diff machinery. In particular, the command only
replaces the textual patch: word diff, function context, color, and
the like cannot apply to its output, while the summary formats, blame,
and git log -L do not run it at all and keep using the builtin diff.
Spell this out so the scope of an external diff driver is clear.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* mm/line-log-limited-ops:
diffcore-pickaxe: limit -G to the -L tracked range
diff: support --check with -L line ranges
diff: support stat formats with -L
diff: extract a line-range diff helper for reuse
diff: emit -L hunk headers via xdiff's formatter
diff: simplify the line-range filter by classifying removals immediately
diff: rename line-range filter struct and clarify fields
Teach -G to only search the line ranges specified by -L.
Teaching -S is left as future work, so it still matches the entire
file even if -L is specified.
Rather than being part of diff.c's builtin implementations, the
diffcore-pickaxe functionality interacts with xdiff-interface as a
separate component. Add a sibling to xdi_diff_outf(), called
diff_emit_line_ranges(), that limits emitted lines to the given line
ranges.
Use diff_emit_line_ranges() when searching text if line ranges have
been specified. If textconv is enabled, use normal diffing instead of
diff_emit_line_ranges() since line range tracking relies on the line
coordinates of the original, pre-textconv file.
Update documentation and add tests accordingly.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Reuse the line_range_filter in builtin_checkdiff() so -L supports
the --check option.
Add orig_hunk_fn field similar to orig_line_fn that forwards
xdiff_emit_hunk_fn calls when we flush filtered hunks. This is necessary
because --check relies on receiving calls to its checkdiff_consume_hunk
function for managing state.
Document and ungate the newly enabled option, and add tests verifying
the new behavior.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Reuse the line_range_filter in builtin_diffstat() so -L supports
the stat formats and add tests verifying the new behavior.
Ungate the newly enabled options and drop "yet" from the generic
-L rejection message ("does not yet support the requested diff
format"). Some rejected formats do not fit -L at all, so "yet"
wrongly implies they are all awaiting support.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Extract logic for initializing the line-range filter and running a diff
for a specific line range. This logic is needed for any diff that
targets a line range independent of the current patch display path.
The subsequent commits use this logic to enable additional line range
targeted diff modes.
No logical behavior change.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, diff's line-range filter implements its own method for
emitting diff hunk headers. This mostly matches what xdiff itself
outputs, but there is a discrepancy for postimage or preimage sides
with 0 line changes. For a side with no lines (count 0), the begin is
the line before the change. The header omits the line count of 1.
Rather than fix this case in the line-range implementation, expose the
function xdiff uses to emit its headers. Reusing it keeps the header
format consistent with and without -L.
Update test scenarios and fixtures to reflect the now consistent header
format.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, the diff line-range filter buffers preimage removal lines
until a postimage line arrives. That line's number confirms whether the
preimage line falls in a relevant range. However, storing preimage
lines in a separate buffer is unnecessary. Worse, the logic has a bug:
a preimage line outside the target range is included when it
immediately follows an in-range postimage line.
Preimage lines will always precede their postimage counterpart both in
content line number and emission order from xdiff's line callback
function. So preimage lines can share the postimage buffer. The filter
flushes them based on whether the postimage lines fall within the
target range.
Remove logic related to storing preimage lines in a separate "removal"
buffer and prepending them to the accumulating_hunk's line buffer.
Instead, store those lines in the accumulating_hunk's line_buffer
immediately and flush everything as appropriate based on postimage
line numbers that arrive. This resolves the bug by construction.
Also, calculate the old and new line counts for the diff hunk header
when flushing rather than storing counters in line_range_filter to
simplify state management further.
Add a test to t/t4211-line-log.sh that verifies the preimage line
emission bug is fixed.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
diff's line-range filtering logic uses the line_range_callback
struct to represent filtering state. However, this name does not
clearly reflect the role it plays. This is especially relevant as
we expand diff's line-range filtering to work with more options,
including --stat and -G.
Also, line_range_callback's fields are terse, while the comment
explaining line_range_callback is verbose and out of place compared to
its surroundings.
Rename line_range_callback to line_range_filter, and replace the verbose
comment with a concise one, instead preferring descriptive field and
variable names that are self-explanatory over comments.
No logical behavior change. Some fields are grouped under a new struct
in the newly renamed line_range_filter. Everything else is just a
rename.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The global variable 'fetch_if_missing' has been moved to a member in
'struct repository', continuing the libification process and
allowing per-repository control (such as for submodules).
* ty/repository-fetch-if-missing:
repository: move fetch_if_missing into struct repository
The pack-objects command has been updated to record the total bytes
written to pack files in trace2 output, allowing performance
analysis of different compression settings by comparing the
resulting pack sizes.
* fr/pack-objects-trace-pack-bytes:
pack-objects: trace pack bytes written
The mechanism to generate a packfile corresponding to the result of
a fetch/push has been made pluggable through a set of object
database callback functions, removing hardcoded references to
'pack-objects' and enabling alternative ODBs to serve packfiles
themselves.
* ps/odb-pluggable-pack-generation:
bundle: generate packfiles via the object database
bundle: get (mostly) rid of `the_repository`
builtin/bundle: refactor option handling for progress meter
send-pack: generate packfiles via the object database
upload-pack: generate packfiles via the object database
odb: introduce interface to generate packfiles
The 'git receive-pack' command has been updated to use a new ODB
transaction interface for writing incoming packfiles, making it more
backend-agnostic.
* jt/receive-pack-pluggable-writes:
odb/transaction: add transaction interface to write packfiles
odb: return temporary ODB source when set
builtin/receive-pack: explicitly pass packfile fd
builtin/receive-pack: report unpack errors via strbuf
builtin/receive-pack: lift global state out of unpack()
builtin/receive-pack: read unpack limit config lazily
builtin/receive-pack: pass shallow file explicitly
odb/transaction: add transaction finalize interface
builtin/receive-pack: properly clean up keep files
The threshold for geometric repacking to trigger based on loose
object count has been adjusted to match that of 'git gc --auto',
preventing over-aggressive repacking during concurrent writes.
* ps/odb-geometric-repack-loose-threshold:
odb/files: be less aggressive with geometric repacking
The trailer parsing machinery has been updated to avoid mistaking
lines that begin with a URL (e.g., 'https://...') as trailer lines.
This prevents intended textual URLs from being mangled or mistakenly
treated as metadata keys.
* kh/trailers-no-urls:
trailers: stop recognizing URLs as trailers
The object database layer has been simplified by eagerly loading
alternate object directories upon initialization, instead of
deferring it to the first object lookup. This eliminates the need
for scattered lazy-loading calls throughout the codebase and paves
the way for integrating alternates with the pluggable backends.
* ps/odb-eagerly-load-alternates:
odb: drop `alternates_db` field
odb: drop `loaded_alternates` field
odb: eagerly initialize alternates
odb: decouple source path comparisons from `the_repository`
setup: create ref and object databases after config is written
The command line completion (in contrib/) has been taught to handle
the experimental 'git history' command.
* vm/complete-history:
completion: complete 'git history split' pathspecs
completion: complete 'git history --update-refs' values
completion: complete 'git history --empty' values
completion: add 'git history' subcommands
The object database (odb) API has been refactored to distinguish
between missing objects and corrupt ones by returning more
descriptive error statuses. Both the packed and loose backends now
faithfully propagate error details using a generic strbuf error
mechanism, removing backend-specific leakage from central lookup
paths.
* ps/odb-generic-corrupt-objects:
odb: handle `OBJECT_INFO_DIE_IF_CORRUPT` generically
odb/source: allow `read_object_info()` to bubble up error messages
odb/source: let callers discern missing and corrupt objects
odb/source: introduce error status when reading objects
odb/source-packed: flag known-bad objects as corrupt and not missing
The DWIM logic in 'git worktree add' sometimes tried to infer a
remote-tracking branch when an explicit '-b' or '-B' option was
given to create a new branch, causing the explicit branch name to
be ignored, which has been corrected.
* yn/worktree-add-no-dwim-with-b:
worktree add: shouldn't dwim if -b or -B is given
A heap-use-after-free bug in the object name parsing code when
reporting failures with a relative path to a sparse directory has
been corrected.
* sk/object-name-use-after-free:
object-name: avoid use-after-free in get_oid_with_context_1()
The documentation for 'git format-rev' has been updated to use the
[synopsis] block definition on code blocks to properly highlight
placeholders, and a quoting inconsistency in the running text has
been fixed.
* kh/format-rev-doc-synopsis:
doc: format-rev: use [synopsis] on code block
doc: format-rev: quote subject placeholder before and after
'git -C <dir> checkout fi<TAB>' did not complete, which has been
corrected.
* jc/complete-checkout:
completion: 'git checkout' completes untracked paths as a last resort
completion: complete tracked paths for "git checkout"
completion: no-op refactoring of checkout completion
'git -C <dir> diff fi<TAB>' did not complete 'file', which has been
corrected.
* jc/complete-diff-tracked-paths:
completion: 'git diff' completes untracked paths as a last resort
completion: complete tracked paths for 'git diff'
completion: no-op refactoring of diff completion
The unused name parameter in 'struct chdir_notify_entry' has been
removed from chdir_notify_register(), chdir_notify_unregister(), and
related callback signatures across several subsystems, simplifying the
API now that trace output no longer uses it.
* ch/chdir-notify-drop-name:
chdir-notify.h: Removed unused param 'name'
The performance of adding numerous new packfiles has been improved
by introducing a fast path for known-new packfiles to skip an
unnecessary traversal in packfile_list_append(), avoiding a
quadratic complexity regression on load.
* js/packfile-fast-append:
packfile: fix perf regression with many packs
'git repack' has been taught '--drop-filtered' to delete local
promisor blobs exceeding a limit (currently 'blob:limit=') in partial
clones, reclaiming space. Guards prevent running during other
operations or if referenced by the index.
* ss/repack-drop-filtered:
builtin/repack: add guards for --drop-filtered
builtin/repack: actually drop filtered promisor blobs
builtin/repack: enumerate promisor blobs for --drop-filtered
repack-promisor: allow excluding objects from the rebuilt promisor pack
list-objects-filter: add list_objects_filter__filter_oidset()
builtin/repack: add --drop-filtered and --dry-run options
Various tests in 't7900-maintenance.sh' have been updated to use a
throwaway repository, and auto-detaching of maintenance tasks is now
disabled for these tests to fix flaky races with concurrent background
maintenance jobs.
* ps/t7900-deflake-maintenance:
t7900: fix flaky "maintenance.strategy" test
t7900: adapt some tests to use a throwaway repository
A client requesting the promisor-remote capability without a value
caused a null pointer dereference, which has been corrected by
rejecting a request without an argument.
* en/serve-promisor-remote-fix:
serve: reject valueless promisor-remote capability
The 'pack-objects' and delta-encoding code paths have been updated to
use 'size_t' instead of 'unsigned long' for object sizes and offset
limits, avoiding potential truncation issues on 64-bit Windows.
* js/pack-objects-delta-size-t:
packfile: widen `unpack_object_header_buffer()` to `size_t`
git-zlib: widen `git_deflate_bound()` to `size_t`
t/helper/test-pack-deltas: widen `do_compress()`'s maxsize local to `size_t`
http-push: widen `start_put()`'s size local from `ssize_t` to `size_t`
diff: widen `deflate_it()`'s bound local from int to `size_t`
archive-zip: widen `zlib_deflate_raw()`'s maxsize local to `size_t`
packfile, git-zlib: widen `use_pack()` and zstream avail fields to `size_t`
delta: widen `create_delta()` and `diff_delta()` to `size_t`
pack-objects: widen `mem_usage` and `try_delta()`'s out-param to `size_t`
pack-objects: widen `free_unpacked()` return to `size_t`
pack-objects: widen delta-cache accounting to `size_t`
delta: widen `create_delta_index()` parameter to `size_t`
diff-delta: widen `struct delta_index`' size fields to `size_t`
The '--shallow-file' option of 'git' command requires a value, but the
code did not check the presence of a value and instead segfaulted
without one, which has been corrected.
* cc/git-shallow-file-wo-value:
git: avoid segfault on "git --shallow-file" without a value
The setting of a now-unused member '.pretty_given' in the sequencer
machinery has been removed.
* en/sequencer-lose-pretty-given:
sequencer: remove unnecessary variable setting
A handful of code paths have been corrected to check return values
from functions like curl_easy_duphandle(), deflateInit(), lseek(),
dup(), and strbuf_getline_lf(), resolving several Coverity warnings
about unchecked returns.
* js/coverity-unchecked-returns-fix:
bisect: handle dup() failure when redirecting stdout
bisect: check get_terms return at all call sites
bisect: check strbuf_getline_lf return when reading terms
transport-helper: warn when export-marks file cannot be finalized
transport-helper: check dup() return in get_exporter
compat/pread: check initial lseek for errors
last-modified: handle repo_parse_commit() failures
reftable tests: check reftable_table_init_ref_iterator() return
reftable/block: check deflateInit() return value
reftable: handle block-writer initialization errors
config: propagate launch_editor() failure in show_editor()
http: die on curl_easy_duphandle failure in get_active_slot
The merge-base computation has been optimized by stopping the walk
early when one side's exclusive commits in the queue are exhausted,
yielding significant speedups for queries with one-sided histories.
* kk/merge-base-exhaustion:
commit-reach: remove commit-date ordering fallback
commit-reach: move min_generation check into paint_queue_get()
commit-reach: terminate merge-base walk when one paint side is exhausted
commit-reach: introduce struct paint_state with per-side counters
t6600: add clock-skew topologies and step counts for edge cases
commit-reach: add trace2 instrumentation to paint_down_to_common()
t6099: add side-exhaustion regression test
t6600: add test cases for side-exhaustion edge cases
test-lib-functions: improve diagnostic output for trace2 data assertions
Documentation/technical: add paint-down-to-common doc
The sequencer has been updated to release the object database before
spawning 'git commit'. This prevents open file handles from
blocking auto-maintenance tasks, such as repacking, on systems like
Windows where open files cannot be easily unlinked.
* js/sequencer-release-odb-before-commit:
sequencer: release the ODB before spawning git commit
The error message given by 'git send-email' when a message file is
missing a 'Subject:' header has been clarified, and the error string
is now terminated with a newline so that Perl avoids appending its
internal source location data.
* hn/send-email-missing-subject-error:
send-email: clarify missing subject error
The 'struct odb_read_stream' and 'struct odb_write_stream'
structures have been consolidated into a single unified 'struct
odb_stream' structure, simplifying object database streaming APIs
and enabling streaming of arbitrary object types.
* ps/odb-streams:
odb/streaming: unify function names to create new streams
odb/streaming: rename `struct input_zstream_data`
odb/streaming: rename `struct read_object_fd_data`
odb/streaming: consolidate read and write streams
odb/streaming: rename `struct odb_read_stream`
odb/streaming: support streaming arbitrary object types
odb/streaming: drop `is_finished` field
odb/streaming: track write stream size in the structure
The usage string of 'git fast-import' has been updated to use the
parse_options() API for displaying help, and its SYNOPSIS in the
documentation has been standardized to match.
* cc/fast-import-usage:
fast-import: remove useless from_stream argument
fast-import: use parse_options() for command line options
fast-import: use callbacks to parse some options
fast-import: use struct option for usage string
fast-import: move command state globals into 'struct fast_import_state'
fast-import: introduce 'struct fast_import_state'
fast-import: factor out option_*() functions
fast-import: use int for some bool flags
fast-import: localize 'i' into the 'for' loops using it
api-parse-options.adoc: document hidden and OPT_*_F option macros
api-parse-options.adoc: document per-option flags
parse-options: introduce OPT_HIDDEN_GROUP
The 'remote-object-info' command for 'git cat-file --batch-command'
has been extended to support the '%(objecttype)' placeholder.
* ps/cat-file-remote-object-info-type:
cat-file: unify default format
serve: advertise type capability
fetch-object-info: parse type from server response
protocol-caps: add type support to object-info
transport: drop remote object-info fields from transport struct
fetch-object-info: die() on the remaining error path
fetch-object-info: use dedicated struct for the results
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: detect malformed server responses
t5701: use test_file_size() to get the size of a file
Michael Montalbo