The provider chain so far holds the diff-hunks store in front of the
terminal builtin computation. Open it to external processes: a pair on
a path whose driver configures diff.<driver>.process is answered by a
long-running process speaking a pkt-line protocol (following the filter
process protocol), registered at the head of the chain and consulted
before the store and before any blob is loaded.
The protocol starts with the smallest request that can carry an answer:
object names alone. A request is the pathname and the pair's
old-oid/new-oid, with no content. The process answers with hunk lines,
with a zero-hunk success that asserts the blobs equivalent (trailing
newlines included), or with status=need-content, on which the pair
falls through to the builtin answer. This serves the two shapes that
need no content pushed to them: a cache keyed on the blob pair, and a
process that fetches the blobs itself (for example over "git cat-file
--batch"). A pair whose side is not a stored blob carries a NULL id;
the provider sends no request and passes it. Because Git holds no
content for the exchange, the answer is used as sent: hunks are
validated for order, overlap, lockstep alignment, and magnitude, then
replayed without the normalization xdiff applies to diffs it computes
itself. The magnitude bound is the blobs' sizes, read from the object
database without loading content: a blob of N bytes holds at most N
lines.
Because the process's answer is authoritative, it outranks the store,
and its head-of-chain position says so. A pair the process answers
never reaches the store and is never recorded, so nothing it produces
enters the store, which holds the builtin answer only. A request it
does not answer, whether need-content, a missing capability, or a
missing id, passes down the chain to the builtin answer, which is what
the store serves, so the store may serve such a pair and a warming run
may record it. Entries recorded before a process was configured are
not purged; a pair the process answers ignores them, and "git diff-hunks
clear" discards them.
The provider gates itself per request. The driver is looked up by the
old-side path, so a renamed file resolves to the same driver, and by
the repository-relative path, so a diff.relative run from a
subdirectory names the pair the same way. Options the process is never
told about select no process: the whitespace-ignoring options, -I,
--anchored, and an algorithm forced by option or configuration (blame
routes its algorithm through xdl_opts, so --histogram is covered). The
request gains its last field, the path; the consumers change only by
filling it, and neither names the process.
The provider's state is its repository's pool of running processes,
keyed by the configured command, so drivers sharing a command share a
process, a submodule speaks to its own, and releasing the provider
(from repo_clear()) stops them. The pool owns a copy of each command
string, so an entry outlives a config re-read. A command that fails
stays as an entry that is not retried: its request and every later one
pass, so the store may serve the path for the rest of the command.
A protocol error in a response never kills the command. The response
is read through a packet reader gentle about framing, so an error takes
one path: a single warning, the process stopped and marked failed, and
the builtin diff for the rest of the command. That covers garbage
bytes, a truncated response, an empty packet, a bare status, and an
unrecognized status. Semantically invalid coordinates cost only their
pair: the response is drained, the pair is computed, and the process
stays alive. A path the protocol cannot carry (an embedded newline, or
one too long for a packet) falls back per path rather than costing the
command its process. The handshake keeps one fatal check: a process
that announces a capability Git did not request aborts the command, as
the long-running filter protocol does.
Consulting is allowed per command, following the allow_textconv
precedent. "git diff", "git log" and "git show", and "git blame" set
allow_diff_process; the plumbing diff commands and the interactive-patch
machinery never set it, so scripted and staging output stays builtin.
The options adjust the flag:
- --no-ext-diff clears it and --ext-diff sets it;
- --diff-process and --no-diff-process set and clear it alone, leaving
external diff drivers as they were;
- format-patch clears it unconditionally, so a generated patch applies
for recipients without the process;
- range-diff passes --no-ext-diff to the "git log" it compares.
git blame and the summary formats consult the process. For blame, a
pair reported equivalent emits no hunks, so the whole commit passes to
its parent. In the stat formats such a pair sums to a zero-count entry,
which the "nothing changed" rule omits, as under -w. The subprocess is
long-running: one startup cost across a traversal, one round-trip per
consulted pair. Answers travel in struct xdl_hunk, new in
xdiff-interface.h, holding xdiff's 1-based coordinates; nothing feeds
them back to xdiff, since only coordinate consumers consult.
A content-carrying request is the natural extension: it would serve
sides that are not stored blobs and processes that want content pushed
to them, and bring patch output and log -L's range tracking to the same
answer. As it stands, a process's answers show in blame and the summary
formats while patch output stays builtin.
t4080 exercises the protocol, the per-command gate, and the error paths:
- each adversarial response shape warns and falls back to builtin, the
request log proving which failures disable the process and which keep
it alive (a malformed hunk line, coordinates past the blob size, a
count overflowing strtol(), overlapping or misaligned hunks, an
unrecognized status, a bare status, an empty packet, a mid-response
crash, and raw garbage);
- a capability-less process and status=abort degrade without noise, and
a failed start warns once and returns the path to the store;
- a trailing token on a hunk line is ignored, pinning field
appendability;
- positive consults for git diff, git show, and diff-tree under
--ext-diff and --diff-process; textconv output and gitlink sides are
never identified; a diff.relative run consults by the repo-relative
path;
- the equivalence answer is pinned from both consumers, and a warming
run past a deferring process records the pair for a later read.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach builtin_diffstat() to consult the hunk provider interface through
diff_provider_consult(), new here: the consult-only entry that answers
without loading content or computing, so it never returns
DIFF_PROVIDER_ERROR. On an answer, the summing callback accumulates the
provided counts directly into the diffstat entry; the blobs were already
loaded for the binary check, so an answer saves the diff run, not the
content load (blame, taught next, skips its loads too). On an
unanswered outcome it computes as before and, with a writer attached,
records what it computed; on unanswered-no-record it computes without
recording.
The provider behind the consult is the diff-hunks store, registered in
front of the terminal builtin computation. Its consult serves a
recorded pair through diff_hunks_replay(), which validates the sequence
before any hunk reaches the callback, so direct accumulation is safe.
The request gains the pair's object ids and the diff options read by the
exclusions below. A side whose bytes are not a stored blob, such as a
working-tree file or a gitlink, has a NULL id; the store passes it by and
the terminal provider computes it. diff_provider_emit_hunks() walks the
same chain, so blame's requests follow these rules the moment blame
supplies identity. The walk also insists, as a BUG check, that a
request's diff options belong to the repository whose chain it walks.
Each exclusion lives with the provider whose key cannot express it. -I
patterns and --anchored shape the diff outside the store key, and break
detection (-B) rescores the pair outside it; the store's consult maps
all three to stop-no-record, so such a request is neither served nor
recorded for any consumer. The consumer-side guard the recording commit
carried for those three comes out here. The compile-time assert on
xpparam_t's layout sits next to that decision, forcing an explicit
keying decision whenever a diff parameter is added. The stat consumer
keeps only the exclusion that is not about the key: --ignore-blank-lines
is part of the key but coalesces hunks differently between the
text-emitting and coordinate-callback paths, so the consumer returns
before consulting. A "log -L" range-scoped stat neither reads nor
records; the line-range filter computes it as before.
"git diff", "git log", "git show", and "git diff-tree" with the --stat,
--numstat, and --shortstat formats consult the interface. Reading is
controlled by core.diffHunks.
An answer is invisible in the output, so the store counts the pairs it
serves and the consultations it cannot, and diff_hunks_read_stats()
reports both; the stat path emits the hits as a trace2 "read-hits" datum
for tests and tuning. The counters live on the store because only the
store knows whether a consultation reached it, and none of its exclusion
legs reaches the replay, so none counts as a miss.
Extend t4220 with the read half:
- output parity with and without the store, at several context lengths
and both directions, and reversed pairs keying apart;
- the consultation made visible through the read-hits datum, and the
trim-divergent pair correct at every context;
- the settings that must bypass the store doing so in both directions
(-I, -B, --anchored, --ignore-blank-lines), asserted through the trace
rather than output parity alone, which a coincidentally equal count
could satisfy;
- a driver-forced algorithm keying apart rather than bypassing: it is
part of the key, so a read under it misses the default entries and a
warm records under its own.
A "log -L" range-scoped stat neither reads nor records.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
To learn which line ranges changed between two blobs, every consumer in
the diff machinery loads both blobs and runs xdiff. There is no other
way to supply that answer, even when it is known elsewhere: a cache may
hold the ranges from the last time the pair was diffed, and a
format-aware process may have its own idea of which lines changed.
Either could answer from the blob object ids alone, but the loading and
computing are hard-wired into each consumer, so such an answer has no
place to enter.
Introduce the hunk provider interface, diff-provider.h, between asking
the question and computing the answer. A provider answers a request
made of the pair's identity, its blob object ids and the parameters
that determine the diff. A provider is either authoritative, so its
answer may deliberately differ from the builtin diff, or not, so its
answer must reproduce the builtin result exactly. Every answer served
from identity passes diff_provider_check_hunk() before a consumer sees
it: coordinates fit int32, hunks are ordered and non-overlapping, and
the unchanged runs between them match on both sides. A failing answer
is discarded and the pair falls through as unanswered.
Providers are repository-lifecycle objects. Each repository owns a
chain of them, built on first consultation and released from
repo_clear(), so a submodule gets its own providers and no provider
state outlives the repository it serves. The chain has a fixed
composition, and each provider gates itself per request, passing when
it does not apply. Chain order is the authority: the first answer
wins. A provider may instead refuse a pair whose request is shaped by
parameters its recording key cannot express. After a refusal, no later
provider answers the pair from identity, and the consumer must not
record what it computes for it. The last provider is the builtin
computation, the only one that computes rather than answering from
identity, so a walk given a fill callback always ends in an answer,
refusal or not.
The walk in diff-provider.c maps a provider's four dispositions
(answer, pass, fail, refuse) onto the consumer-facing outcomes, and
checks with BUG() that only the computing provider fails and that it
passes on a walk with no fill callback. The implementor contract, the
provider struct, its dispositions, and the shared check, lives in
diff-provider-internal.h, as refs/refs-internal.h is to refs.h;
consumers see only diff-provider.h.
The consumer surface is two types. struct diff_provider_request names
what is diffed and under which parameters; each later commit that
consults on more state adds the field it keys on (the object ids and
diff options, then the path). enum diff_provider_outcome flattens two
dependent axes into four points: the response state (answered,
unanswered, failed) and, only when unanswered, whether the caller may
record what it computes. The record rule rides in the outcome, not a
separate flag, so -Wswitch forces every consumer to place the no-record
arm. A provider added later maps onto these values inside the walk, so
consumer code is written once.
diff_provider_emit_hunks() is the consumer entry: the caller states the
request, a hunk callback, and a content-loading callback that reaches
the terminal provider only when the ranges are computed. Blame's
pass_blame_to_parent() is the first consumer, since it knows both blob
ids before reading either blob; its loads move into the fill callback.
With only the terminal provider registered, every request still
computes, so behavior is unchanged. (Blame's -C/-M split detection
diffs partial buffers with no blob identity and stays on xdi_diff().)
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Michael Montalbo