A new 'diff.<driver>.process' configuration has been introduced to
allow a long-running external process to act as a hunk provider,
enabling external tools to control which lines Git considers changed
while leaving all output formatting (word diff, color, blame, etc.) to
Git's standard pipeline.
* mm/diff-process-hunks:
fixup! diff: consult oid-only hunk providers via diff.<driver>.process
diff: consult oid-only hunk providers via diff.<driver>.process
userdiff: add diff.<driver>.process config
sub-process: add a gentle status read
sub-process: separate process lifecycle from hashmap management
blame: read precomputed hunks
diff: read precomputed hunks for stat output
diff: record precomputed hunks during stat output
diff-hunks: add the store format, library, and command
diff: introduce a hunk provider interface
gitattributes: document how external diff drivers relate to diff features
The experimental 'git history' command has been taught a new 'squash'
subcommand to fold a range of commits into a single commit, with any
descendants replayed on top.
* hn/history-squash:
history: support editing squashed commit messages
history: create squashed commits without editing
history: protect branches when squashing a range
history: validate squash revision ranges
history: add skeleton for squash subcommand
sequencer: share the squash message marker helpers and flags
history: give commit_tree_ext a message template
history: extract helper for a commit's parent tree
The experimental 'git format-rev' has been taught a few more
formatting options.
* kh/format-rev-more-options:
format-rev: learn --abbrev, --color, and --date
doc: rev-list-options.adoc: factor out --date alts
format-rev: factor option variables into a struct
format-rev: place BUG calls first in callback
format-rev: use lower case for opts description
The HTTP transport has been taught to check the revocation status of
the server certificate using the stapled OCSP response during the
TLS handshake via a new 'http.sslVerifyStatus' configuration
variable.
* gg/http-ssl-verify-status:
http: add http.sslVerifyStatus to check stapled OCSP responses
A new 'uploadpack.lazyFetchTrusted' configuration variable has been
introduced to allow 'upload-pack' to lazily fetch missing objects from
configured promisor remotes when serving trusted repositories.
* cc/lazy-fetch-trusted-bit:
builtin/upload-pack: set GIT_NO_LAZY_FETCH to 0 on trusted repo
upload-pack: read uploadpack.lazyFetchTrusted
setup: add 'allow_dot' arg to path_allowlist_apply()
setup: extract path_allowlist_apply()
promisor-remote: factor out lazy_fetch_objects()
The build-time knob 'USE_NSEC' for nanosecond stat precision has been
converted to a runtime configuration 'core.useNanosec', allowing
distributions to bundle one binary that adapts to filesystem
capabilities dynamically.
* dk/use-nsec-runtime:
core: convert build-time USE_NSEC into runtime core.useNanosec
environment: align repo_config_values_init with struct declaration
meson: expose knob for xmlto relative links in manuals
The parser for hex object names has been updated to reject uppercase
hexadecimal characters when running in the breaking changes mode, in
preparation for Git 3.0.
* bc/restrict-hex-to-lowercase:
hex: allow only lowercase object IDs in breaking changes mode
object-name: use hexval
hex: label usages of hex parsing for object IDs
hex: make hex_to_bytes accept kind of hex to use
hex: allow specifying hex type with hex2chr
hex: add functionality for lowercase-only hex
The 'git fetch' command can now configure how submodule fetch errors
are handled via 'fetch.submoduleErrors' and '--submodule-errors',
making them non-fatal. A premature failure during recursive submodule
fetches has been fixed by deferring the error until the OID-based
retry phase fails.
* pz/fetch-submodule-errors-config:
fetch: add fetch.submoduleErrors to make submodule fetch errors non-fatal
submodule: fix premature failure in recursive submodule fetch
Support for skipping the editor when continuing a rebase after
conflict resolution has been added with the '--no-edit' option, and
forcing it with '--edit'. A new configuration variable
'rebase.noEdit' can be used to set the default behavior.
* hs/rebase-continue-edit:
rebase: add --[no-]edit to --continue
The 'git repo info' command has been taught more keys to output
paths of various repository components (such as the working tree
root, superproject working tree, object database, etc.), supporting
both absolute and relative path formats.
* kj/repo-info-more-path-keys:
repo: add path.cdup
repo: add path.git-prefix
repo: add path.grafts with absolute and relative suffixes
repo: add path.index with absolute and relative suffixes
repo: add path.hooks with absolute and relative suffixes
repo: add path.superproject-root with absolute and relative suffixes
repo: add path.toplevel with absolute and relative suffix formatting
The 'git log -L<range>:<path>' command has been taught to limit
various 'diff' operations, such as '--stat', '--check', and '-G', to
the specified range and path.
* mm/line-log-limited-ops:
diffcore-pickaxe: scope -G to the -L tracked range
diff: support --check with -L line ranges
line-log: support diff stat formats with -L
diff: extract a line-range diff helper for reuse
diff: emit -L hunk headers via xdiff's formatter
diff: simplify the line-range filter by classifying removals immediately
diff: rename and group the line-range filter for clarity
Support for '-m', '-F', '-c', or '-C' options to supply a commit log
message from outside the editor has been added for all 'git commit
--fixup' variations.
* ec/commit-fixup-options:
commit: allow -c/-C for all kinds of --fixup
commit: allow -m/-F for all kinds of --fixup
Racy Git problems persist today, manifesting themselves in the
performance of commands like "git diff" in new worktrees [1]. We have
long had a build knob "USE_NSEC" to tell Git to use in-core nanosecond
precision when available, which mitigates most if not all racy issues,
but most builds we know about don't use it. In part, that's because
someone distributing Git can't safely enable it at compile-time if they
don't know exactly what platforms their distribution will be used on.
[1]: https://lore.kernel.org/git/CALnO6CADMJSixqYvL1Yo8qKX5rWhKQ+2OoSEuPUh-yoeK9TseQ@mail.gmail.com
These days, most platforms are likely to be safe for the USE_NSEC code.
Regardless, we want to give users the ability to benefit from it. This
requires exposing the compile-time gated code as a runtime option.
In addition, update the Racy Git documentation and other mentions of
USE_NSEC in the code.
Due to the conversion from #ifdef to runtime check, using the flag
"--ignore-space-change" may be particularly helpful when viewing changes
from this patch.
Signed-off-by: D. Ben Knoble <ben.knoble@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Makefile-based builds have had this knob for most of the project's life,
since a479a564dc (Documentation/Makefile: allow
man.base.url.for.relative.link to be set from Make, 2009-12-03).
Meson, however, hard-codes the equivalent of $prefix/$mandir, which is
not really where all the HTML docs are stored in most distro builds.
Plus, this value is missing a trailing slash, so links come out broken,
like this in git.1:
1. Git User’s Manual
/usr/share/manuser-manual.html
Of course we can do better:
1. Change the default to match Make: use file://$(htmldir)/ (with
trailing slash!) to form a local URL pointing at the HTML docs. This
is safe because all current uses of link:<relative> point at HTML
docs:
git grep 'link:[[:alnum:]]' Documentation | grep -ve html -e http
produces only a single result (Documentation/howto/howto-index.sh)
which can be ignored. Since nothing else [*] in the normal build sets
MAN_BASE_URL, this seems like the right default.
2. Provide a configurable knob, just like the Makefile, so distributions
that build with Meson (like Gentoo) can decide where to make the
links if they need to. Those that set htmldir probably won't need to
tweak this any further, though.
[*]: Well, Git's todo branch has a script dodoc.sh to build and archive
docs for kernel.org; these docs are pulled by Homebrew
installations, for example. It sets MAN_BASE_URL to "git_htmldocs",
so the equivalent note on macOS + Homebrew is
1. Git User’s Manual
git-htmldocs/user-manual.html
which is not functional either, but that's a problem for
downstream. In any case, users can recover the right path with
"git --html-path".
Signed-off-by: D. Ben Knoble <ben.knoble@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `git fetch` and `git clone` commands have been optimized to
download packfile URIs in parallel when the new
`fetch.packfileURIThreads` configuration is set, significantly
speeding up fetches from servers that advertise multiple packfiles.
* ps/fetch-packfile-uris-parallel:
fetch-pack: allow parallelizing packfile URI fetches
fetch-pack: prepare for threaded fetching of packfile URIs
The 'git replay' command has been taught the '--linearize' option to
drop merge commits and linearize the replayed history, mimicking 'git
rebase --no-rebase-merges'.
* tc/replay-linearize:
replay: offer an option to linearize the commit topology
replay: resolve the replay base outside pick_regular_commit()
replay: add helper to put entry into replayed_commits
'git worktree add' did not prevent DWIM behavior when '-b' or '-B' was
specified, which has been corrected.
* yn/worktree-ambiguous-remote-advice:
worktree add: treat multiple matches with --guess-remote as an error
worktree add: improve message for ambiguous remote branch name
checkout: improve message for ambiguous remote branch name
checkout: extract function to display advice for ambiguous remotes
The global variable 'fetch_if_missing' has been moved to a member in
'struct repository', continuing the libification process and
allowing per-repository control (such as for submodules).
* ty/repository-fetch-if-missing:
repository: move fetch_if_missing into struct repository
One of the stated goals of git-replay(1) is to allow implementing the
git-rebase(1) functionality on the server side.
The default mode of git-rebase(1) is to act as if `--no-rebase-merges`
was given. This mode drops merge commits instead of replaying them, and
linearizes the history into a sequence of regular (single-parent)
commits.
Add option `--linearize` to git-replay(1) to do the same. Each replayed
commit is stacked on top of the previously replayed one. When a merge is
encountered, the commits reachable from all of its sides are replayed
into the single line and the merge itself is dropped.
If a ref was pointing to a merge commit, that ref is updated to the
merge's last replayed ancestor.
git-replay(1) accepts multiple branches, for example:
$ git replay --onto main topic1 topic2
Without `--linearize` this replays 'topic1' and 'topic2' onto 'main'
(keeping shared portions of history shared and divergent parts
divergent) and updates both refs.
Due to current implementation limitations, replaying multiple branches
with `--linearize` is disallowed to avoid concatenating unrelated
histories into a single line. For the same reason disallow the use of
`--contained` with `--linearize`.
Users who want to linearize multiple branches are advised to do this in
separate git-replay(1) invocations. Linearizing multiple branches at
once might be added later.
Note that `--linearize` is not modeled after git-rebase(1)'s
`--rebase-merges[=<mode>]` interface. Recreating merges, by preserving
their topology, is a distinct operation that would be a separate mode.
`--linearize` only drops merges and replays commits linearly. So
git-replay(1) uses its own option rather than reusing that interface.
Based-on-patches-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The threshold for geometric repacking to trigger based on loose
object count has been adjusted to match that of 'git gc --auto',
preventing over-aggressive repacking during concurrent writes.
* ps/odb-geometric-repack-loose-threshold:
odb/files: be less aggressive with geometric repacking
The trailer parsing machinery has been updated to avoid mistaking
lines that begin with a URL (e.g., 'https://...') as trailer lines.
This prevents intended textual URLs from being mangled or mistakenly
treated as metadata keys.
* kh/trailers-no-urls:
trailers: stop recognizing URLs as trailers
The documentation for 'git format-rev' has been updated to use the
[synopsis] block definition on code blocks to properly highlight
placeholders, and a quoting inconsistency in the running text has
been fixed.
* kh/format-rev-doc-synopsis:
doc: format-rev: use [synopsis] on code block
doc: format-rev: quote subject placeholder before and after
When 'git worktree add <path>' is invoked without <commit-ish> and
with the --guess-remote option (or when worktree.guessRemote is set to
true), it tries to find a remote-tracking branch matching the basename
of <path>.
Currently, the behavior when multiple matches are found is the same as
when no match is found: it falls back to creating a branch from
HEAD. This has been the behavior since 71d6682d8c (worktree: add
--guess-remote option to add subcommand, 2017-11-29), when the option
was first introduced.
However, if the specified <path> matches any remote-tracking branch,
we infer that the user intended to use one of the remote-tracking
branches as the start-point rather than HEAD. So we abort the creation
of the branch and worktree when there are multiple matches, and
instruct the user to choose the start-point.
Signed-off-by: Yoichi NAKAYAMA <yoichi.nakayama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Scripts sometimes need the relative path from the current working
directory to the repository's working tree root (cdup). While this
information can be retrieved through `git rev-parse --show-cdup`,
`git repo info` does not currently expose it as a scriptable key.
Introduce the `path.cdup` key to `git repo info`. The key returns the
path from the current working directory to the root of the working tree,
returning the empty string when invoked from the working tree root.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Scripts sometimes need the path from the repository's working tree root
to the current working directory. While this information can be derived
through existing Git commands, `git repo info` does not currently expose
it as a scriptable key.
Introduce the `path.git-prefix` key to `git repo info`. The key returns
the path from the working tree root to the current working directory,
returning the empty string when invoked from the working tree root.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The repository grafts file specifies alternate parent relationships for
commits and may be used by repository tooling that needs to inspect or
manage grafts. Scripts currently retrieve its location by invoking
`git rev-parse --git-path info/grafts`.
Introduce `path.grafts.absolute` and `path.grafts.relative` keys to
`git repo info`. This exposes the grafts file location as a scriptable
config-like key using standard format rules, allowing scripts to
retrieve it through the same interface as other repository path
information.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The repository index is a fundamental component used by Git and related
tooling to track the working tree state. Scripts that interact with the
index currently retrieve its location by invoking
`git rev-parse --git-path index`.
Introduce `path.index.absolute` and `path.index.relative` keys to
`git repo info`. This exposes the index file location as a scriptable
config-like key using standard format rules, allowing scripts to
retrieve it through the same interface as other repository path
information.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Hooks are an integral part of a repository's configuration and are
commonly used by tooling to automate repository-specific workflows.
Currently, scripts typically retrieve the hooks directory by invoking
`git rev-parse --git-path hooks`.
Introduce `path.hooks.absolute` and `path.hooks.relative` keys to
`git repo info`. This exposes the hooks directory as a scriptable
config-like key using standard format rules, allowing scripts to
retrieve it through the same interface as other repository path
information.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Scripts working in multi-repository setups often need to identify the
top-level working tree of a superproject from within a submodule.
Currently, this is only exposed via `git rev-parse
--show-superproject-working-tree`.
Introduce `path.superproject-root.absolute` and
`path.superproject-root.relative` keys to `git repo info`.
This exposes the core submodule context via a scriptable config-like key
using standard format rules.
If requested when not inside a submodule, the command returns an empty
string.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Scripts frequently need to find the root directory of a repository's
working tree. Currently, this requires using `git rev-parse --show-toplevel`
or inferring it from other repository information.
Introduce `path.toplevel.absolute` and `path.toplevel.relative` keys
to `git repo info`. This allows scripts to retrieve the top-level
working tree path in a predictable, strictly formatted manner without
relying on `rev-parse`.
If requested in a bare repository where no working tree exists, the
command returns an empty string.
Mentored-by: Justin Tobler <jltobler@gmail.com>
Mentored-by: Lucas Seiki Oshiro <lucasseikioshiro@gmail.com>
Signed-off-by: K Jayatheerth <jayatheerthkulkarni2005@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'git repack' has been taught '--drop-filtered' to delete local
promisor blobs exceeding a limit (currently 'blob:limit=') in partial
clones, reclaiming space. Guards prevent running during other
operations or if referenced by the index.
* ss/repack-drop-filtered:
builtin/repack: add guards for --drop-filtered
builtin/repack: actually drop filtered promisor blobs
builtin/repack: enumerate promisor blobs for --drop-filtered
repack-promisor: allow excluding objects from the rebuilt promisor pack
list-objects-filter: add list_objects_filter__filter_oidset()
builtin/repack: add --drop-filtered and --dry-run options
The merge-base computation has been optimized by stopping the walk
early when one side's exclusive commits in the queue are exhausted,
yielding significant speedups for queries with one-sided histories.
* kk/merge-base-exhaustion:
commit-reach: remove commit-date ordering fallback
commit-reach: move min_generation check into paint_queue_get()
commit-reach: terminate merge-base walk when one paint side is exhausted
commit-reach: introduce struct paint_state with per-side counters
t6600: add clock-skew topologies and step counts for edge cases
commit-reach: add trace2 instrumentation to paint_down_to_common()
t6099: add side-exhaustion regression test
t6600: add test cases for side-exhaustion edge cases
test-lib-functions: improve diagnostic output for trace2 data assertions
Documentation/technical: add paint-down-to-common doc
The usage string of 'git fast-import' has been updated to use the
parse_options() API for displaying help, and its SYNOPSIS in the
documentation has been standardized to match.
* cc/fast-import-usage:
fast-import: remove useless from_stream argument
fast-import: use parse_options() for command line options
fast-import: use callbacks to parse some options
fast-import: use struct option for usage string
fast-import: move command state globals into 'struct fast_import_state'
fast-import: introduce 'struct fast_import_state'
fast-import: factor out option_*() functions
fast-import: use int for some bool flags
fast-import: localize 'i' into the 'for' loops using it
api-parse-options.adoc: document hidden and OPT_*_F option macros
api-parse-options.adoc: document per-option flags
parse-options: introduce OPT_HIDDEN_GROUP
The 'remote-object-info' command for 'git cat-file --batch-command'
has been extended to support the '%(objecttype)' placeholder.
* ps/cat-file-remote-object-info-type:
cat-file: unify default format
serve: advertise type capability
fetch-object-info: parse type from server response
protocol-caps: add type support to object-info
transport: drop remote object-info fields from transport struct
fetch-object-info: die() on the remaining error path
fetch-object-info: use dedicated struct for the results
fetch-object-info: pass arguments directly instead of a struct
fetch-object-info: detect malformed server responses
t5701: use test_file_size() to get the size of a file
When cloning from a server that supports packfile URIs we may see
multiple URIs being announced by the server. If so, the expectation is
that the client will download all of those packfiles. This is being done
sequentially, where we fetch one packfile after the other.
In many cases this should be fine, but there are scenarios where it's
not. When packfiles are for example hosted by object storage (think AWS
S3 or GCS) then the way to achieve high performance is typically to
parallelize downloading the data as a single connection is often capped
at a certain bandwidth. Furthermore, when the server announces a bunch
of smaller packfiles, then the overhead of establishing the connection
may eventually add up.
Despite the limitations caused by the network bandwidth and latency, Git
also runs git-index-pack(1) on all of the fetched packfiles. This is
another task that can be easily parallelized for another speedup.
All of these limitations can be addressed by parallelizing the fetch.
Introduce a new configuration option that allows the user to ask for
this: by default we continue to not parallelize the fetch to retain the
status quo. But when configured to 0 (where we auto-detect the number of
cores) or a value larger than 1 we perform the fetches concurrently.
With this infrastructure in place we can significantly speed up such
fetches. Using a local HTTP server demonstrates the speedup when using a
throttled connection of 2MB/s and downloading 8x1MB packfiles:
Benchmark 1: 2MB/s, 8x1MB packfiles, 1 threads
Time (mean ± σ): 4.321 s ± 0.003 s [User: 0.195 s, System: 0.113 s]
Range (min … max): 4.318 s … 4.325 s 5 runs
Benchmark 2: 2MB/s, 8x1MB packfiles, 2 threads
Time (mean ± σ): 2.284 s ± 0.241 s [User: 0.191 s, System: 0.114 s]
Range (min … max): 2.173 s … 2.714 s 5 runs
Benchmark 3: 2MB/s, 8x1MB packfiles, 4 threads
Time (mean ± σ): 1.212 s ± 0.238 s [User: 0.192 s, System: 0.105 s]
Range (min … max): 1.102 s … 1.638 s 5 runs
Benchmark 4: 2MB/s, 8x1MB packfiles, 8 threads
Time (mean ± σ): 569.9 ms ± 2.5 ms [User: 183.4 ms, System: 116.0 ms]
Range (min … max): 566.5 ms … 573.4 ms 5 runs
Summary
2MB/s, 8x1MB packfiles, 8 threads ran
2.13 ± 0.42 times faster than 2MB/s, 8x1MB packfiles, 4 threads
4.01 ± 0.42 times faster than 2MB/s, 8x1MB packfiles, 2 threads
7.58 ± 0.03 times faster than 2MB/s, 8x1MB packfiles, 1 threads
Quite unsurprisingly, we scale almost linearly with the number of
threads in this case as we're limited by the bandwidth of a single
connection. But we can also demonstrate a speedup on an unthrottled
connection when downloading slightly larger packfiles:
Benchmark 1: unthrottled, 8x16MB packfiles, 1 threads
Time (mean ± σ): 2.434 s ± 0.031 s [User: 2.067 s, System: 0.329 s]
Range (min … max): 2.381 s … 2.460 s 5 runs
Benchmark 2: unthrottled, 8x16MB packfiles, 2 threads
Time (mean ± σ): 1.353 s ± 0.129 s [User: 2.025 s, System: 0.328 s]
Range (min … max): 1.288 s … 1.583 s 5 runs
Benchmark 3: unthrottled, 8x16MB packfiles, 4 threads
Time (mean ± σ): 702.9 ms ± 23.9 ms [User: 1732.5 ms, System: 313.9 ms]
Range (min … max): 660.7 ms … 718.5 ms 5 runs
Benchmark 4: unthrottled, 8x16MB packfiles, 8 threads
Time (mean ± σ): 455.1 ms ± 7.7 ms [User: 1730.0 ms, System: 372.3 ms]
Range (min … max): 442.8 ms … 462.8 ms 5 runs
Summary
unthrottled, 8x16MB packfiles, 8 threads ran
1.54 ± 0.06 times faster than unthrottled, 8x16MB packfiles, 4 threads
2.97 ± 0.29 times faster than unthrottled, 8x16MB packfiles, 2 threads
5.35 ± 0.11 times faster than unthrottled, 8x16MB packfiles, 1 threads
In this case, the speedup is caused by us running git-index-pack(1) in
parallel. The improvement isn't linear, but still quite significant.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When performing auto-maintenance with geometric repacking we have two
conditions that may trigger a repack:
- Either the geometric sequence of packfiles is invalidated.
- Or we have too many loose objects.
The first condition shouldn't trigger all that often: it may be hit when
we fetch a new packfile, but users tend to not do that all the time. The
second condition is what typically triggers more regularly though, as
every command that ends up writing new objects may cause us to cross the
threshold of loose objects. It is thus preferable to not be too
aggressive here, as otherwise we may end up repacking objects quite
often.
For the geometric-repacking strategy though we have a default of 100
objects, only. As we're approximating the count of objects by only
reading the "objects/17/" shared, we'd only need 2 objects in there
before we perform a repack by default, which is quite aggressive.
git-gc(1) on the other hand has a default of 6700, so it is quite a bit
more conservative here.
Being this aggressive is also causing problems as reported by our users.
When running lots of concurrent writers, those writes will constantly
end up spawning maintenance jobs that end up repacking objects. As we
also prune objects, a concurrently running process that tries to write
an object may see that the sharding directories get removed under their
feet. While we try re-creating such leading directories, we only do so a
single time, and it may happen that the directory vanishes again before
we had the chance to create the loose object. This is not a new problem,
but it is exacerbated by us running maintenance this aggressively.
Improve the status quo by reducing the frequency at which we pack loose
objects to the same frequency that git-gc(1) uses.
Reported-by: Stefan Haller <lists@haller-berlin.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
An HTTPS URL starts with an alphanumeric scheme followed by a colon.
That means that they will be recognized as trailers in a trailer block.
That turns out to be a problem in practice. Let’s stop recognizing these
as trailers by failing the trailer parsing when we:
1. find the separator;
2. the separator and the next two characters form `://`; and
3. we haven’t parsed any whitespace yet.
The simplest example of how this can be a problem is for people who do
not use trailers but may leave URLs at the end of the commit message.
Now, while these authors might not use trailers themselves, other
authors may have used trailers and this metadata confusion can become a
problem once someone tries to extract that metadata (and non-metadata).
Let’s now look at some examples in the Linux Kernel[1] to see how this
is a problem in practice.
There are commits which contain intended non-trailer lines which start
with URLs. These are comments. Example with just the trailers:[2]
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
[bhelgaas: squash fixes:
https://lore.kernel.org/r/20260108013956.14351-2-bagasdotme@gmail.comhttps://lore.kernel.org/r/20260108013956.14351-3-bagasdotme@gmail.com]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Link: https://patch.msgid.link/20251210132907.58799-4-xueshuai@linux.alibaba.com
Those `[]` pairs delimit the “squash fixes” comment.
Now, any of these two commands:
git log --format='%(trailers:only)' -1 <commit>
git log -1 --format=%B <commit> |
git interpret-trailers --only-trailers
Will both wrongly (according to the surmised user intent) include these
two URL lines as trailers and also mangle the URLs, e.g.:
https: //lore.kernel.org/r/20260108013956.14351-2-bagasdotme@gmail.com
Because the `--only-trailers` mode (or `only` for the git-log(1) format)
normalizes the output to a colon and a space.
Another example is linewrapping mistakes; a `Link` trailer with a
URL where the URL ended up on the next line, presumably because the
user’s editor linewrapped the “too long” line. Example with just the
trailers:[3]
Link: https://patch.msgid.link/20260216-work-xattr-socket-v1-4-c2efa4f74cb7@kernel.org
Link:
https://lore.kernel.org/3cnmtqmakpbb2uwhenrj7kdqu3uefykiykjllgfbtpkiwhaa4s@sghkevv7jned [1]
Acked-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Christian Brauner <brauner@kernel.org>
Now, this intended trailer is already ruined, but interpreting the URL
as a standalone trailer only compounds the mistake.
Yet another example is the trailer machinery normalizing the trailer
block before application, resulting in a `https` trailer key in the
commit message itself. Example with just the trailers:[4]
https: //sashiko.dev/#/patchset/20260429114208.941011-1-holger.brunck%40hitachienergy.com
Fixes: c19b6d246a35 ("drivers/net: support hdlc function for QE-UCC")
Signed-off-by: Holger Brunck <holger.brunck@hitachienergy.com>
Link: https://patch.msgid.link/20260507155332.3452319-1-holger.brunck@hitachienergy.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
We have a helpful `Link` that points to the original patch.[5] Following
it we can see that that `https` trailer was indeed a URL
originally (again just the trailer block here):
https://sashiko.dev/#/patchset/20260429114208.941011-1-holger.brunck%40hitachienergy.com
Fixes: c19b6d246a35 ("drivers/net: support hdlc function for QE-UCC")
Signed-off-by: Holger Brunck <holger.brunck@hitachienergy.com>
So how did it end up as a `https` trailer? My theory is that the trailer
block was normalized on patch application, causing a URL comment to be
wrongly normalized and cemented in the commit message as a trailer.[6]
† 1: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/
† 2: commit 8236fc613d44e59f6736d6c3e9efffaf26ab7f00
† 3: commit 5bd97f5c5f241a5610c4412d1b93995a26241f81
† 4: commit 496c0c4c53bbe1bad97e82cd12103df61a6e459d
† 5: https://patch.msgid.link/20260507155332.3452319-1-holger.brunck@hitachienergy.com
† 6: There are only four commits in the Linux Kernel of this kind, and
three of them have the same recurring person in the signoff chain.
***
Note that this check has some benign false positives. A trailer key
can start with a digit, but a URL scheme can not start with a digit.
That means that a line that starts with `1://` will be rejected even
though it cannot be a URL. I don’t think this will reject any real
trailers, so I think the implementation simplicity is worth it.
And these false positives are just for a limited start fragment check;
a mere heuristic, not a URL parser.
Helped-by: Jeff King <peff@peff.net>
Acked-by: Jeff King <peff@peff.net>
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Open the editor by default when squashing and provide --no-edit as the
opt-out. Record the exact commits selected by the revision walk,
rearrange that todo list with the sequencer's autosquash machinery, and
build the message template from the resulting order.
Match interactive rebase's treatment of marker messages: comment out
fixup! messages, retain squash! bodies, and let amend! replace its target
unless a preceding squash! requires both bodies. This keeps message
editing aligned with the marker validation used by the no-edit path.
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Create one replacement commit from the resolved range when --no-edit is
selected. Preserve the authorship and all parents of the oldest commit,
use the tip tree, and replay descendants through the existing history
rewrite machinery. Record the complete revision expression in the
reflog and retain dry-run and update-refs behavior.
Resolve fixup!, squash! and amend! subjects while walking the range.
Reject markers whose targets are not selected and refuse any no-edit
fold that would discard a squash! or amend! message. A range made
entirely from related markers can still be consolidated, with the last
applicable amend! body supplying the message.
Inspired-by: Sergey Chernov <serega.morph@gmail.com>
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A local branch that descends from the selected graph without containing
its tip cannot be replayed as a descendant of the squashed commit. Find
those branches with ref-filter before creating any replacement objects
and refuse the operation unless --update-refs=head was requested.
Limit this protection to local branches, matching the refs that the
default history rewrite mode updates; tags and remote-tracking refs
remain untouched. Sort the blocking refs and print their short branch
names so the user can decide whether to move them or leave them behind.
Add advice.historyUpdateRefs for the hint that points to
--update-refs=head.
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add the entry point and option parsing for "git history squash". Pass
the remaining arguments through setup_revisions() so the command accepts
revision ranges and rev-list options, while restoring the ordering and
simplification settings required by the fold if an option changes them.
Require at least one BOTTOM revision. The squashed commit needs a commit
outside the selected range to serve as its base, so a single positive
revision is not a sufficient range.
Keep this step limited to defining the revision input contract so graph
validation and the rewrite can be added independently.
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano