git/Documentation
Taylor Blau b2ec8e90c2 midx: do not require packs to be sorted in lexicographic order
The MIDX file format currently requires that pack files be identified by
the lexicographic ordering of their names (that is, a pack having a
checksum beginning with "abc" would have a numeric pack_int_id which is
smaller than the same value for a pack beginning with "bcd").

As a result, it is impossible to combine adjacent MIDX layers together
without permuting bits from bitmaps that are in more recent layer(s).

To see why, consider the following example:

          | packs       | preferred pack
  --------+-------------+---------------
  MIDX #0 | { X, Y, Z } | Y
  MIDX #1 | { A, B, C } | B
  MIDX #2 | { D, E, F } | D

, where MIDX #2's base MIDX is MIDX #1, and so on. Suppose that we want
to combine MIDX layers #0 and #1, to create a new layer #0' containing
the packs from both layers. With the original three MIDX layers, objects
are laid out in the bitmap in the order they appear in their source
pack, and the packs themselves are arranged according to the pseudo-pack
order. In this case, that ordering is Y, X, Z, B, A, C.

But recall that the pseudo-pack ordering is defined by the order that
packs appear in the MIDX, with the exception of the preferred pack,
which sorts ahead of all other packs regardless of its position within
the MIDX. In the above example, that means that pack 'Y' could be placed
anywhere (so long as it is designated as preferred), however, all other
packs must be placed in the location listed above.

Because that ordering isn't sorted lexicographically, it is impossible
to compact MIDX layers in the above configuration without permuting the
object-to-bit-position mapping. Changing this mapping would affect all
bitmaps belonging to newer layers, rendering the bitmaps associated with
MIDX #2 unreadable.

One of the goals of MIDX compaction is that we are able to shrink the
length of the MIDX chain *without* invalidating bitmaps that belong to
newer layers, and the lexicographic ordering constraint is at odds with
this goal.

However, packs do not *need* to be lexicographically ordered within the
MIDX. As far as I can gather, the only reason they are sorted lexically
is to make it possible to perform a binary search over the pack names in
a MIDX, necessary to make `midx_contains_pack()`'s performance
logarithmic in the number of packs rather than linear.

Relax this constraint by allowing MIDX writes to proceed with packs that
are not arranged in lexicographic order. `midx_contains_pack()` will
lazily instantiate a `pack_names_sorted` array on the MIDX, which will
be used to implement the binary search over pack names.

This change produces MIDXs which may not be correctly read with external
tools or older versions of Git. Though older versions of Git know how to
gracefully degrade and ignore any MIDX(s) they consider corrupt,
external tools may not be as robust. To avoid unintentionally breaking
any such tools, guard this change behind a version bump in the MIDX's
on-disk format.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-02-24 11:16:33 -08:00
..
RelNotes The 6th batch 2026-02-20 11:36:18 -08:00
config Merge branch 'pc/lockfile-pid' 2026-02-17 13:30:41 -08:00
howto meson: make GIT_HTML_PATH configurable 2025-11-06 09:58:56 -08:00
includes
mergetools doc: check well-formedness of delimited sections 2025-08-11 14:16:03 -07:00
technical Merge branch 'en/xdiff-cleanup-2' 2025-12-05 14:49:56 +09:00
.gitignore
BreakingChanges.adoc refs/files: deprecate writing symrefs as symbolic links 2025-10-15 09:11:08 -07:00
CodingGuidelines CodingGuidelines: document // comments 2026-02-11 13:29:36 -08:00
DecisionMaking.adoc
Makefile Merge branch 'en/xdiff-cleanup-2' 2025-12-05 14:49:56 +09:00
MyFirstContribution.adoc Merge branch 'sd/doc-my1c-api-config-reference-fix' 2026-02-13 13:39:26 -08:00
MyFirstObjectWalk.adoc Merge branch 'jc/you-still-use-whatchanged' 2025-06-25 14:07:35 -07:00
ReviewingGuidelines.adoc
SubmittingPatches Merge branch 'cc/doc-submitting-patches-with-ai' 2025-11-04 07:48:06 -08:00
ToolsForGit.adoc doc: add a blank line around block delimiters 2025-03-10 09:58:06 -07:00
asciidoc.conf.in doc: convert git-show to synopsis style 2026-02-05 21:14:05 -08:00
asciidoctor-extensions.rb.in doc: git-log: convert rev list options to new doc format 2025-07-07 13:45:00 -07:00
blame-options.adoc doc: blame-options: convert to new doc format 2026-01-09 06:15:31 -08:00
build-docdep.perl doc: fix build-docdep.perl 2025-03-01 10:26:15 -08:00
cat-texi.perl
cmd-list.sh Documentation: stop depending on Perl to generate command list 2025-04-16 07:30:30 -07:00
config.adoc Merge branch 'jc/doc-includeif-hasconfig-remote-url-fix' into maint-2.51 2025-10-15 10:29:34 -07:00
date-formats.adoc
diff-algorithm-option.adoc blame: make diff algorithm configurable 2025-11-17 09:31:59 -08:00
diff-context-options.adoc add-patch: add diff.context command line overrides 2025-07-29 08:52:45 -07:00
diff-format.adoc doc: check well-formedness of delimited sections 2025-08-11 14:16:03 -07:00
diff-generate-patch.adoc Merge branch 'ma/doc-diff-cc-headers' 2025-06-18 13:53:34 -07:00
diff-options.adoc blame: make diff algorithm configurable 2025-11-17 09:31:59 -08:00
doc-diff
docbook-xsl.css
docbook.xsl
docinfo-html.in
everyday.adoco
fetch-options.adoc doc: convert git pull to synopsis style 2025-11-19 15:00:42 -08:00
fix-texi.sh Documentation: stop depending on Perl to massage user manual 2025-04-16 07:30:29 -07:00
for-each-ref-options.adoc doc: fix some style issues in git-clone and for-each-ref-options 2026-02-05 21:14:05 -08:00
fsck-msgids.adoc builtin/fsck: move generic HEAD check into `refs_fsck()` 2026-01-12 06:55:41 -08:00
generate-mergetool-list.sh
git-add.adoc add-patch: fully document option P 2025-10-21 14:35:44 -07:00
git-am.adoc Merge branch 'kh/doc-committer-date-is-author-date' 2025-12-05 14:49:57 +09:00
git-annotate.adoc
git-apply.adoc apply docs: clarify wording for --intent-to-add 2025-07-07 06:41:11 -07:00
git-archimport.adoc
git-archive.adoc
git-backfill.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-bisect-lk2009.adoc
git-bisect.adoc bisect: update usage and docs to match each other 2025-10-28 15:41:42 -07:00
git-blame.adoc doc: git-blame: convert to new doc format 2026-01-09 06:15:53 -08:00
git-branch.adoc doc: branch: fix inline-verbatim 2025-05-05 10:48:07 -07:00
git-bugreport.adoc
git-bundle.adoc
git-cat-file.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-check-attr.adoc doc: check for absence of multiple terms in each entry of desc list 2025-08-11 14:16:04 -07:00
git-check-ignore.adoc doc: check for absence of multiple terms in each entry of desc list 2025-08-11 14:16:04 -07:00
git-check-mailmap.adoc
git-check-ref-format.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-checkout-index.adoc
git-checkout.adoc doc: fix asciidoc markup issues in several files 2025-12-20 14:55:43 +09:00
git-cherry-pick.adoc
git-cherry.adoc
git-citool.adoc
git-clean.adoc
git-clone.adoc doc: fix some style issues in git-clone and for-each-ref-options 2026-02-05 21:14:05 -08:00
git-column.adoc doc: column: fix blank lines around block delimiters 2025-06-01 17:20:16 -07:00
git-commit-graph.adoc commit-graph: add new config for changed-paths & recommend it in scalar 2025-10-22 10:40:11 -07:00
git-commit-tree.adoc
git-commit.adoc doc: commit: link to git-status(1) on all format options 2025-11-14 08:56:35 -08:00
git-config.adoc Merge branch 'ps/config-doc-get-urlmatch-fix' 2026-01-21 16:16:28 -08:00
git-count-objects.adoc Merge branch 'ds/doc-count-objects-fix' into maint-2.51 2025-10-14 13:40:54 -07:00
git-credential-cache--daemon.adoc
git-credential-cache.adoc Merge branch 'bc/doc-adoc-not-txt' 2025-02-14 17:53:47 -08:00
git-credential-store.adoc
git-credential.adoc
git-cvsexportcommit.adoc
git-cvsimport.adoc
git-cvsserver.adoc doc: add a blank line around block delimiters 2025-03-10 09:58:06 -07:00
git-daemon.adoc git-daemon doc: update mark-up of synopsis option descriptions 2025-05-12 09:25:26 -07:00
git-describe.adoc
git-diagnose.adoc
git-diff-files.adoc
git-diff-index.adoc
git-diff-pairs.adoc builtin/diff-pairs: allow explicit diff queue flush 2025-03-03 08:17:47 -08:00
git-diff-tree.adoc
git-diff.adoc diff --no-index: support limiting by pathspec 2025-05-22 14:20:11 -07:00
git-difftool.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-fast-export.adoc fast-(import|export): improve on commit signature output format 2025-07-09 16:08:29 -07:00
git-fast-import.adoc fast-import: add 'strip-if-invalid' mode to --signed-commits=<mode> 2025-11-26 08:43:44 -08:00
git-fetch-pack.adoc
git-fetch.adoc doc: convert git fetch to synopsis style 2025-11-19 15:00:37 -08:00
git-filter-branch.adoc
git-fmt-merge-msg.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-for-each-ref.adoc Merge branch 'ja/doc-lint-sections-and-synopsis' 2025-08-25 14:22:02 -07:00
git-for-each-repo.adoc
git-format-patch.adoc Merge branch 'js/doc-sending-patch-via-thunderbird' 2025-09-02 08:21:26 -07:00
git-fsck-objects.adoc
git-fsck.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-fsmonitor--daemon.adoc
git-gc.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-get-tar-commit-id.adoc
git-grep.adoc
git-gui.adoc
git-hash-object.adoc
git-help.adoc
git-history.adoc builtin/history: implement "reword" subcommand 2026-01-13 05:41:17 -08:00
git-hook.adoc
git-http-backend.adoc Merge branch 'ps/doc-http-upload-archive-service' 2025-02-14 17:53:49 -08:00
git-http-fetch.adoc doc: check for absence of multiple terms in each entry of desc list 2025-08-11 14:16:04 -07:00
git-http-push.adoc
git-imap-send.adoc docs: explain how to use `git imap-send --list` command to get a list of available folders 2025-07-22 08:49:15 -07:00
git-index-pack.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-init-db.adoc
git-init.adoc breaking-changes: switch default branch to main 2025-09-10 13:34:58 -07:00
git-instaweb.adoc
git-interpret-trailers.adoc doc: interpret-trailers: close all pairs of single quotes 2025-08-22 11:12:27 -07:00
git-last-modified.adoc last-modified: change default max-depth to 0 2026-01-20 14:13:04 -08:00
git-log.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-ls-files.adoc
git-ls-remote.adoc
git-ls-tree.adoc
git-mailinfo.adoc
git-mailsplit.adoc
git-maintenance.adoc maintenance: add 'is-needed' subcommand 2025-11-10 09:28:48 -08:00
git-merge-base.adoc
git-merge-file.adoc merge-file: honor merge.conflictStyle outside of a repository 2026-02-07 17:04:26 -08:00
git-merge-index.adoc
git-merge-one-file.adoc
git-merge-tree.adoc Merge branch 'en/doc-merge-tree-describe-merge-base' 2025-10-10 12:51:46 -07:00
git-merge.adoc Merge branch 'ts/merge-orig-head-doc-fix' 2025-07-14 11:19:28 -07:00
git-mergetool--lib.adoc
git-mergetool.adoc doc: convert git-mergetool manpage to new synopsis style 2025-05-27 07:51:01 -07:00
git-mktag.adoc
git-mktree.adoc
git-multi-pack-index.adoc git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h' 2026-02-24 11:16:32 -08:00
git-mv.adoc doc: convert git-mv to new documentation format 2025-04-14 14:43:52 -07:00
git-name-rev.adoc
git-notes.adoc doc: notes: use stuck form throughout 2025-05-27 15:31:08 -07:00
git-p4.adoc doc: check well-formedness of delimited sections 2025-08-11 14:16:03 -07:00
git-pack-objects.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-pack-redundant.adoc
git-pack-refs.adoc doc: pack-refs: factor out common options 2025-09-19 10:02:56 -07:00
git-patch-id.adoc doc: patch-id: --verbatim locks in --stable 2026-01-09 06:08:37 -08:00
git-prune-packed.adoc
git-prune.adoc
git-pull.adoc Merge branch 'je/doc-pull' 2025-12-14 17:04:37 +09:00
git-push.adoc doc: convert git push to synopsis style 2025-11-19 15:00:45 -08:00
git-quiltimport.adoc
git-range-diff.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-read-tree.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-rebase.adoc Merge branch 'ja/doc-misc-fixes' 2025-12-28 17:36:16 +09:00
git-receive-pack.adoc builtin/receive-pack: add option to skip connectivity check 2025-05-20 11:43:36 -07:00
git-reflog.adoc builtin/reflog: implement subcommand to write new entries 2025-08-06 07:36:30 -07:00
git-refs.adoc Merge branch 'ms/refs-optimize' 2025-10-02 12:26:12 -07:00
git-remote-ext.adoc
git-remote-fd.adoc
git-remote-helpers.adoco
git-remote.adoc doc: convert git-remote to synopsis style 2025-12-21 11:33:10 +09:00
git-repack.adoc doc: correct minor wording issues 2025-12-20 14:55:43 +09:00
git-replace.adoc
git-replay.adoc Merge branch 'pw/replay-drop-empty' 2026-02-09 12:09:09 -08:00
git-repo.adoc Merge branch 'jt/repo-struct-more-objinfo' 2025-12-30 12:58:19 +09:00
git-request-pull.adoc
git-rerere.adoc rerere: minor documantation update 2026-02-09 11:58:53 -08:00
git-reset.adoc doc: git-reset: clarify `git reset <pathspec>` 2026-01-06 08:10:14 +09:00
git-restore.adoc Merge branch 'lm/add-p-context' 2025-08-04 08:10:33 -07:00
git-rev-list.adoc
git-rev-parse.adoc Merge branch 'bc/sha1-256-interop-01' 2025-10-22 11:38:58 -07:00
git-revert.adoc
git-rm.adoc doc: convert git-rm to new documentation format 2025-04-14 14:43:52 -07:00
git-send-email.adoc doc: send-email: correct --no-signed-off-by-cc misspelling 2026-02-05 11:38:54 -08:00
git-send-pack.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-sh-i18n--envsubst.adoc
git-sh-i18n.adoc
git-sh-setup.adoc
git-shell.adoc
git-shortlog.adoc Merge branch 'kh/doc-shortlog-fix' 2026-02-11 12:29:07 -08:00
git-show-branch.adoc
git-show-index.adoc Merge branch 'bc/doc-adoc-not-txt' 2025-02-14 17:53:47 -08:00
git-show-ref.adoc
git-show.adoc doc: convert git-show to synopsis style 2026-02-05 21:14:05 -08:00
git-sparse-checkout.adoc Merge branch 'ds/sparse-checkout-clean' 2025-10-28 10:29:09 -07:00
git-stage.adoc doc: convert git stage to use synopsis block 2025-12-21 11:33:10 +09:00
git-stash.adoc doc: convert git-stash.adoc to synopis style 2025-10-05 16:09:03 -07:00
git-status.adoc doc: convert git-status tables to AsciiDoc format 2025-12-21 11:33:10 +09:00
git-stripspace.adoc doc: stripspace: mention where the default comes from 2025-05-27 15:31:05 -07:00
git-submodule.adoc doc: convert git-submodule to synopsis style 2026-02-05 21:14:05 -08:00
git-svn.adoc doc: check well-formedness of delimited sections 2025-08-11 14:16:03 -07:00
git-switch.adoc builtin: unmark git-switch and git-restore as experimental 2025-07-28 14:24:03 -07:00
git-symbolic-ref.adoc
git-tag.adoc Merge branch 'cc/fast-import-strip-signed-tags' 2025-10-28 10:29:09 -07:00
git-tools.adoc
git-unpack-file.adoc
git-unpack-objects.adoc
git-update-index.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-update-ref.adoc Merge branch 'sb/doc-update-ref-markup-fix' 2026-01-23 13:34:35 -08:00
git-update-server-info.adoc
git-upload-archive.adoc
git-upload-pack.adoc doc: check for absence of the form --[no-]parameter 2025-08-11 14:16:04 -07:00
git-var.adoc git-var doc: fix usage of $ENV_VAR vs ENV_VAR 2025-05-12 09:25:27 -07:00
git-verify-commit.adoc git-verify-* doc: update mark-up of synopsis option descriptions 2025-05-12 09:25:27 -07:00
git-verify-pack.adoc git-verify-* doc: update mark-up of synopsis option descriptions 2025-05-12 09:25:27 -07:00
git-verify-tag.adoc git-verify-* doc: update mark-up of synopsis option descriptions 2025-05-12 09:25:27 -07:00
git-version.adoc help: include unsafe SHA-1 build info in version 2025-04-07 14:39:27 -07:00
git-web--browse.adoc
git-whatchanged.adoc whatchanged: remove not-even-shorter clause 2025-09-17 13:47:24 -07:00
git-worktree.adoc Merge branch 'sb/doc-worktree-prune-expire-improvement' 2026-02-09 12:09:10 -08:00
git-write-tree.adoc git-{var,write-tree} docs: update mark-up of synopsis option descriptions 2025-05-12 09:25:26 -07:00
git.adoc Merge branch 'js/progress-delay-fix' into maint-2.51 2025-10-14 13:40:54 -07:00
gitattributes.adoc doc: add a blank line around block delimiters 2025-03-10 09:58:06 -07:00
gitcli.adoc doc: clarify command equivalence comment 2025-11-04 09:25:51 -08:00
gitcore-tutorial.adoc
gitcredentials.adoc Merge branch 'mh/doc-credential-url-prefix' into maint-2.51 2025-10-15 10:29:35 -07:00
gitcvs-migration.adoc
gitdatamodel.adoc doc: remove stray text in Git data model 2025-12-03 00:17:07 -08:00
gitdiffcore.adoc
giteveryday.adoc
gitfaq.adoc Merge branch 'bc/doc-stash-import-export' 2026-01-16 12:40:27 -08:00
gitformat-bundle.adoc
gitformat-chunk.adoc
gitformat-commit-graph.adoc
gitformat-index.adoc
gitformat-loose.adoc docs: add documentation for loose objects 2025-10-09 17:46:14 -07:00
gitformat-pack.adoc midx: do not require packs to be sorted in lexicographic order 2026-02-24 11:16:33 -08:00
gitformat-signature.adoc
gitglossary.adoc
githooks.adoc doc: join default pre-commit paragraphs 2025-12-08 22:20:14 +09:00
gitignore.adoc doc: document backslash in gitignore patterns 2025-10-29 09:17:21 -07:00
gitk.adoc Merge branch 'js/doc-gitk-history' into maint-2.51 2025-10-14 13:40:53 -07:00
gitmailmap.adoc
gitmodules.adoc
gitnamespaces.adoc
gitpacking.adoc
gitprotocol-capabilities.adoc
gitprotocol-common.adoc doc: add a blank line around block delimiters 2025-03-10 09:58:06 -07:00
gitprotocol-http.adoc doc: clarify server behavior for invalid 'want' lines in HTTP protocol 2025-11-06 09:45:38 -08:00
gitprotocol-pack.adoc
gitprotocol-v2.adoc promisor-remote: allow a server to advertise more fields 2025-09-08 10:30:54 -07:00
gitremote-helpers.adoc Merge branch 'bs/remote-helpers-doc-markup-fix' 2025-07-14 11:19:22 -07:00
gitrepository-layout.adoc docs: fix repository-layout when building with breaking changes 2025-03-05 07:25:11 -08:00
gitrevisions.adoc
gitsubmodules.adoc doc: check well-formedness of delimited sections 2025-08-11 14:16:03 -07:00
gittutorial-2.adoc
gittutorial.adoc
gitweb.adoc doc: add a blank line around block delimiters 2025-03-10 09:58:06 -07:00
gitweb.conf.adoc doc: test linkgit macros for well-formedness 2025-08-11 14:16:03 -07:00
gitworkflows.adoc
glossary-content.adoc doc: add an explanation of Git's data model 2025-11-12 12:21:15 -08:00
i18n.adoc Merge branch 'bc/doc-adoc-not-txt' 2025-02-14 17:53:47 -08:00
install-doc-quick.sh
install-webdoc.sh
line-range-format.adoc doc: git-log: convert line range format to new doc format 2025-07-07 13:45:00 -07:00
line-range-options.adoc doc: git-log: convert line range options to new doc format 2025-07-07 13:45:00 -07:00
lint-delimited-sections.perl doc: check well-formedness of delimited sections 2025-08-11 14:16:03 -07:00
lint-documentation-style.perl doc lint: check that synopsis manpages have synopsis inlines 2025-08-11 14:16:04 -07:00
lint-fsck-msgids.perl
lint-gitlink.perl lint-gitlink: preemptively ignore all /ifn?def|endif/ macros 2026-01-21 08:26:00 -08:00
lint-man-end-blurb.perl
lint-man-section-order.perl
lint-manpages.sh
manpage-bold-literal.xsl
manpage-normal.xsl
manpage.xsl
merge-options.adoc doc: convert git pull to synopsis style 2025-11-19 15:00:42 -08:00
merge-strategies.adoc doc: convert merge strategies to synopsis format 2025-05-27 07:51:00 -07:00
meson.build builtin: add new "history" command 2026-01-13 05:41:17 -08:00
object-format-disclaimer.adoc
pack-refs-options.adoc doc: pack-refs: factor out common options 2025-09-19 10:02:56 -07:00
pretty-formats.adoc doc: convert git-show to synopsis style 2026-02-05 21:14:05 -08:00
pretty-options.adoc doc: do not break sentences into "lego" pieces 2025-10-05 16:10:53 -07:00
pull-fetch-param.adoc doc: pull-fetch-param typofix 2025-11-24 10:55:48 -08:00
ref-reachability-filters.adoc
ref-storage-format.adoc doc: fix some style issues in git-clone and for-each-ref-options 2026-02-05 21:14:05 -08:00
rerere-options.adoc doc: rerere-options.adoc: link to git-rerere(1) 2026-02-10 12:27:07 -08:00
rev-list-description.adoc doc: git-log convert rev-list-description to new doc format 2025-07-07 13:45:00 -07:00
rev-list-options.adoc replay: drop rev-list formatting options from manual 2026-01-20 09:38:56 -08:00
revisions.adoc
scalar.adoc scalar: document config settings 2025-12-16 09:42:44 +09:00
sequencer.adoc
signoff-option.adoc signoff-option: linkify the reference to gitfaq 2025-12-19 21:51:01 +09:00
texi.xsl
trace2-target-values.adoc
transfer-data-leaks.adoc
urls-remotes.adoc doc: convert git pull to synopsis style 2025-11-19 15:00:42 -08:00
urls.adoc
user-manual.adoc config: drop `git_config()` wrapper 2025-07-23 08:15:18 -07:00