Documentation: describe incremental MIDX format
Prepare to implement incremental multi-pack indexes (MIDXs) over the next several commits by first describing the relevant prerequisites (like a new chunk in the MIDX format, the directory structure for incremental MIDXs, etc.) The format is described in detail in the patch contents below, but the high-level description is as follows. Incremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and each `*.midx` within that directory has a single "parent" MIDX, which is the MIDX layer immediately before it in the MIDX chain. The chain order resides in a file 'multi-pack-index-chain' in the same directory. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>maint
Taylor Blau
Junio C Hamano
parent
04f5a52757
commit
6eb1a7d7b0
|
|
@ -61,6 +61,109 @@ Design Details
|
|||
- The MIDX file format uses a chunk-based approach (similar to the
|
||||
commit-graph file) that allows optional data to be added.
|
||||
|
||||
Incremental multi-pack indexes
|
||||
------------------------------
|
||||
|
||||
As repositories grow in size, it becomes more expensive to write a
|
||||
multi-pack index (MIDX) that includes all packfiles. To accommodate
|
||||
this, the "incremental multi-pack indexes" feature allows for combining
|
||||
a "chain" of multi-pack indexes.
|
||||
|
||||
Each individual component of the chain need only contain a small number
|
||||
of packfiles. Appending to the chain does not invalidate earlier parts
|
||||
of the chain, so repositories can control how much time is spent
|
||||
updating the MIDX chain by determining the number of packs in each layer
|
||||
of the MIDX chain.
|
||||
|
||||
=== Design state
|
||||
|
||||
At present, the incremental multi-pack indexes feature is missing two
|
||||
important components:
|
||||
|
||||
- The ability to rewrite earlier portions of the MIDX chain (i.e., to
|
||||
"compact" some collection of adjacent MIDX layers into a single
|
||||
MIDX). At present the only supported way of shrinking a MIDX chain
|
||||
is to rewrite the entire chain from scratch without the `--split`
|
||||
flag.
|
||||
+
|
||||
There are no fundamental limitations that stand in the way of being able
|
||||
to implement this feature. It is omitted from the initial implementation
|
||||
in order to reduce the complexity, but will be added later.
|
||||
|
||||
- Support for reachability bitmaps. The classic single MIDX
|
||||
implementation does support reachability bitmaps (see the section
|
||||
titled "multi-pack-index reverse indexes" in
|
||||
linkgit:gitformat-pack[5] for more details).
|
||||
+
|
||||
As above, there are no fundamental limitations that stand in the way of
|
||||
extending the incremental MIDX format to support reachability bitmaps.
|
||||
The design below specifically takes this into account, and support for
|
||||
reachability bitmaps will be added in a future patch series. It is
|
||||
omitted from the current implementation for the same reason as above.
|
||||
+
|
||||
In brief, to support reachability bitmaps with the incremental MIDX
|
||||
feature, the concept of the pseudo-pack order is extended across each
|
||||
layer of the incremental MIDX chain to form a concatenated pseudo-pack
|
||||
order. This concatenation takes place in the same order as the chain
|
||||
itself (in other words, the concatenated pseudo-pack order for a chain
|
||||
`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by
|
||||
the pseudo-pack order for `$H2`, followed by the pseudo-pack order for
|
||||
`$H3`).
|
||||
+
|
||||
The layout will then be extended so that each layer of the incremental
|
||||
MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap
|
||||
are offset by the number of objects in the previous layers of the chain.
|
||||
|
||||
=== File layout
|
||||
|
||||
Instead of storing a single `multi-pack-index` file (with an optional
|
||||
`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental
|
||||
MIDXs are stored in the following layout:
|
||||
|
||||
----
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx
|
||||
----
|
||||
|
||||
The `multi-pack-index-chain` file contains a list of the incremental
|
||||
MIDX files in the chain, in order. The above example shows a chain whose
|
||||
`multi-pack-index-chain` file would contain the following lines:
|
||||
|
||||
----
|
||||
$H1
|
||||
$H2
|
||||
$H3
|
||||
----
|
||||
|
||||
The `multi-pack-index-$H1.midx` file contains the first layer of the
|
||||
multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains
|
||||
the second layer of the chain, and so on.
|
||||
|
||||
When both an incremental- and non-incremental MIDX are present, the
|
||||
non-incremental MIDX is always read first.
|
||||
|
||||
=== Object positions for incremental MIDXs
|
||||
|
||||
In the original multi-pack-index design, we refer to objects via their
|
||||
lexicographic position (by object IDs) within the repository's singular
|
||||
multi-pack-index. In the incremental multi-pack-index design, we refer
|
||||
to objects via their index into a concatenated lexicographic ordering
|
||||
among each component in the MIDX chain.
|
||||
|
||||
If `objects_nr()` is a function that returns the number of objects in a
|
||||
given MIDX layer, then the index of an object at lexicographic position
|
||||
`i` within, say, $H3 is defined as:
|
||||
|
||||
----
|
||||
objects_nr($H2) + objects_nr($H1) + i
|
||||
----
|
||||
|
||||
(in the C implementation, this is often computed as `i +
|
||||
m->num_objects_in_base`).
|
||||
|
||||
Future Work
|
||||
-----------
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue