You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1093 lines
36 KiB
1093 lines
36 KiB
reftable |
|
-------- |
|
|
|
Overview |
|
~~~~~~~~ |
|
|
|
Problem statement |
|
^^^^^^^^^^^^^^^^^ |
|
|
|
Some repositories contain a lot of references (e.g. android at 866k, |
|
rails at 31k). The existing packed-refs format takes up a lot of space |
|
(e.g. 62M), and does not scale with additional references. Lookup of a |
|
single reference requires linearly scanning the file. |
|
|
|
Atomic pushes modifying multiple references require copying the entire |
|
packed-refs file, which can be a considerable amount of data moved |
|
(e.g. 62M in, 62M out) for even small transactions (2 refs modified). |
|
|
|
Repositories with many loose references occupy a large number of disk |
|
blocks from the local file system, as each reference is its own file |
|
storing 41 bytes (and another file for the corresponding reflog). This |
|
negatively affects the number of inodes available when a large number of |
|
repositories are stored on the same filesystem. Readers can be penalized |
|
due to the larger number of syscalls required to traverse and read the |
|
`$GIT_DIR/refs` directory. |
|
|
|
|
|
Objectives |
|
^^^^^^^^^^ |
|
|
|
* Near constant time lookup for any single reference, even when the |
|
repository is cold and not in process or kernel cache. |
|
* Near constant time verification if an object name is referred to by at least |
|
one reference (for allow-tip-sha1-in-want). |
|
* Efficient enumeration of an entire namespace, such as `refs/tags/`. |
|
* Support atomic push with `O(size_of_update)` operations. |
|
* Combine reflog storage with ref storage for small transactions. |
|
* Separate reflog storage for base refs and historical logs. |
|
|
|
Description |
|
^^^^^^^^^^^ |
|
|
|
A reftable file is a portable binary file format customized for |
|
reference storage. References are sorted, enabling linear scans, binary |
|
search lookup, and range scans. |
|
|
|
Storage in the file is organized into variable sized blocks. Prefix |
|
compression is used within a single block to reduce disk space. Block |
|
size and alignment is tunable by the writer. |
|
|
|
Performance |
|
^^^^^^^^^^^ |
|
|
|
Space used, packed-refs vs. reftable: |
|
|
|
[cols=",>,>,>,>,>",options="header",] |
|
|=============================================================== |
|
|repository |packed-refs |reftable |% original |avg ref |avg obj |
|
|android |62.2 M |36.1 M |58.0% |33 bytes |5 bytes |
|
|rails |1.8 M |1.1 M |57.7% |29 bytes |4 bytes |
|
|git |78.7 K |48.1 K |61.0% |50 bytes |4 bytes |
|
|git (heads) |332 b |269 b |81.0% |33 bytes |0 bytes |
|
|=============================================================== |
|
|
|
Scan (read 866k refs), by reference name lookup (single ref from 866k |
|
refs), and by SHA-1 lookup (refs with that SHA-1, from 866k refs): |
|
|
|
[cols=",>,>,>,>",options="header",] |
|
|========================================================= |
|
|format |cache |scan |by name |by SHA-1 |
|
|packed-refs |cold |402 ms |409,660.1 usec |412,535.8 usec |
|
|packed-refs |hot | |6,844.6 usec |20,110.1 usec |
|
|reftable |cold |112 ms |33.9 usec |323.2 usec |
|
|reftable |hot | |20.2 usec |320.8 usec |
|
|========================================================= |
|
|
|
Space used for 149,932 log entries for 43,061 refs, reflog vs. reftable: |
|
|
|
[cols=",>,>",options="header",] |
|
|================================ |
|
|format |size |avg entry |
|
|$GIT_DIR/logs |173 M |1209 bytes |
|
|reftable |5 M |37 bytes |
|
|================================ |
|
|
|
Details |
|
~~~~~~~ |
|
|
|
Peeling |
|
^^^^^^^ |
|
|
|
References stored in a reftable are peeled, a record for an annotated |
|
(or signed) tag records both the tag object, and the object it refers |
|
to. This is analogous to storage in the packed-refs format. |
|
|
|
Reference name encoding |
|
^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Reference names are an uninterpreted sequence of bytes that must pass |
|
linkgit:git-check-ref-format[1] as a valid reference name. |
|
|
|
Key unicity |
|
^^^^^^^^^^^ |
|
|
|
Each entry must have a unique key; repeated keys are disallowed. |
|
|
|
Network byte order |
|
^^^^^^^^^^^^^^^^^^ |
|
|
|
All multi-byte, fixed width fields are in network byte order. |
|
|
|
Varint encoding |
|
^^^^^^^^^^^^^^^ |
|
|
|
Varint encoding is identical to the ofs-delta encoding method used |
|
within pack files. |
|
|
|
Decoder works such as: |
|
|
|
.... |
|
val = buf[ptr] & 0x7f |
|
while (buf[ptr] & 0x80) { |
|
ptr++ |
|
val = ((val + 1) << 7) | (buf[ptr] & 0x7f) |
|
} |
|
.... |
|
|
|
Ordering |
|
^^^^^^^^ |
|
|
|
Blocks are lexicographically ordered by their first reference. |
|
|
|
Directory/file conflicts |
|
^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
The reftable format accepts both `refs/heads/foo` and |
|
`refs/heads/foo/bar` as distinct references. |
|
|
|
This property is useful for retaining log records in reftable, but may |
|
confuse versions of Git using `$GIT_DIR/refs` directory tree to maintain |
|
references. Users of reftable may choose to continue to reject `foo` and |
|
`foo/bar` type conflicts to prevent problems for peers. |
|
|
|
File format |
|
~~~~~~~~~~~ |
|
|
|
Structure |
|
^^^^^^^^^ |
|
|
|
A reftable file has the following high-level structure: |
|
|
|
.... |
|
first_block { |
|
header |
|
first_ref_block |
|
} |
|
ref_block* |
|
ref_index* |
|
obj_block* |
|
obj_index* |
|
log_block* |
|
log_index* |
|
footer |
|
.... |
|
|
|
A log-only file omits the `ref_block`, `ref_index`, `obj_block` and |
|
`obj_index` sections, containing only the file header and log block: |
|
|
|
.... |
|
first_block { |
|
header |
|
} |
|
log_block* |
|
log_index* |
|
footer |
|
.... |
|
|
|
in a log-only file the first log block immediately follows the file |
|
header, without padding to block alignment. |
|
|
|
Block size |
|
^^^^^^^^^^ |
|
|
|
The file's block size is arbitrarily determined by the writer, and does |
|
not have to be a power of 2. The block size must be larger than the |
|
longest reference name or log entry used in the repository, as |
|
references cannot span blocks. |
|
|
|
Powers of two that are friendly to the virtual memory system or |
|
filesystem (such as 4k or 8k) are recommended. Larger sizes (64k) can |
|
yield better compression, with a possible increased cost incurred by |
|
readers during access. |
|
|
|
The largest block size is `16777215` bytes (15.99 MiB). |
|
|
|
Block alignment |
|
^^^^^^^^^^^^^^^ |
|
|
|
Writers may choose to align blocks at multiples of the block size by |
|
including `padding` filled with NUL bytes at the end of a block to round |
|
out to the chosen alignment. When alignment is used, writers must |
|
specify the alignment with the file header's `block_size` field. |
|
|
|
Block alignment is not required by the file format. Unaligned files must |
|
set `block_size = 0` in the file header, and omit `padding`. Unaligned |
|
files with more than one ref block must include the link:#Ref-index[ref |
|
index] to support fast lookup. Readers must be able to read both aligned |
|
and non-aligned files. |
|
|
|
Very small files (e.g. a single ref block) may omit `padding` and the ref |
|
index to reduce total file size. |
|
|
|
Header (version 1) |
|
^^^^^^^^^^^^^^^^^^ |
|
|
|
A 24-byte header appears at the beginning of the file: |
|
|
|
.... |
|
'REFT' |
|
uint8( version_number = 1 ) |
|
uint24( block_size ) |
|
uint64( min_update_index ) |
|
uint64( max_update_index ) |
|
.... |
|
|
|
Aligned files must specify `block_size` to configure readers with the |
|
expected block alignment. Unaligned files must set `block_size = 0`. |
|
|
|
The `min_update_index` and `max_update_index` describe bounds for the |
|
`update_index` field of all log records in this file. When reftables are |
|
used in a stack for link:#Update-transactions[transactions], these |
|
fields can order the files such that the prior file's |
|
`max_update_index + 1` is the next file's `min_update_index`. |
|
|
|
Header (version 2) |
|
^^^^^^^^^^^^^^^^^^ |
|
|
|
A 28-byte header appears at the beginning of the file: |
|
|
|
.... |
|
'REFT' |
|
uint8( version_number = 2 ) |
|
uint24( block_size ) |
|
uint64( min_update_index ) |
|
uint64( max_update_index ) |
|
uint32( hash_id ) |
|
.... |
|
|
|
The header is identical to `version_number=1`, with the 4-byte hash ID |
|
("sha1" for SHA1 and "s256" for SHA-256) append to the header. |
|
|
|
For maximum backward compatibility, it is recommended to use version 1 when |
|
writing SHA1 reftables. |
|
|
|
First ref block |
|
^^^^^^^^^^^^^^^ |
|
|
|
The first ref block shares the same block as the file header, and is 24 |
|
bytes smaller than all other blocks in the file. The first block |
|
immediately begins after the file header, at position 24. |
|
|
|
If the first block is a log block (a log-only file), its block header |
|
begins immediately at position 24. |
|
|
|
Ref block format |
|
^^^^^^^^^^^^^^^^ |
|
|
|
A ref block is written as: |
|
|
|
.... |
|
'r' |
|
uint24( block_len ) |
|
ref_record+ |
|
uint24( restart_offset )+ |
|
uint16( restart_count ) |
|
|
|
padding? |
|
.... |
|
|
|
Blocks begin with `block_type = 'r'` and a 3-byte `block_len` which |
|
encodes the number of bytes in the block up to, but not including the |
|
optional `padding`. This is always less than or equal to the file's |
|
block size. In the first ref block, `block_len` includes 24 bytes for |
|
the file header. |
|
|
|
The 2-byte `restart_count` stores the number of entries in the |
|
`restart_offset` list, which must not be empty. Readers can use |
|
`restart_count` to binary search between restarts before starting a |
|
linear scan. |
|
|
|
Exactly `restart_count` 3-byte `restart_offset` values precedes the |
|
`restart_count`. Offsets are relative to the start of the block and |
|
refer to the first byte of any `ref_record` whose name has not been |
|
prefix compressed. Entries in the `restart_offset` list must be sorted, |
|
ascending. Readers can start linear scans from any of these records. |
|
|
|
A variable number of `ref_record` fill the middle of the block, |
|
describing reference names and values. The format is described below. |
|
|
|
As the first ref block shares the first file block with the file header, |
|
all `restart_offset` in the first block are relative to the start of the |
|
file (position 0), and include the file header. This forces the first |
|
`restart_offset` to be `28`. |
|
|
|
ref record |
|
++++++++++ |
|
|
|
A `ref_record` describes a single reference, storing both the name and |
|
its value(s). Records are formatted as: |
|
|
|
.... |
|
varint( prefix_length ) |
|
varint( (suffix_length << 3) | value_type ) |
|
suffix |
|
varint( update_index_delta ) |
|
value? |
|
.... |
|
|
|
The `prefix_length` field specifies how many leading bytes of the prior |
|
reference record's name should be copied to obtain this reference's |
|
name. This must be 0 for the first reference in any block, and also must |
|
be 0 for any `ref_record` whose offset is listed in the `restart_offset` |
|
table at the end of the block. |
|
|
|
Recovering a reference name from any `ref_record` is a simple concat: |
|
|
|
.... |
|
this_name = prior_name[0..prefix_length] + suffix |
|
.... |
|
|
|
The `suffix_length` value provides the number of bytes available in |
|
`suffix` to copy from `suffix` to complete the reference name. |
|
|
|
The `update_index` that last modified the reference can be obtained by |
|
adding `update_index_delta` to the `min_update_index` from the file |
|
header: `min_update_index + update_index_delta`. |
|
|
|
The `value` follows. Its format is determined by `value_type`, one of |
|
the following: |
|
|
|
* `0x0`: deletion; no value data (see transactions, below) |
|
* `0x1`: one object name; value of the ref |
|
* `0x2`: two object names; value of the ref, peeled target |
|
* `0x3`: symbolic reference: `varint( target_len ) target` |
|
|
|
Symbolic references use `0x3`, followed by the complete name of the |
|
reference target. No compression is applied to the target name. |
|
|
|
Types `0x4..0x7` are reserved for future use. |
|
|
|
Ref index |
|
^^^^^^^^^ |
|
|
|
The ref index stores the name of the last reference from every ref block |
|
in the file, enabling reduced disk seeks for lookups. Any reference can |
|
be found by searching the index, identifying the containing block, and |
|
searching within that block. |
|
|
|
The index may be organized into a multi-level index, where the 1st level |
|
index block points to additional ref index blocks (2nd level), which may |
|
in turn point to either additional index blocks (e.g. 3rd level) or ref |
|
blocks (leaf level). Disk reads required to access a ref go up with |
|
higher index levels. Multi-level indexes may be required to ensure no |
|
single index block exceeds the file format's max block size of |
|
`16777215` bytes (15.99 MiB). To achieve constant O(1) disk seeks for |
|
lookups the index must be a single level, which is permitted to exceed |
|
the file's configured block size, but not the format's max block size of |
|
15.99 MiB. |
|
|
|
If present, the ref index block(s) appears after the last ref block. |
|
|
|
If there are at least 4 ref blocks, a ref index block should be written |
|
to improve lookup times. Cold reads using the index require 2 disk reads |
|
(read index, read block), and binary searching < 4 blocks also requires |
|
<= 2 reads. Omitting the index block from smaller files saves space. |
|
|
|
If the file is unaligned and contains more than one ref block, the ref |
|
index must be written. |
|
|
|
Index block format: |
|
|
|
.... |
|
'i' |
|
uint24( block_len ) |
|
index_record+ |
|
uint24( restart_offset )+ |
|
uint16( restart_count ) |
|
|
|
padding? |
|
.... |
|
|
|
The index blocks begin with `block_type = 'i'` and a 3-byte `block_len` |
|
which encodes the number of bytes in the block, up to but not including |
|
the optional `padding`. |
|
|
|
The `restart_offset` and `restart_count` fields are identical in format, |
|
meaning and usage as in ref blocks. |
|
|
|
To reduce the number of reads required for random access in very large |
|
files the index block may be larger than other blocks. However, readers |
|
must hold the entire index in memory to benefit from this, so it's a |
|
time-space tradeoff in both file size and reader memory. |
|
|
|
Increasing the file's block size decreases the index size. Alternatively |
|
a multi-level index may be used, keeping index blocks within the file's |
|
block size, but increasing the number of blocks that need to be |
|
accessed. |
|
|
|
index record |
|
++++++++++++ |
|
|
|
An index record describes the last entry in another block. Index records |
|
are written as: |
|
|
|
.... |
|
varint( prefix_length ) |
|
varint( (suffix_length << 3) | 0 ) |
|
suffix |
|
varint( block_position ) |
|
.... |
|
|
|
Index records use prefix compression exactly like `ref_record`. |
|
|
|
Index records store `block_position` after the suffix, specifying the |
|
absolute position in bytes (from the start of the file) of the block |
|
that ends with this reference. Readers can seek to `block_position` to |
|
begin reading the block header. |
|
|
|
Readers must examine the block header at `block_position` to determine |
|
if the next block is another level index block, or the leaf-level ref |
|
block. |
|
|
|
Reading the index |
|
+++++++++++++++++ |
|
|
|
Readers loading the ref index must first read the footer (below) to |
|
obtain `ref_index_position`. If not present, the position will be 0. The |
|
`ref_index_position` is for the 1st level root of the ref index. |
|
|
|
Obj block format |
|
^^^^^^^^^^^^^^^^ |
|
|
|
Object blocks are optional. Writers may choose to omit object blocks, |
|
especially if readers will not use the object name to ref mapping. |
|
|
|
Object blocks use unique, abbreviated 2-32 object name keys, mapping to |
|
ref blocks containing references pointing to that object directly, or as |
|
the peeled value of an annotated tag. Like ref blocks, object blocks use |
|
the file's standard block size. The abbreviation length is available in |
|
the footer as `obj_id_len`. |
|
|
|
To save space in small files, object blocks may be omitted if the ref |
|
index is not present, as brute force search will only need to read a few |
|
ref blocks. When missing, readers should brute force a linear search of |
|
all references to lookup by object name. |
|
|
|
An object block is written as: |
|
|
|
.... |
|
'o' |
|
uint24( block_len ) |
|
obj_record+ |
|
uint24( restart_offset )+ |
|
uint16( restart_count ) |
|
|
|
padding? |
|
.... |
|
|
|
Fields are identical to ref block. Binary search using the restart table |
|
works the same as in reference blocks. |
|
|
|
Because object names are abbreviated by writers to the shortest unique |
|
abbreviation within the reftable, obj key lengths have a variable length. Their |
|
length must be at least 2 bytes. Readers must compare only for common prefix |
|
match within an obj block or obj index. |
|
|
|
obj record |
|
++++++++++ |
|
|
|
An `obj_record` describes a single object abbreviation, and the blocks |
|
containing references using that unique abbreviation: |
|
|
|
.... |
|
varint( prefix_length ) |
|
varint( (suffix_length << 3) | cnt_3 ) |
|
suffix |
|
varint( cnt_large )? |
|
varint( position_delta )* |
|
.... |
|
|
|
Like in reference blocks, abbreviations are prefix compressed within an |
|
obj block. On large reftables with many unique objects, higher block |
|
sizes (64k), and higher restart interval (128), a `prefix_length` of 2 |
|
or 3 and `suffix_length` of 3 may be common in obj records (unique |
|
abbreviation of 5-6 raw bytes, 10-12 hex digits). |
|
|
|
Each record contains `position_count` number of positions for matching |
|
ref blocks. For 1-7 positions the count is stored in `cnt_3`. When |
|
`cnt_3 = 0` the actual count follows in a varint, `cnt_large`. |
|
|
|
The use of `cnt_3` bets most objects are pointed to by only a single |
|
reference, some may be pointed to by a couple of references, and very |
|
few (if any) are pointed to by more than 7 references. |
|
|
|
A special case exists when `cnt_3 = 0` and `cnt_large = 0`: there are no |
|
`position_delta`, but at least one reference starts with this |
|
abbreviation. A reader that needs exact reference names must scan all |
|
references to find which specific references have the desired object. |
|
Writers should use this format when the `position_delta` list would have |
|
overflowed the file's block size due to a high number of references |
|
pointing to the same object. |
|
|
|
The first `position_delta` is the position from the start of the file. |
|
Additional `position_delta` entries are sorted ascending and relative to |
|
the prior entry, e.g. a reader would perform: |
|
|
|
.... |
|
pos = position_delta[0] |
|
prior = pos |
|
for (j = 1; j < position_count; j++) { |
|
pos = prior + position_delta[j] |
|
prior = pos |
|
} |
|
.... |
|
|
|
With a position in hand, a reader must linearly scan the ref block, |
|
starting from the first `ref_record`, testing each reference's object names |
|
(for `value_type = 0x1` or `0x2`) for full equality. Faster searching by |
|
object name within a single ref block is not supported by the reftable format. |
|
Smaller block sizes reduce the number of candidates this step must |
|
consider. |
|
|
|
Obj index |
|
^^^^^^^^^ |
|
|
|
The obj index stores the abbreviation from the last entry for every obj |
|
block in the file, enabling reduced disk seeks for all lookups. It is |
|
formatted exactly the same as the ref index, but refers to obj blocks. |
|
|
|
The obj index should be present if obj blocks are present, as obj blocks |
|
should only be written in larger files. |
|
|
|
Readers loading the obj index must first read the footer (below) to |
|
obtain `obj_index_position`. If not present, the position will be 0. |
|
|
|
Log block format |
|
^^^^^^^^^^^^^^^^ |
|
|
|
Unlike ref and obj blocks, log blocks are always unaligned. |
|
|
|
Log blocks are variable in size, and do not match the `block_size` |
|
specified in the file header or footer. Writers should choose an |
|
appropriate buffer size to prepare a log block for deflation, such as |
|
`2 * block_size`. |
|
|
|
A log block is written as: |
|
|
|
.... |
|
'g' |
|
uint24( block_len ) |
|
zlib_deflate { |
|
log_record+ |
|
uint24( restart_offset )+ |
|
uint16( restart_count ) |
|
} |
|
.... |
|
|
|
Log blocks look similar to ref blocks, except `block_type = 'g'`. |
|
|
|
The 4-byte block header is followed by the deflated block contents using |
|
zlib deflate. The `block_len` in the header is the inflated size |
|
(including 4-byte block header), and should be used by readers to |
|
preallocate the inflation output buffer. A log block's `block_len` may |
|
exceed the file's block size. |
|
|
|
Offsets within the log block (e.g. `restart_offset`) still include the |
|
4-byte header. Readers may prefer prefixing the inflation output buffer |
|
with the 4-byte header. |
|
|
|
Within the deflate container, a variable number of `log_record` describe |
|
reference changes. The log record format is described below. See ref |
|
block format (above) for a description of `restart_offset` and |
|
`restart_count`. |
|
|
|
Because log blocks have no alignment or padding between blocks, readers |
|
must keep track of the bytes consumed by the inflater to know where the |
|
next log block begins. |
|
|
|
log record |
|
++++++++++ |
|
|
|
Log record keys are structured as: |
|
|
|
.... |
|
ref_name '\0' reverse_int64( update_index ) |
|
.... |
|
|
|
where `update_index` is the unique transaction identifier. The |
|
`update_index` field must be unique within the scope of a `ref_name`. |
|
See the update transactions section below for further details. |
|
|
|
The `reverse_int64` function inverses the value so lexicographical |
|
ordering the network byte order encoding sorts the more recent records |
|
with higher `update_index` values first: |
|
|
|
.... |
|
reverse_int64(int64 t) { |
|
return 0xffffffffffffffff - t; |
|
} |
|
.... |
|
|
|
Log records have a similar starting structure to ref and index records, |
|
utilizing the same prefix compression scheme applied to the log record |
|
key described above. |
|
|
|
.... |
|
varint( prefix_length ) |
|
varint( (suffix_length << 3) | log_type ) |
|
suffix |
|
log_data { |
|
old_id |
|
new_id |
|
varint( name_length ) name |
|
varint( email_length ) email |
|
varint( time_seconds ) |
|
sint16( tz_offset ) |
|
varint( message_length ) message |
|
}? |
|
.... |
|
|
|
Log record entries use `log_type` to indicate what follows: |
|
|
|
* `0x0`: deletion; no log data. |
|
* `0x1`: standard git reflog data using `log_data` above. |
|
|
|
The `log_type = 0x0` is mostly useful for `git stash drop`, removing an |
|
entry from the reflog of `refs/stash` in a transaction file (below), |
|
without needing to rewrite larger files. Readers reading a stack of |
|
reflogs must treat this as a deletion. |
|
|
|
For `log_type = 0x1`, the `log_data` section follows |
|
linkgit:git-update-ref[1] logging and includes: |
|
|
|
* two object names (old id, new id) |
|
* varint string of committer's name |
|
* varint string of committer's email |
|
* varint time in seconds since epoch (Jan 1, 1970) |
|
* 2-byte timezone offset in minutes (signed) |
|
* varint string of message |
|
|
|
`tz_offset` is the absolute number of minutes from GMT the committer was |
|
at the time of the update. For example `GMT-0800` is encoded in reftable |
|
as `sint16(-480)` and `GMT+0230` is `sint16(150)`. |
|
|
|
The committer email does not contain `<` or `>`, it's the value normally |
|
found between the `<>` in a git commit object header. |
|
|
|
The `message_length` may be 0, in which case there was no message |
|
supplied for the update. |
|
|
|
Contrary to traditional reflog (which is a file), renames are encoded as |
|
a combination of ref deletion and ref creation. A deletion is a log |
|
record with a zero new_id, and a creation is a log record with a zero old_id. |
|
|
|
Reading the log |
|
+++++++++++++++ |
|
|
|
Readers accessing the log must first read the footer (below) to |
|
determine the `log_position`. The first block of the log begins at |
|
`log_position` bytes since the start of the file. The `log_position` is |
|
not block aligned. |
|
|
|
Importing logs |
|
++++++++++++++ |
|
|
|
When importing from `$GIT_DIR/logs` writers should globally order all |
|
log records roughly by timestamp while preserving file order, and assign |
|
unique, increasing `update_index` values for each log line. Newer log |
|
records get higher `update_index` values. |
|
|
|
Although an import may write only a single reftable file, the reftable |
|
file must span many unique `update_index`, as each log line requires its |
|
own `update_index` to preserve semantics. |
|
|
|
Log index |
|
^^^^^^^^^ |
|
|
|
The log index stores the log key |
|
(`refname \0 reverse_int64(update_index)`) for the last log record of |
|
every log block in the file, supporting bounded-time lookup. |
|
|
|
A log index block must be written if 2 or more log blocks are written to |
|
the file. If present, the log index appears after the last log block. |
|
There is no padding used to align the log index to block alignment. |
|
|
|
Log index format is identical to ref index, except the keys are 9 bytes |
|
longer to include `'\0'` and the 8-byte `reverse_int64(update_index)`. |
|
Records use `block_position` to refer to the start of a log block. |
|
|
|
Reading the index |
|
+++++++++++++++++ |
|
|
|
Readers loading the log index must first read the footer (below) to |
|
obtain `log_index_position`. If not present, the position will be 0. |
|
|
|
Footer |
|
^^^^^^ |
|
|
|
After the last block of the file, a file footer is written. It begins |
|
like the file header, but is extended with additional data. |
|
|
|
.... |
|
HEADER |
|
|
|
uint64( ref_index_position ) |
|
uint64( (obj_position << 5) | obj_id_len ) |
|
uint64( obj_index_position ) |
|
|
|
uint64( log_position ) |
|
uint64( log_index_position ) |
|
|
|
uint32( CRC-32 of above ) |
|
.... |
|
|
|
If a section is missing (e.g. ref index) the corresponding position |
|
field (e.g. `ref_index_position`) will be 0. |
|
|
|
* `obj_position`: byte position for the first obj block. |
|
* `obj_id_len`: number of bytes used to abbreviate object names in |
|
obj blocks. |
|
* `log_position`: byte position for the first log block. |
|
* `ref_index_position`: byte position for the start of the ref index. |
|
* `obj_index_position`: byte position for the start of the obj index. |
|
* `log_index_position`: byte position for the start of the log index. |
|
|
|
The size of the footer is 68 bytes for version 1, and 72 bytes for |
|
version 2. |
|
|
|
Reading the footer |
|
++++++++++++++++++ |
|
|
|
Readers must first read the file start to determine the version |
|
number. Then they seek to `file_length - FOOTER_LENGTH` to access the |
|
footer. A trusted external source (such as `stat(2)`) is necessary to |
|
obtain `file_length`. When reading the footer, readers must verify: |
|
|
|
* 4-byte magic is correct |
|
* 1-byte version number is recognized |
|
* 4-byte CRC-32 matches the other 64 bytes (including magic, and |
|
version) |
|
|
|
Once verified, the other fields of the footer can be accessed. |
|
|
|
Empty tables |
|
++++++++++++ |
|
|
|
A reftable may be empty. In this case, the file starts with a header |
|
and is immediately followed by a footer. |
|
|
|
Binary search |
|
^^^^^^^^^^^^^ |
|
|
|
Binary search within a block is supported by the `restart_offset` fields |
|
at the end of the block. Readers can binary search through the restart |
|
table to locate between which two restart points the sought reference or |
|
key should appear. |
|
|
|
Each record identified by a `restart_offset` stores the complete key in |
|
the `suffix` field of the record, making the compare operation during |
|
binary search straightforward. |
|
|
|
Once a restart point lexicographically before the sought reference has |
|
been identified, readers can linearly scan through the following record |
|
entries to locate the sought record, terminating if the current record |
|
sorts after (and therefore the sought key is not present). |
|
|
|
Restart point selection |
|
+++++++++++++++++++++++ |
|
|
|
Writers determine the restart points at file creation. The process is |
|
arbitrary, but every 16 or 64 records is recommended. Every 16 may be |
|
more suitable for smaller block sizes (4k or 8k), every 64 for larger |
|
block sizes (64k). |
|
|
|
More frequent restart points reduces prefix compression and increases |
|
space consumed by the restart table, both of which increase file size. |
|
|
|
Less frequent restart points makes prefix compression more effective, |
|
decreasing overall file size, with increased penalties for readers |
|
walking through more records after the binary search step. |
|
|
|
A maximum of `65535` restart points per block is supported. |
|
|
|
Considerations |
|
~~~~~~~~~~~~~~ |
|
|
|
Lightweight refs dominate |
|
^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
The reftable format assumes the vast majority of references are single |
|
object names valued with common prefixes, such as Gerrit Code Review's |
|
`refs/changes/` namespace, GitHub's `refs/pulls/` namespace, or many |
|
lightweight tags in the `refs/tags/` namespace. |
|
|
|
Annotated tags storing the peeled object cost an additional object name per |
|
reference. |
|
|
|
Low overhead |
|
^^^^^^^^^^^^ |
|
|
|
A reftable with very few references (e.g. git.git with 5 heads) is 269 |
|
bytes for reftable, vs. 332 bytes for packed-refs. This supports |
|
reftable scaling down for transaction logs (below). |
|
|
|
Block size |
|
^^^^^^^^^^ |
|
|
|
For a Gerrit Code Review type repository with many change refs, larger |
|
block sizes (64 KiB) and less frequent restart points (every 64) yield |
|
better compression due to more references within the block compressing |
|
against the prior reference. |
|
|
|
Larger block sizes reduce the index size, as the reftable will require |
|
fewer blocks to store the same number of references. |
|
|
|
Minimal disk seeks |
|
^^^^^^^^^^^^^^^^^^ |
|
|
|
Assuming the index block has been loaded into memory, binary searching |
|
for any single reference requires exactly 1 disk seek to load the |
|
containing block. |
|
|
|
Scans and lookups dominate |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Scanning all references and lookup by name (or namespace such as |
|
`refs/heads/`) are the most common activities performed on repositories. |
|
Object names are stored directly with references to optimize this use case. |
|
|
|
Logs are infrequently read |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Logs are infrequently accessed, but can be large. Deflating log blocks |
|
saves disk space, with some increased penalty at read time. |
|
|
|
Logs are stored in an isolated section from refs, reducing the burden on |
|
reference readers that want to ignore logs. Further, historical logs can |
|
be isolated into log-only files. |
|
|
|
Logs are read backwards |
|
^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Logs are frequently accessed backwards (most recent N records for master |
|
to answer `master@{4}`), so log records are grouped by reference, and |
|
sorted descending by update index. |
|
|
|
Repository format |
|
~~~~~~~~~~~~~~~~~ |
|
|
|
Version 1 |
|
^^^^^^^^^ |
|
|
|
A repository must set its `$GIT_DIR/config` to configure reftable: |
|
|
|
.... |
|
[core] |
|
repositoryformatversion = 1 |
|
[extensions] |
|
refStorage = reftable |
|
.... |
|
|
|
Layout |
|
^^^^^^ |
|
|
|
A collection of reftable files are stored in the `$GIT_DIR/reftable/` directory. |
|
Their names should have a random element, such that each filename is globally |
|
unique; this helps avoid spurious failures on Windows, where open files cannot |
|
be removed or overwritten. It suggested to use |
|
`${min_update_index}-${max_update_index}-${random}.ref` as a naming convention. |
|
|
|
Log-only files use the `.log` extension, while ref-only and mixed ref |
|
and log files use `.ref`. extension. |
|
|
|
The stack ordering file is `$GIT_DIR/reftable/tables.list` and lists the |
|
current files, one per line, in order, from oldest (base) to newest |
|
(most recent): |
|
|
|
.... |
|
$ cat .git/reftable/tables.list |
|
00000001-00000001-RANDOM1.log |
|
00000002-00000002-RANDOM2.ref |
|
00000003-00000003-RANDOM3.ref |
|
.... |
|
|
|
Readers must read `$GIT_DIR/reftable/tables.list` to determine which |
|
files are relevant right now, and search through the stack in reverse |
|
order (last reftable is examined first). |
|
|
|
Reftable files not listed in `tables.list` may be new (and about to be |
|
added to the stack by the active writer), or ancient and ready to be |
|
pruned. |
|
|
|
Backward compatibility |
|
^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Older clients should continue to recognize the directory as a git |
|
repository so they don't look for an enclosing repository in parent |
|
directories. To this end, a reftable-enabled repository must contain the |
|
following dummy files |
|
|
|
* `.git/HEAD`, a regular file containing `ref: refs/heads/.invalid`. |
|
* `.git/refs/`, a directory |
|
* `.git/refs/heads`, a regular file |
|
|
|
Readers |
|
^^^^^^^ |
|
|
|
Readers can obtain a consistent snapshot of the reference space by |
|
following: |
|
|
|
1. Open and read the `tables.list` file. |
|
2. Open each of the reftable files that it mentions. |
|
3. If any of the files is missing, goto 1. |
|
4. Read from the now-open files as long as necessary. |
|
|
|
Update transactions |
|
^^^^^^^^^^^^^^^^^^^ |
|
|
|
Although reftables are immutable, mutations are supported by writing a |
|
new reftable and atomically appending it to the stack: |
|
|
|
1. Acquire `tables.list.lock`. |
|
2. Read `tables.list` to determine current reftables. |
|
3. Select `update_index` to be most recent file's |
|
`max_update_index + 1`. |
|
4. Prepare temp reftable `tmp_XXXXXX`, including log entries. |
|
5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}-${random}.ref`. |
|
6. Copy `tables.list` to `tables.list.lock`, appending file from (5). |
|
7. Rename `tables.list.lock` to `tables.list`. |
|
|
|
During step 4 the new file's `min_update_index` and `max_update_index` |
|
are both set to the `update_index` selected by step 3. All log records |
|
for the transaction use the same `update_index` in their keys. This |
|
enables later correlation of which references were updated by the same |
|
transaction. |
|
|
|
Because a single `tables.list.lock` file is used to manage locking, the |
|
repository is single-threaded for writers. Writers may have to busy-spin |
|
(with backoff) around creating `tables.list.lock`, for up to an |
|
acceptable wait period, aborting if the repository is too busy to |
|
mutate. Application servers wrapped around repositories (e.g. Gerrit |
|
Code Review) can layer their own lock/wait queue to improve fairness to |
|
writers. |
|
|
|
Reference deletions |
|
^^^^^^^^^^^^^^^^^^^ |
|
|
|
Deletion of any reference can be explicitly stored by setting the `type` |
|
to `0x0` and omitting the `value` field of the `ref_record`. This serves |
|
as a tombstone, overriding any assertions about the existence of the |
|
reference from earlier files in the stack. |
|
|
|
Compaction |
|
^^^^^^^^^^ |
|
|
|
A partial stack of reftables can be compacted by merging references |
|
using a straightforward merge join across reftables, selecting the most |
|
recent value for output, and omitting deleted references that do not |
|
appear in remaining, lower reftables. |
|
|
|
A compacted reftable should set its `min_update_index` to the smallest |
|
of the input files' `min_update_index`, and its `max_update_index` |
|
likewise to the largest input `max_update_index`. |
|
|
|
For sake of illustration, assume the stack currently consists of |
|
reftable files (from oldest to newest): A, B, C, and D. The compactor is |
|
going to compact B and C, leaving A and D alone. |
|
|
|
1. Obtain lock `tables.list.lock` and read the `tables.list` file. |
|
2. Obtain locks `B.lock` and `C.lock`. Ownership of these locks |
|
prevents other processes from trying to compact these files. |
|
3. Release `tables.list.lock`. |
|
4. Compact `B` and `C` into a temp file |
|
`${min_update_index}-${max_update_index}_XXXXXX`. |
|
5. Reacquire lock `tables.list.lock`. |
|
6. Verify that `B` and `C` are still in the stack, in that order. This |
|
should always be the case, assuming that other processes are adhering to |
|
the locking protocol. |
|
7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to |
|
`${min_update_index}-${max_update_index}-${random}.ref`. |
|
8. Write the new stack to `tables.list.lock`, replacing `B` and `C` |
|
with the file from (4). |
|
9. Rename `tables.list.lock` to `tables.list`. |
|
10. Delete `B` and `C`, perhaps after a short sleep to avoid forcing |
|
readers to backtrack. |
|
|
|
This strategy permits compactions to proceed independently of updates. |
|
|
|
Each reftable (compacted or not) is uniquely identified by its name, so |
|
open reftables can be cached by their name. |
|
|
|
Windows |
|
^^^^^^^ |
|
|
|
On windows, and other systems that do not allow deleting or renaming to open |
|
files, compaction may succeed, but other readers may prevent obsolete tables |
|
from being deleted. |
|
|
|
On these platforms, the following strategy can be followed: on closing a |
|
reftable stack, reload `tables.list`, and delete any tables no longer mentioned |
|
in `tables.list`. |
|
|
|
Irregular program exit may still leave about unused files. In this case, a |
|
cleanup operation can read `tables.list`, note its modification timestamp, and |
|
delete any unreferenced `*.ref` files that are older. |
|
|
|
|
|
Alternatives considered |
|
~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
bzip packed-refs |
|
^^^^^^^^^^^^^^^^ |
|
|
|
`bzip2` can significantly shrink a large packed-refs file (e.g. 62 MiB |
|
compresses to 23 MiB, 37%). However the bzip format does not support |
|
random access to a single reference. Readers must inflate and discard |
|
while performing a linear scan. |
|
|
|
Breaking packed-refs into chunks (individually compressing each chunk) |
|
would reduce the amount of data a reader must inflate, but still leaves |
|
the problem of indexing chunks to support readers efficiently locating |
|
the correct chunk. |
|
|
|
Given the compression achieved by reftable's encoding, it does not seem |
|
necessary to add the complexity of bzip/gzip/zlib. |
|
|
|
Michael Haggerty's alternate format |
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
|
|
|
Michael Haggerty proposed |
|
link:https://lore.kernel.org/git/CAMy9T_HCnyc1g8XWOOWhe7nN0aEFyyBskV2aOMb_fe%2BwGvEJ7A%40mail.gmail.com/[an |
|
alternate] format to reftable on the Git mailing list. This format uses |
|
smaller chunks, without the restart table, and avoids block alignment |
|
with padding. Reflog entries immediately follow each ref, and are thus |
|
interleaved between refs. |
|
|
|
Performance testing indicates reftable is faster for lookups (51% |
|
faster, 11.2 usec vs. 5.4 usec), although reftable produces a slightly |
|
larger file (+ ~3.2%, 28.3M vs 29.2M): |
|
|
|
[cols=">,>,>,>",options="header",] |
|
|===================================== |
|
|format |size |seek cold |seek hot |
|
|mh-alt |28.3 M |23.4 usec |11.2 usec |
|
|reftable |29.2 M |19.9 usec |5.4 usec |
|
|===================================== |
|
|
|
JGit Ketch RefTree |
|
^^^^^^^^^^^^^^^^^^ |
|
|
|
https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03073.html[JGit Ketch] |
|
proposed |
|
link:https://lore.kernel.org/git/CAJo%3DhJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg%40mail.gmail.com/[RefTree], |
|
an encoding of references inside Git tree objects stored as part of the |
|
repository's object database. |
|
|
|
The RefTree format adds additional load on the object database storage |
|
layer (more loose objects, more objects in packs), and relies heavily on |
|
the packer's delta compression to save space. Namespaces which are flat |
|
(e.g. thousands of tags in refs/tags) initially create very large loose |
|
objects, and so RefTree does not address the problem of copying many |
|
references to modify a handful. |
|
|
|
Flat namespaces are not efficiently searchable in RefTree, as tree |
|
objects in canonical formatting cannot be binary searched. This fails |
|
the need to handle a large number of references in a single namespace, |
|
such as GitHub's `refs/pulls`, or a project with many tags. |
|
|
|
LMDB |
|
^^^^ |
|
|
|
David Turner proposed |
|
https://lore.kernel.org/git/1455772670-21142-26-git-send-email-dturner@twopensource.com/[using |
|
LMDB], as LMDB is lightweight (64k of runtime code) and GPL-compatible |
|
license. |
|
|
|
A downside of LMDB is its reliance on a single C implementation. This |
|
makes embedding inside JGit (a popular reimplementation of Git) |
|
difficult, and hoisting onto virtual storage (for JGit DFS) virtually |
|
impossible. |
|
|
|
A common format that can be supported by all major Git implementations |
|
(git-core, JGit, libgit2) is strongly preferred.
|
|
|