Tree:
f6526728f9
main
maint
master
next
seen
todo
gitgui-0.10.0
gitgui-0.10.1
gitgui-0.10.2
gitgui-0.11.0
gitgui-0.12.0
gitgui-0.13.0
gitgui-0.14.0
gitgui-0.15.0
gitgui-0.16.0
gitgui-0.17.0
gitgui-0.18.0
gitgui-0.19.0
gitgui-0.20.0
gitgui-0.21.0
gitgui-0.6.0
gitgui-0.6.1
gitgui-0.6.2
gitgui-0.6.3
gitgui-0.6.4
gitgui-0.6.5
gitgui-0.7.0
gitgui-0.7.0-rc1
gitgui-0.7.1
gitgui-0.7.2
gitgui-0.7.3
gitgui-0.7.4
gitgui-0.7.5
gitgui-0.8.0
gitgui-0.8.1
gitgui-0.8.2
gitgui-0.8.3
gitgui-0.8.4
gitgui-0.9.0
gitgui-0.9.1
gitgui-0.9.2
gitgui-0.9.3
junio-gpg-pub
v0.99
v0.99.1
v0.99.2
v0.99.3
v0.99.4
v0.99.5
v0.99.6
v0.99.7
v0.99.7a
v0.99.7b
v0.99.7c
v0.99.7d
v0.99.8
v0.99.8a
v0.99.8b
v0.99.8c
v0.99.8d
v0.99.8e
v0.99.8f
v0.99.8g
v0.99.9
v0.99.9a
v0.99.9b
v0.99.9c
v0.99.9d
v0.99.9e
v0.99.9f
v0.99.9g
v0.99.9h
v0.99.9i
v0.99.9j
v0.99.9k
v0.99.9l
v0.99.9m
v0.99.9n
v1.0.0
v1.0.0a
v1.0.0b
v1.0.1
v1.0.10
v1.0.11
v1.0.12
v1.0.13
v1.0.2
v1.0.3
v1.0.4
v1.0.5
v1.0.6
v1.0.7
v1.0.8
v1.0.9
v1.0rc1
v1.0rc2
v1.0rc3
v1.0rc4
v1.0rc5
v1.0rc6
v1.1.0
v1.1.1
v1.1.2
v1.1.3
v1.1.4
v1.1.5
v1.1.6
v1.2.0
v1.2.1
v1.2.2
v1.2.3
v1.2.4
v1.2.5
v1.2.6
v1.3.0
v1.3.0-rc1
v1.3.0-rc2
v1.3.0-rc3
v1.3.0-rc4
v1.3.1
v1.3.2
v1.3.3
v1.4.0
v1.4.0-rc1
v1.4.0-rc2
v1.4.1
v1.4.1-rc1
v1.4.1-rc2
v1.4.1.1
v1.4.2
v1.4.2-rc1
v1.4.2-rc2
v1.4.2-rc3
v1.4.2-rc4
v1.4.2.1
v1.4.2.2
v1.4.2.3
v1.4.2.4
v1.4.3
v1.4.3-rc1
v1.4.3-rc2
v1.4.3-rc3
v1.4.3.1
v1.4.3.2
v1.4.3.3
v1.4.3.4
v1.4.3.5
v1.4.4
v1.4.4-rc1
v1.4.4-rc2
v1.4.4.1
v1.4.4.2
v1.4.4.3
v1.4.4.4
v1.4.4.5
v1.5.0
v1.5.0-rc0
v1.5.0-rc1
v1.5.0-rc2
v1.5.0-rc3
v1.5.0-rc4
v1.5.0.1
v1.5.0.2
v1.5.0.3
v1.5.0.4
v1.5.0.5
v1.5.0.6
v1.5.0.7
v1.5.1
v1.5.1-rc1
v1.5.1-rc2
v1.5.1-rc3
v1.5.1.1
v1.5.1.2
v1.5.1.3
v1.5.1.4
v1.5.1.5
v1.5.1.6
v1.5.2
v1.5.2-rc0
v1.5.2-rc1
v1.5.2-rc2
v1.5.2-rc3
v1.5.2.1
v1.5.2.2
v1.5.2.3
v1.5.2.4
v1.5.2.5
v1.5.3
v1.5.3-rc0
v1.5.3-rc1
v1.5.3-rc2
v1.5.3-rc3
v1.5.3-rc4
v1.5.3-rc5
v1.5.3-rc6
v1.5.3-rc7
v1.5.3.1
v1.5.3.2
v1.5.3.3
v1.5.3.4
v1.5.3.5
v1.5.3.6
v1.5.3.7
v1.5.3.8
v1.5.4
v1.5.4-rc0
v1.5.4-rc1
v1.5.4-rc2
v1.5.4-rc3
v1.5.4-rc4
v1.5.4-rc5
v1.5.4.1
v1.5.4.2
v1.5.4.3
v1.5.4.4
v1.5.4.5
v1.5.4.6
v1.5.4.7
v1.5.5
v1.5.5-rc0
v1.5.5-rc1
v1.5.5-rc2
v1.5.5-rc3
v1.5.5.1
v1.5.5.2
v1.5.5.3
v1.5.5.4
v1.5.5.5
v1.5.5.6
v1.5.6
v1.5.6-rc0
v1.5.6-rc1
v1.5.6-rc2
v1.5.6-rc3
v1.5.6.1
v1.5.6.2
v1.5.6.3
v1.5.6.4
v1.5.6.5
v1.5.6.6
v1.6.0
v1.6.0-rc0
v1.6.0-rc1
v1.6.0-rc2
v1.6.0-rc3
v1.6.0.1
v1.6.0.2
v1.6.0.3
v1.6.0.4
v1.6.0.5
v1.6.0.6
v1.6.1
v1.6.1-rc1
v1.6.1-rc2
v1.6.1-rc3
v1.6.1-rc4
v1.6.1.1
v1.6.1.2
v1.6.1.3
v1.6.1.4
v1.6.2
v1.6.2-rc0
v1.6.2-rc1
v1.6.2-rc2
v1.6.2.1
v1.6.2.2
v1.6.2.3
v1.6.2.4
v1.6.2.5
v1.6.3
v1.6.3-rc0
v1.6.3-rc1
v1.6.3-rc2
v1.6.3-rc3
v1.6.3-rc4
v1.6.3.1
v1.6.3.2
v1.6.3.3
v1.6.3.4
v1.6.4
v1.6.4-rc0
v1.6.4-rc1
v1.6.4-rc2
v1.6.4-rc3
v1.6.4.1
v1.6.4.2
v1.6.4.3
v1.6.4.4
v1.6.4.5
v1.6.5
v1.6.5-rc0
v1.6.5-rc1
v1.6.5-rc2
v1.6.5-rc3
v1.6.5.1
v1.6.5.2
v1.6.5.3
v1.6.5.4
v1.6.5.5
v1.6.5.6
v1.6.5.7
v1.6.5.8
v1.6.5.9
v1.6.6
v1.6.6-rc0
v1.6.6-rc1
v1.6.6-rc2
v1.6.6-rc3
v1.6.6-rc4
v1.6.6.1
v1.6.6.2
v1.6.6.3
v1.7.0
v1.7.0-rc0
v1.7.0-rc1
v1.7.0-rc2
v1.7.0.1
v1.7.0.2
v1.7.0.3
v1.7.0.4
v1.7.0.5
v1.7.0.6
v1.7.0.7
v1.7.0.8
v1.7.0.9
v1.7.1
v1.7.1-rc0
v1.7.1-rc1
v1.7.1-rc2
v1.7.1.1
v1.7.1.2
v1.7.1.3
v1.7.1.4
v1.7.10
v1.7.10-rc0
v1.7.10-rc1
v1.7.10-rc2
v1.7.10-rc3
v1.7.10-rc4
v1.7.10.1
v1.7.10.2
v1.7.10.3
v1.7.10.4
v1.7.10.5
v1.7.11
v1.7.11-rc0
v1.7.11-rc1
v1.7.11-rc2
v1.7.11-rc3
v1.7.11.1
v1.7.11.2
v1.7.11.3
v1.7.11.4
v1.7.11.5
v1.7.11.6
v1.7.11.7
v1.7.12
v1.7.12-rc0
v1.7.12-rc1
v1.7.12-rc2
v1.7.12-rc3
v1.7.12.1
v1.7.12.2
v1.7.12.3
v1.7.12.4
v1.7.2
v1.7.2-rc0
v1.7.2-rc1
v1.7.2-rc2
v1.7.2-rc3
v1.7.2.1
v1.7.2.2
v1.7.2.3
v1.7.2.4
v1.7.2.5
v1.7.3
v1.7.3-rc0
v1.7.3-rc1
v1.7.3-rc2
v1.7.3.1
v1.7.3.2
v1.7.3.3
v1.7.3.4
v1.7.3.5
v1.7.4
v1.7.4-rc0
v1.7.4-rc1
v1.7.4-rc2
v1.7.4-rc3
v1.7.4.1
v1.7.4.2
v1.7.4.3
v1.7.4.4
v1.7.4.5
v1.7.5
v1.7.5-rc0
v1.7.5-rc1
v1.7.5-rc2
v1.7.5-rc3
v1.7.5.1
v1.7.5.2
v1.7.5.3
v1.7.5.4
v1.7.6
v1.7.6-rc0
v1.7.6-rc1
v1.7.6-rc2
v1.7.6-rc3
v1.7.6.1
v1.7.6.2
v1.7.6.3
v1.7.6.4
v1.7.6.5
v1.7.6.6
v1.7.7
v1.7.7-rc0
v1.7.7-rc1
v1.7.7-rc2
v1.7.7-rc3
v1.7.7.1
v1.7.7.2
v1.7.7.3
v1.7.7.4
v1.7.7.5
v1.7.7.6
v1.7.7.7
v1.7.8
v1.7.8-rc0
v1.7.8-rc1
v1.7.8-rc2
v1.7.8-rc3
v1.7.8-rc4
v1.7.8.1
v1.7.8.2
v1.7.8.3
v1.7.8.4
v1.7.8.5
v1.7.8.6
v1.7.9
v1.7.9-rc0
v1.7.9-rc1
v1.7.9-rc2
v1.7.9.1
v1.7.9.2
v1.7.9.3
v1.7.9.4
v1.7.9.5
v1.7.9.6
v1.7.9.7
v1.8.0
v1.8.0-rc0
v1.8.0-rc1
v1.8.0-rc2
v1.8.0-rc3
v1.8.0.1
v1.8.0.2
v1.8.0.3
v1.8.1
v1.8.1-rc0
v1.8.1-rc1
v1.8.1-rc2
v1.8.1-rc3
v1.8.1.1
v1.8.1.2
v1.8.1.3
v1.8.1.4
v1.8.1.5
v1.8.1.6
v1.8.2
v1.8.2-rc0
v1.8.2-rc1
v1.8.2-rc2
v1.8.2-rc3
v1.8.2.1
v1.8.2.2
v1.8.2.3
v1.8.3
v1.8.3-rc0
v1.8.3-rc1
v1.8.3-rc2
v1.8.3-rc3
v1.8.3.1
v1.8.3.2
v1.8.3.3
v1.8.3.4
v1.8.4
v1.8.4-rc0
v1.8.4-rc1
v1.8.4-rc2
v1.8.4-rc3
v1.8.4-rc4
v1.8.4.1
v1.8.4.2
v1.8.4.3
v1.8.4.4
v1.8.4.5
v1.8.5
v1.8.5-rc0
v1.8.5-rc1
v1.8.5-rc2
v1.8.5-rc3
v1.8.5.1
v1.8.5.2
v1.8.5.3
v1.8.5.4
v1.8.5.5
v1.8.5.6
v1.9-rc0
v1.9-rc1
v1.9-rc2
v1.9.0
v1.9.0-rc3
v1.9.1
v1.9.2
v1.9.3
v1.9.4
v1.9.5
v2.0.0
v2.0.0-rc0
v2.0.0-rc1
v2.0.0-rc2
v2.0.0-rc3
v2.0.0-rc4
v2.0.1
v2.0.2
v2.0.3
v2.0.4
v2.0.5
v2.1.0
v2.1.0-rc0
v2.1.0-rc1
v2.1.0-rc2
v2.1.1
v2.1.2
v2.1.3
v2.1.4
v2.10.0
v2.10.0-rc0
v2.10.0-rc1
v2.10.0-rc2
v2.10.1
v2.10.2
v2.10.3
v2.10.4
v2.10.5
v2.11.0
v2.11.0-rc0
v2.11.0-rc1
v2.11.0-rc2
v2.11.0-rc3
v2.11.1
v2.11.2
v2.11.3
v2.11.4
v2.12.0
v2.12.0-rc0
v2.12.0-rc1
v2.12.0-rc2
v2.12.1
v2.12.2
v2.12.3
v2.12.4
v2.12.5
v2.13.0
v2.13.0-rc0
v2.13.0-rc1
v2.13.0-rc2
v2.13.1
v2.13.2
v2.13.3
v2.13.4
v2.13.5
v2.13.6
v2.13.7
v2.14.0
v2.14.0-rc0
v2.14.0-rc1
v2.14.1
v2.14.2
v2.14.3
v2.14.4
v2.14.5
v2.14.6
v2.15.0
v2.15.0-rc0
v2.15.0-rc1
v2.15.0-rc2
v2.15.1
v2.15.2
v2.15.3
v2.15.4
v2.16.0
v2.16.0-rc0
v2.16.0-rc1
v2.16.0-rc2
v2.16.1
v2.16.2
v2.16.3
v2.16.4
v2.16.5
v2.16.6
v2.17.0
v2.17.0-rc0
v2.17.0-rc1
v2.17.0-rc2
v2.17.1
v2.17.2
v2.17.3
v2.17.4
v2.17.5
v2.17.6
v2.18.0
v2.18.0-rc0
v2.18.0-rc1
v2.18.0-rc2
v2.18.1
v2.18.2
v2.18.3
v2.18.4
v2.18.5
v2.19.0
v2.19.0-rc0
v2.19.0-rc1
v2.19.0-rc2
v2.19.1
v2.19.2
v2.19.3
v2.19.4
v2.19.5
v2.19.6
v2.2.0
v2.2.0-rc0
v2.2.0-rc1
v2.2.0-rc2
v2.2.0-rc3
v2.2.1
v2.2.2
v2.2.3
v2.20.0
v2.20.0-rc0
v2.20.0-rc1
v2.20.0-rc2
v2.20.1
v2.20.2
v2.20.3
v2.20.4
v2.20.5
v2.21.0
v2.21.0-rc0
v2.21.0-rc1
v2.21.0-rc2
v2.21.1
v2.21.2
v2.21.3
v2.21.4
v2.22.0
v2.22.0-rc0
v2.22.0-rc1
v2.22.0-rc2
v2.22.0-rc3
v2.22.1
v2.22.2
v2.22.3
v2.22.4
v2.22.5
v2.23.0
v2.23.0-rc0
v2.23.0-rc1
v2.23.0-rc2
v2.23.1
v2.23.2
v2.23.3
v2.23.4
v2.24.0
v2.24.0-rc0
v2.24.0-rc1
v2.24.0-rc2
v2.24.1
v2.24.2
v2.24.3
v2.24.4
v2.25.0
v2.25.0-rc0
v2.25.0-rc1
v2.25.0-rc2
v2.25.1
v2.25.2
v2.25.3
v2.25.4
v2.25.5
v2.26.0
v2.26.0-rc0
v2.26.0-rc1
v2.26.0-rc2
v2.26.1
v2.26.2
v2.26.3
v2.27.0
v2.27.0-rc0
v2.27.0-rc1
v2.27.0-rc2
v2.27.1
v2.28.0
v2.28.0-rc0
v2.28.0-rc1
v2.28.0-rc2
v2.28.1
v2.29.0
v2.29.0-rc0
v2.29.0-rc1
v2.29.0-rc2
v2.29.1
v2.29.2
v2.29.3
v2.3.0
v2.3.0-rc0
v2.3.0-rc1
v2.3.0-rc2
v2.3.1
v2.3.10
v2.3.2
v2.3.3
v2.3.4
v2.3.5
v2.3.6
v2.3.7
v2.3.8
v2.3.9
v2.30.0
v2.30.0-rc0
v2.30.0-rc1
v2.30.0-rc2
v2.30.1
v2.30.2
v2.30.3
v2.30.4
v2.30.5
v2.30.6
v2.30.7
v2.30.8
v2.30.9
v2.31.0
v2.31.0-rc0
v2.31.0-rc1
v2.31.0-rc2
v2.31.1
v2.31.2
v2.31.3
v2.31.4
v2.31.5
v2.31.6
v2.31.7
v2.31.8
v2.32.0
v2.32.0-rc0
v2.32.0-rc1
v2.32.0-rc2
v2.32.0-rc3
v2.32.1
v2.32.2
v2.32.3
v2.32.4
v2.32.5
v2.32.6
v2.32.7
v2.33.0
v2.33.0-rc0
v2.33.0-rc1
v2.33.0-rc2
v2.33.1
v2.33.2
v2.33.3
v2.33.4
v2.33.5
v2.33.6
v2.33.7
v2.33.8
v2.34.0
v2.34.0-rc0
v2.34.0-rc1
v2.34.0-rc2
v2.34.1
v2.34.2
v2.34.3
v2.34.4
v2.34.5
v2.34.6
v2.34.7
v2.34.8
v2.35.0
v2.35.0-rc0
v2.35.0-rc1
v2.35.0-rc2
v2.35.1
v2.35.2
v2.35.3
v2.35.4
v2.35.5
v2.35.6
v2.35.7
v2.35.8
v2.36.0
v2.36.0-rc0
v2.36.0-rc1
v2.36.0-rc2
v2.36.1
v2.36.2
v2.36.3
v2.36.4
v2.36.5
v2.36.6
v2.37.0
v2.37.0-rc0
v2.37.0-rc1
v2.37.0-rc2
v2.37.1
v2.37.2
v2.37.3
v2.37.4
v2.37.5
v2.37.6
v2.37.7
v2.38.0
v2.38.0-rc0
v2.38.0-rc1
v2.38.0-rc2
v2.38.1
v2.38.2
v2.38.3
v2.38.4
v2.38.5
v2.39.0
v2.39.0-rc0
v2.39.0-rc1
v2.39.0-rc2
v2.39.1
v2.39.2
v2.39.3
v2.4.0
v2.4.0-rc0
v2.4.0-rc1
v2.4.0-rc2
v2.4.0-rc3
v2.4.1
v2.4.10
v2.4.11
v2.4.12
v2.4.2
v2.4.3
v2.4.4
v2.4.5
v2.4.6
v2.4.7
v2.4.8
v2.4.9
v2.40.0
v2.40.0-rc0
v2.40.0-rc1
v2.40.0-rc2
v2.40.1
v2.41.0
v2.41.0-rc0
v2.41.0-rc1
v2.41.0-rc2
v2.5.0
v2.5.0-rc0
v2.5.0-rc1
v2.5.0-rc2
v2.5.0-rc3
v2.5.1
v2.5.2
v2.5.3
v2.5.4
v2.5.5
v2.5.6
v2.6.0
v2.6.0-rc0
v2.6.0-rc1
v2.6.0-rc2
v2.6.0-rc3
v2.6.1
v2.6.2
v2.6.3
v2.6.4
v2.6.5
v2.6.6
v2.6.7
v2.7.0
v2.7.0-rc0
v2.7.0-rc1
v2.7.0-rc2
v2.7.0-rc3
v2.7.1
v2.7.2
v2.7.3
v2.7.4
v2.7.5
v2.7.6
v2.8.0
v2.8.0-rc0
v2.8.0-rc1
v2.8.0-rc2
v2.8.0-rc3
v2.8.0-rc4
v2.8.1
v2.8.2
v2.8.3
v2.8.4
v2.8.5
v2.8.6
v2.9.0
v2.9.0-rc0
v2.9.0-rc1
v2.9.0-rc2
v2.9.1
v2.9.2
v2.9.3
v2.9.4
v2.9.5
${ noResults }
116 Commits (f6526728f950cacfd5b5e42bcc65f2c47f3da654)
Author | SHA1 | Message | Date |
---|---|---|---|
Jeff King | dc1daacdcc |
pack-bitmap: check pack validity when opening bitmap
When pack-objects adds an entry to its list of objects to pack, it may mark the packfile and offset that contains the file, which we can later use to output the object verbatim. If the packfile is deleted while we are running (e.g., by another process running "git repack"), we may die in use_pack() if the pack file cannot be opened. We worked around this in |
4 years ago |
Taylor Blau | ddcb189d9d |
pack-bitmap: clarify comment in filter_bitmap_exclude_type()
The code that eventually became filter_bitmap_exclude_type() was originally introduced in |
4 years ago |
Jeff King | aa9ad6fee5 |
bitmaps: don't recurse into trees already in the bitmap
If an object is already mentioned in a reachability bitmap we are building, then by definition so are all of the objects it can reach. We have an optimization to stop traversing commits when we see they are already in the bitmap, but we don't do the same for trees. It's generally unavoidable to recurse into trees for commits not yet covered by bitmaps (since most commits generally do have unique top-level trees). But they usually have subtrees that are shared with other commits (i.e., all of the subtrees the commit _didn't_ touch). And some of those commits (and their trees) may be covered by the bitmap. Usually this isn't _too_ big a deal, because we'll visit those subtrees only once in total for the whole walk. But if you have a large number of unbitmapped commits, and if your tree is big, then you may end up opening a lot of sub-trees for no good reason. We can use the same optimization we do for commits here: when we are about to open a tree, see if it's in the bitmap (either the one we are building, or the "seen" bitmap which covers the UNINTERESTING side of the bitmap when doing a set-difference). This works especially well because we'll visit all commits before hitting any trees. So even in a history like: A -- B if "A" has a bitmap on disk but "B" doesn't, we'll already have OR-ed in the results from A before looking at B's tree (so we really will only look at trees touched by B). For most repositories, the timings produced by p5310 are unspectacular. Here's linux.git: Test HEAD^ HEAD -------------------------------------------------------------------- 5310.4: simulated clone 6.00(5.90+0.10) 5.98(5.90+0.08) -0.3% 5310.5: simulated fetch 2.98(5.45+0.18) 2.85(5.31+0.18) -4.4% 5310.7: rev-list (commits) 0.32(0.29+0.03) 0.33(0.30+0.03) +3.1% 5310.8: rev-list (objects) 1.48(1.44+0.03) 1.49(1.44+0.05) +0.7% Any improvement there is within the noise (the +3.1% on test 7 has to be noise, since we are not recursing into trees, and thus the new code isn't even run). The results for git.git are likewise uninteresting. But here are numbers from some other real-world repositories (that are not public). This one's tree is comparable in size to linux.git, but has ~16k refs (and so less complete bitmap coverage): Test HEAD^ HEAD ------------------------------------------------------------------------- 5310.4: simulated clone 38.34(39.86+0.74) 33.95(35.53+0.76) -11.5% 5310.5: simulated fetch 2.29(6.31+0.35) 2.20(5.97+0.41) -3.9% 5310.7: rev-list (commits) 0.99(0.86+0.13) 0.96(0.85+0.11) -3.0% 5310.8: rev-list (objects) 11.32(11.04+0.27) 6.59(6.37+0.21) -41.8% And here's another with a very large tree (~340k entries), and a fairly large number of refs (~10k): Test HEAD^ HEAD ------------------------------------------------------------------------- 5310.3: simulated clone 53.83(54.71+1.54) 39.77(40.76+1.50) -26.1% 5310.4: simulated fetch 19.91(20.11+0.56) 19.79(19.98+0.67) -0.6% 5310.6: rev-list (commits) 0.54(0.44+0.11) 0.51(0.43+0.07) -5.6% 5310.7: rev-list (objects) 24.32(23.59+0.73) 9.85(9.49+0.36) -59.5% This patch provides substantial improvements in these larger cases, and have any drawbacks for smaller ones (the cost of the bitmap check is quite small compared to an actual tree traversal). Note that we have to add a version of revision.c's include_check callback which handles non-commits. We could possibly consolidate this into a single callback for all objects types, as there's only one user of the feature which would need converted (pack-bitmap.c:should_include). That would in theory let us avoid duplicating any logic. But when I tried it, the code ended up much worse to read, with lots of repeated "if it's a commit do this, otherwise do that". Having two separate callbacks splits that naturally, and matches the existing split of show_commit/show_object callbacks. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Jeff King | 1e951c6473 |
pack-bitmap: clean up include_check after use
When a bitmap walk has to traverse (to fill in non-bitmapped objects), we use rev_info's include_check mechanism to let us stop the traversal early. But after setting the function and its data parameter, we never clean it up. This means that if the rev_info is used for a subsequent traversal without bitmaps, it will unexpectedly call into our include_check function (worse, it will do so pointing to a now-defunct stack variable in include_check_data, likely resulting in a segfault). There's no code which does this now, but it's an accident waiting to happen. Let's clean up after ourselves in the bitmap code. Reported-by: David Emett <dave@sp4m.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Patrick Steinhardt | 9cf68b27d5 |
rev-list: allow filtering of provided items
When providing an object filter, it is currently impossible to also filter provided items. E.g. when executing `git rev-list HEAD` , the commit this reference points to will be treated as user-provided and is thus excluded from the filtering mechanism. This makes it harder than necessary to properly use the new `--filter=object:type` filter given that even if the user wants to only see blobs, he'll still see commits of provided references. Improve this by introducing a new `--filter-provided-objects` option to the git-rev-parse(1) command. If given, then all user-provided references will be subject to filtering. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Patrick Steinhardt | 169a15ebd6 |
pack-bitmap: implement combined filter
When the user has multiple objects filters specified, then this is internally represented by having a "combined" filter. These combined filters aren't yet supported by bitmap indices and can thus not be accelerated. Fix this by implementing support for these combined filters. The implementation is quite trivial: when there's a combined filter, we simply recurse into `filter_bitmap()` for all of the sub-filters. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Patrick Steinhardt | 7ab6aafa58 |
pack-bitmap: implement object type filter
The preceding commit has added a new object filter for git-rev-list(1) which allows to filter objects by type. Implement the equivalent filter for packfile bitmaps so that we can answer these queries fast. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | 3f267a1128 |
builtin/pack-objects.c: respect 'pack.preferBitmapTips'
When writing a new pack with a bitmap, it is sometimes convenient to indicate some reference prefixes which should receive priority when selecting which commits to receive bitmaps. A truly motivated caller could accomplish this by setting 'pack.islandCore', (since all commits in the core island are similarly marked as preferred) but this requires callers to opt into using delta islands, which they may or may not want to do. Introduce a new multi-valued configuration, 'pack.preferBitmapTips' to allow callers to specify a list of reference prefixes. All references which have a prefix contained in 'pack.preferBitmapTips' will mark their tips as "preferred" in the same way as commits are marked as preferred for selection by 'pack.islandCore'. The choice of the verb "prefer" is intentional: marking the NEEDS_BITMAP flag on an object does *not* guarantee that that object will receive a bitmap. It merely guarantees that that commit will receive a bitmap over any *other* commit in the same window by bitmap_writer_select_commits(). The test this patch adds reflects this quirk, too. It only tests that a commit (which didn't receive bitmaps by default) is selected for bitmaps after changing the value of 'pack.preferBitmapTips' to include it. Other commits may lose their bitmaps as a byproduct of how the selection process works (bitmap_writer_select_commits() ignores the remainder of a window after seeing a commit with the NEEDS_BITMAP flag). This configuration will aide in selecting important references for multi-pack bitmaps, since they do not respect the same pack.islandCore configuration. (They could, but doing so may be confusing, since it is packs--not bitmaps--which are influenced by the delta-islands configuration). In a fork network repository (one which lists all forks of a given repository as remotes), for example, it is useful to set pack.preferBitmapTips to 'refs/remotes/<root>/heads' and 'refs/remotes/<root>/tags', where '<root>' is an opaque identifier referring to the repository which is at the base of the fork chain. Suggested-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | dff5e49e51 |
pack-bitmap: add 'test_bitmap_commits()' helper
The next patch will add a 'bitmap' test-tool which prints the list of commits that have bitmaps computed. The test helper could implement this itself, but it would need access to the 'bitmaps' field of the 'pack_bitmap' struct. To avoid exposing this private detail, implement the entirety of the helper behind a test_bitmap_commits() function in pack-bitmap.c. There is some precedence for this with test_bitmap_walk() which is used to implement the '--test-bitmap' flag in 'git rev-list' (and is also implemented in pack-bitmap.c). A caller will be added in the next patch. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Patrick Steinhardt | 540cdc11ad |
pack-bitmap: avoid traversal of objects referenced by uninteresting tag
When preparing the bitmap walk, we first establish the set of of have and want objects by iterating over the set of pending objects: if an object is marked as uninteresting, it's declared as an object we already have, otherwise as an object we want. These two sets are then used to compute which transitively referenced objects we need to obtain. One special case here are tag objects: when a tag is requested, we resolve it to its first not-tag object and add both resolved objects as well as the tag itself into either the have or want set. Given that the uninteresting-property always propagates to referenced objects, it is clear that if the tag is uninteresting, so are its children and vice versa. But we fail to propagate the flag, which effectively means that referenced objects will always be interesting except for the case where they have already been marked as uninteresting explicitly. This mislabeling does not impact correctness: we now have it in our "wants" set, and given that we later do an `AND NOT` of the bitmaps of "wants" and "haves" sets it is clear that the result must be the same. But we now start to needlessly traverse the tag's referenced objects in case it is uninteresting, even though we know that each referenced object will be uninteresting anyway. In the worst case, this can lead to a complete graph walk just to establish that we do not care for any object. Fix the issue by propagating the `UNINTERESTING` flag to pointees of tag objects and add a benchmark with negative revisions to p5310. This shows some nice performance benefits, tested with linux.git: Test HEAD~ HEAD --------------------------------------------------------------------------------------------------------------- 5310.3: repack to disk 193.18(181.46+16.42) 194.61(183.41+15.83) +0.7% 5310.4: simulated clone 25.93(24.88+1.05) 25.81(24.73+1.08) -0.5% 5310.5: simulated fetch 2.64(5.30+0.69) 2.59(5.16+0.65) -1.9% 5310.6: pack to file (bitmap) 58.75(57.56+6.30) 58.29(57.61+5.73) -0.8% 5310.7: rev-list (commits) 1.45(1.18+0.26) 1.46(1.22+0.24) +0.7% 5310.8: rev-list (objects) 15.35(14.22+1.13) 15.30(14.23+1.07) -0.3% 5310.9: rev-list with tag negated via --not --all (objects) 22.49(20.93+1.56) 0.11(0.09+0.01) -99.5% 5310.10: rev-list with negative tag (objects) 0.61(0.44+0.16) 0.51(0.35+0.16) -16.4% 5310.11: rev-list count with blob:none 12.15(11.19+0.96) 12.18(11.19+0.99) +0.2% 5310.12: rev-list count with blob:limit=1k 17.77(15.71+2.06) 17.75(15.63+2.12) -0.1% 5310.13: rev-list count with tree:0 1.69(1.31+0.38) 1.68(1.28+0.39) -0.6% 5310.14: simulated partial clone 20.14(19.15+0.98) 19.98(18.93+1.05) -0.8% 5310.16: clone (partial bitmap) 12.78(13.89+1.07) 12.72(13.99+1.01) -0.5% 5310.17: pack to file (partial bitmap) 42.07(45.44+2.72) 41.44(44.66+2.80) -1.5% 5310.18: rev-list with tree filter (partial bitmap) 0.44(0.29+0.15) 0.46(0.32+0.14) +4.5% While most benchmarks are probably in the range of noise, the newly added 5310.9 and 5310.10 benchmarks consistenly perform better. Signed-off-by: Patrick Steinhardt <ps@pks.im>. Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
René Scharfe | ca56dadb4b |
use CALLOC_ARRAY
Add and apply a semantic patch for converting code that open-codes CALLOC_ARRAY to use it instead. It shortens the code and infers the element size automatically. Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Jeff King | 16950f8384 |
rev-list: add --disk-usage option for calculating disk usage
It can sometimes be useful to see which refs are contributing to the overall repository size (e.g., does some branch have a bunch of objects not found elsewhere in history, which indicates that deleting it would shrink the size of a clone). You can find that out by generating a list of objects, getting their sizes from cat-file, and then summing them, like: git rev-list --objects --no-object-names main..branch git cat-file --batch-check='%(objectsize:disk)' | perl -lne '$total += $_; END { print $total }' Though note that the caveats from git-cat-file(1) apply here. We "blame" base objects more than their deltas, even though the relationship could easily be flipped. Still, it can be a useful rough measure. But one problem is that it's slow to run. Teaching rev-list to sum up the sizes can be much faster for two reasons: 1. It skips all of the piping of object names and sizes. 2. If bitmaps are in use, for objects that are in the bitmapped packfile we can skip the oid_object_info() lookup entirely, and just ask the revindex for the on-disk size. This patch implements a --disk-usage option which produces the same answer in a fraction of the time. Here are some timings using a clone of torvalds/linux: [rev-list piped to cat-file, no bitmaps] $ time git rev-list --objects --no-object-names --all | git cat-file --buffer --batch-check='%(objectsize:disk)' | perl -lne '$total += $_; END { print $total }' 1459938510 real 0m29.635s user 0m38.003s sys 0m1.093s [internal, no bitmaps] $ time git rev-list --disk-usage --objects --all 1459938510 real 0m31.262s user 0m30.885s sys 0m0.376s Even though the wall-clock time is slightly worse due to parallelism, notice the CPU savings between the two. We saved 21% of the CPU just by avoiding the pipes. But the real win is with bitmaps. If we use them without the new option: [rev-list piped to cat-file, bitmaps] $ time git rev-list --objects --no-object-names --all --use-bitmap-index | git cat-file --batch-check='%(objectsize:disk)' | perl -lne '$total += $_; END { print $total }' 1459938510 real 0m6.244s user 0m8.452s sys 0m0.311s then we're faster to generate the list of objects, but we still spend a lot of time piping and looking things up. But if we do both together: [internal, bitmaps] $ time git rev-list --disk-usage --objects --all --use-bitmap-index 1459938510 real 0m0.219s user 0m0.169s sys 0m0.049s then we get the same answer much faster. For "--all", that answer will correspond closely to "du objects/pack", of course. But we're actually checking reachability here, so we're still fast when we ask for more interesting things: $ time git rev-list --disk-usage --use-bitmap-index v5.0..v5.10 374798628 real 0m0.429s user 0m0.356s sys 0m0.072s Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | 78232bf65d |
rebuild_existing_bitmaps(): convert to new revindex API
Remove another instance of looking at the revindex directly by instead calling 'pack_pos_to_index()'. Unlike other patches, this caller only cares about the index position of each object in the loop. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | 011f3fd5cd |
try_partial_reuse(): convert to new revindex API
Remove another instance of direct revindex manipulation by calling 'pack_pos_to_offset()' instead (the caller here does not care about the index position of the object at position 'pos'). Note that we cannot just use the existing "offset" variable to store the value we get from pack_pos_to_offset(). It is incremented by unpack_object_header(), but we later need the original value. Since we'll no longer have revindex->offset to read it from, we'll store that in a separate variable ("header" since it points to the entry's header bytes). Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | a78a90324d |
get_size_by_pos(): convert to new revindex API
Remove another caller that holds onto a 'struct revindex_entry' by replacing the direct indexing with calls to 'pack_pos_to_offset()' and 'pack_pos_to_index()'. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | cf98f2e8e0 |
show_objects_for_type(): convert to new revindex API
Avoid storing the revindex entry directly, since this structure will soon be removed from the public interface. Instead, store the offset and index position by calling 'pack_pos_to_offset()' and 'pack_pos_to_index()', respectively. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | 57665086af |
bitmap_position_packfile(): convert to new revindex API
Replace find_revindex_position() with its counterpart in the new API, offset_to_pack_pos(). Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | 83578051a9 |
pack-bitmap: factor out 'add_commit_to_bitmap()'
'find_objects()' currently needs to interact with the bitmaps khash pretty closely. To make 'find_objects()' read a little more straightforwardly, remove some of the khash-level details into a new function that describes what it does: 'add_commit_to_bitmap()'. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | 98c31f366a |
pack-bitmap: factor out 'bitmap_for_commit()'
A couple of callers within pack-bitmap.c duplicate logic to lookup a given object id in the bitamps khash. Factor this out into a new function, 'bitmap_for_commit()' to reduce some code duplication. Make this new function non-static, since it will be used in later commits from outside of pack-bitmap.c. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Jeff King | 449fa5ee06 |
pack-bitmap-write: ignore BITMAP_FLAG_REUSE
The on-disk bitmap format has a flag to mark a bitmap to be "reused". This is a rather curious feature, and works like this: - a run of pack-objects would decide to mark the last 80% of the bitmaps it generates with the reuse flag - the next time we generate bitmaps, we'd see those reuse flags from the last run, and mark those commits as special: - we'd be more likely to select those commits to get bitmaps in the new output - when generating the bitmap for a selected commit, we'd reuse the old bitmap as-is (rearranging the bits to match the new pack, of course) However, neither of these behaviors particularly makes sense. Just because a commit happened to be bitmapped last time does not make it a good candidate for having a bitmap this time. In particular, we may choose bitmaps based on how recent they are in history, or whether a ref tip points to them, and those things will change. We're better off re-considering fresh which commits are good candidates. Reusing the existing bitmap _is_ a reasonable thing to do to save computation. But only reusing exact bitmaps is a weak form of this. If we have an old bitmap for A and now want a new bitmap for its child, we should be able to compute that only by looking at trees and that are new to the child. But this code would consider only exact reuse (which is perhaps why it was eager to select those commits in the first place). Furthermore, the recent switch to the reverse-edge algorithm for generating bitmaps dropped this optimization entirely (and yet still performs better). So let's do a few cleanups: - drop the whole "reusing bitmaps" phase of generating bitmaps. It's not helping anything, and is mostly unused code (or worse, code that is using CPU but not doing anything useful) - drop the use of the on-disk reuse flag to select commits to bitmap - stop setting the on-disk reuse flag in bitmaps we generate (since nothing respects it anymore) We will keep a few innards of the reuse code, which will help us implement a more capable version of the "reuse" optimization: - simplify rebuild_existing_bitmaps() into a function that only builds the mapping of bits between the old and new orders, but doesn't actually convert any bitmaps - make rebuild_bitmap() public; we'll call it lazily to convert bitmaps as we traverse (using the mapping created above) Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Taylor Blau | c6b0c3910c |
pack-bitmap.c: check reads more aggressively when loading
Before 'load_bitmap_entries_v1()' reads an actual EWAH bitmap, it should check that it can safely do so by ensuring that there are at least 6 bytes available to be read (four for the commit's index position, and then two more for the xor offset and flags, respectively). Likewise, it should check that the commit index it read refers to a legitimate object in the pack. The first fix catches a truncation bug that was exposed when testing, and the second is purely precautionary. There are some possible future improvements, not pursued here. They are: - Computing the correct boundary of the bitmap itself in the caller and ensuring that we don't read past it. This may or may not be worth it, since in a truncation situation, all bets are off: (is the trailer still there and the bitmap entries malformed, or is the trailer truncated?). The best we can do is try to read what's there as if it's correct data (and protect ourselves when it's obviously bogus). - Avoid the magic "6" by teaching read_be32() and read_u8() (both of which are custom helpers for this function) to check sizes before advancing the pointers. - Adding more tests in this area. Testing these truncation situations are remarkably fragile to even subtle changes in the bitmap generation. So, the resulting tests are likely to be quite brittle. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Jeff King | 2978b00691 |
rev-list: die when --test-bitmap detects a mismatch
You can use "git rev-list --test-bitmap HEAD" to check that bitmaps produce the same answer we'd get from a regular traversal. But if we detect an error, we only print "mismatch", and still exit with a successful error code. That makes the uses of --test-bitmap in the test suite (e.g., in t5310) mostly pointless: even if we saw an error, the tests wouldn't notice. Let's instead call die(), which will let these tests work as designed, and alert us if the bitmaps are bogus. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Jeff King | ec6c7b4367 |
pack-bitmap: bounds-check size of cache extension
A .bitmap file may have a "name hash cache" extension, which puts a sequence of uint32_t values (one per object) at the end of the file. When we see a flag indicating this extension, we blindly subtract the appropriate number of bytes from our available length. However, if the .bitmap file is too short, we'll underflow our length variable and wrap around, thinking we have a very large length. This can lead to reading out-of-bounds bytes while loading individual ewah bitmaps. We can fix this by checking the number of available bytes when we parse the header. The existing "truncated bitmap" test is now split into two tests: one where we don't have this extension at all (and hence actually do try to read a truncated ewah bitmap) and one where we realize up-front that we can't even fit in the cache structure. We'll check stderr in each case to make sure we hit the error we're expecting. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
4 years ago |
Jeff King | ca51090200 |
pack-bitmap: fix header size check
When we parse a .bitmap header, we first check that we have enough bytes
to make a valid header. We do that based on sizeof(struct
bitmap_disk_header). However, as of
|
4 years ago |
Jeff King | 9639474b6d |
pack-bitmap: pass object filter to fill-in traversal
Sometimes a bitmap traversal still has to walk some commits manually, because those commits aren't included in the bitmap packfile (e.g., due to a push or commit since the last full repack). If we're given an object filter, we don't pass it down to this traversal. It's not necessary for correctness because the bitmap code has its own filters to post-process the bitmap result (which it must, to filter out the objects that _are_ mentioned in the bitmapped packfile). And with blob filters, there was no performance reason to pass along those filters, either. The fill-in traversal could omit them from the result, but it wouldn't save us any time to do so, since we'd still have to walk each tree entry to see if it's a blob or not. But now that we support tree filters, there's opportunity for savings. A tree:depth=0 filter means we can avoid accessing trees entirely, since we know we won't them (or any of the subtrees or blobs they point to). The new test in p5310 shows this off (the "partial bitmap" state is one where HEAD~100 and its ancestors are all in a bitmapped pack, but HEAD~100..HEAD are not). Here are the results (run against linux.git): Test HEAD^ HEAD ------------------------------------------------------------------------------------------------- [...] 5310.16: rev-list with tree filter (partial bitmap) 0.19(0.17+0.02) 0.03(0.02+0.01) -84.2% The absolute number of savings isn't _huge_, but keep in mind that we only omitted 100 first-parent links (in the version of linux.git here, that's 894 actual commits). In a more pathological case, we might have a much larger proportion of non-bitmapped commits. I didn't bother creating such a case in the perf script because the setup is expensive, and this is plenty to show the savings as a percentage. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Taylor Blau | b0a8d4820b |
pack-bitmap.c: support 'tree:0' filtering
In the previous patch, we made it easy to define other filters that exclude all objects of a certain type. Use that in order to implement bitmap-level filtering for the '--filter=tree:<n>' filter when 'n' is equal to 0. The general case is not helped by bitmaps, since for values of 'n > 0', the object filtering machinery requires a full-blown tree traversal in order to determine the depth of a given tree. Caching this is non-obvious, too, since the same tree object can have a different depth depending on the context (e.g., a tree was moved up in the directory hierarchy between two commits). But, the 'n = 0' case can be helped, and this patch does so. Running p5310.11 in this tree and on master with the kernel, we can see that this case is helped substantially: Test master this tree -------------------------------------------------------------------------------- 5310.11: rev-list count with tree:0 10.68(10.39+0.27) 0.06(0.04+0.01) -99.4% Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Taylor Blau | 856e12c18a |
pack-bitmap.c: make object filtering functions generic
In
|
5 years ago |
Jeff King | 500e4f2366 |
pack-bitmap: use object_id when loading on-disk bitmaps
A pack bitmap file contains the index position of the commit for each bitmap, which we then translate into an object id via nth_packed_object_sha1(). In preparation for that function going away, we can switch to the more type-safe nth_packed_object_id(). Note that even though the result ends up in an object_id this does incur an extra copy of the hash (into our temporary object_id, and then into the final malloc'd stored_bitmap struct). This shouldn't make any measurable difference. If it did, we could avoid this copy _and_ the copy of the rest of the items by allocating the stored_bitmap struct beforehand and reading directly into it from the bitmap file. Or better still, if this is a bottleneck, we could introduce an on-disk index to the bitmap file so we don't have to read every single entry to use just one of them. So it's not worth worrying about micro-optimizing out this one hash copy. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 0763671b8e |
nth_packed_object_oid(): use customary integer return
Our nth_packed_object_sha1() function returns NULL for error. So when we wrapped it with nth_packed_object_oid(), we kept the same semantics. But it's a bit funny, because the caller actually passes in an out parameter, and the pointer we return is just that same struct they passed to us (or NULL). It's not too terrible, but it does make the interface a little non-idiomatic. Let's switch to our usual "0 for success, negative for error" return value. Most callers either don't check it, or are trivially converted. The one that requires the biggest change is actually improved, as we can ditch an extra aliased pointer variable. Since we are changing the interface in a subtle way that the compiler wouldn't catch, let's also change the name to catch any topics in flight. We can drop the 'o' and make it nth_packed_object_id(). That's slightly shorter, but also less redundant since the 'o' stands for "object" already. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 84243da129 |
pack-bitmap: implement BLOB_LIMIT filtering
Just as the previous commit implemented BLOB_NONE, we can support BLOB_LIMIT filters by looking at the sizes of any blobs in the result and unsetting their bits as appropriate. This is slightly more expensive than BLOB_NONE, but still produces a noticeable speedup (these results are on git.git): Test HEAD~2 HEAD ------------------------------------------------------------------------------------ 5310.9: rev-list count with blob:none 1.80(1.77+0.02) 0.22(0.20+0.02) -87.8% 5310.10: rev-list count with blob:limit=1k 1.99(1.96+0.03) 0.29(0.25+0.03) -85.4% The implementation is similar to the BLOB_NONE one, with the exception that we have to go object-by-object while walking the blob-type bitmap (since we can't mask out the matches, but must look up the size individually for each blob). The trick with using ctz64() is taken from show_objects_for_type(), which likewise needs to find individual bits (but wants to quickly skip over big chunks without blobs). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 4f3bd5606a |
pack-bitmap: implement BLOB_NONE filtering
We can easily support BLOB_NONE filters with bitmaps. Since we know the types of all of the objects, we just need to clear the result bits of any blobs. Note two subtleties in the implementation (which I also called out in comments): - we have to include any blobs that were specifically asked for (and not reached through graph traversal) to match the non-bitmap version - we have to handle in-pack and "ext_index" objects separately. Arguably prepare_bitmap_walk() could be adding these ext_index objects to the type bitmaps. But it doesn't for now, so let's match the rest of the bitmap code here (it probably wouldn't be an efficiency improvement to do so since the cost of extending those bitmaps is about the same as our loop here, but it might make the code a bit simpler). Here are perf results for the new test on git.git: Test HEAD^ HEAD -------------------------------------------------------------------------------- 5310.9: rev-list count with blob:none 1.67(1.62+0.05) 0.22(0.21+0.02) -86.8% Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 6663ae0a08 |
pack-bitmap: basic noop bitmap filter infrastructure
Currently you can't use object filters with bitmaps, but we plan to support at least some filters with bitmaps. Let's introduce some infrastructure that will help us do that: - prepare_bitmap_walk() now accepts a list_objects_filter_options parameter (which can be NULL for no filtering; all the current callers pass this) - we'll bail early if the filter is incompatible with bitmaps (just as we would if there were no bitmaps at all). Currently all filters are incompatible. - we'll filter the resulting bitmap; since there are no supported filters yet, this is always a noop. There should be no behavior change yet, but we'll support some actual filters in a future patch. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 4eb707ebd6 |
rev-list: allow commit-only bitmap traversals
Ever since we added reachability bitmap support, we've been able to use it with rev-list to get the full list of objects, like: git rev-list --objects --use-bitmap-index --all But you can't do so without --objects, since we weren't ready to just show the commits. However, the internals of the bitmap code are mostly ready for this: they avoid opening up trees when walking to fill in the bitmaps. We just need to actually pass in the rev_info to traverse_bitmap_commit_list() so it knows which types to bother triggering our callback for. For completeness, the perf test now covers both the existing --objects case, as well as the new commits-only behavior (the objects one got way faster when we introduced bitmaps, but obviously isn't improved now). Here are numbers for linux.git: Test HEAD^ HEAD ------------------------------------------------------------------------ 5310.7: rev-list (commits) 8.29(8.10+0.19) 1.76(1.72+0.04) -78.8% 5310.8: rev-list (objects) 8.06(7.94+0.12) 8.14(7.94+0.13) +1.0% That run was cheating a little, as I didn't have any commit-graph in the repository, and we'd built it by default these days when running git-gc. Here are numbers with a commit-graph: Test HEAD^ HEAD ------------------------------------------------------------------------ 5310.7: rev-list (commits) 0.70(0.58+0.12) 0.51(0.46+0.04) -27.1% 5310.8: rev-list (objects) 6.20(6.09+0.10) 6.27(6.16+0.11) +1.1% Still an improvement, but a lot less impressive. We could have the perf script remove any commit-graph to show the out-sized effect, but it probably makes sense to leave it in what would be a more typical setup. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | d90fe06ea7 |
pack-bitmap: refuse to do a bitmap traversal with pathspecs
rev-list has refused to use bitmaps with pathspec limiting since
|
5 years ago |
Jeff King | acac50dd8c |
pack-bitmap: fix leak of haves/wants object lists
When we do a bitmap-aware revision traversal, we create an object_list for each of the "haves" and "wants" tips. After creating the result bitmaps these are no longer needed or used, but we never free the list memory. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 551cf8b655 |
pack-bitmap: factor out type iterator initialization
When count_object_type() wants to iterate over the bitmap of all objects of a certain type, we have to pair up OBJ_COMMIT with bitmap->commits, and so forth. Since we're about to add more code to iterate over these bitmaps, let's pull the initialization into its own function. We can also use this to simplify traverse_bitmap_commit_list(). It accomplishes the same thing by manually passing the object type and the bitmap to show_objects_for_type(), but using our helper we just need the object type. Note there's one small code change here: previously we'd simply return zero when counting an unknown object type, and now we'll BUG(). This shouldn't matter in practice, as all of the callers pass in only usual commit/tree/blob/tag types. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | d2ea031046 |
pack-bitmap: don't rely on bitmap_git->reuse_objects
We no longer compute bitmap_git->reuse_objects, so we cannot rely on it anymore to terminate the loop early; we have to iterate to the end. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | bb514de356 |
pack-objects: improve partial packfile reuse
The old code to reuse deltas from an existing packfile just tried to dump a whole segment of the pack verbatim. That's faster than the traditional way of actually adding objects to the packing list, but it didn't kick in very often. This new code is really going for a middle ground: do _some_ per-object work, but way less than we'd traditionally do. The general strategy of the new code is to make a bitmap of objects from the packfile we'll include, and then iterate over it, writing out each object exactly as it is in our on-disk pack, but _not_ adding it to our packlist (which costs memory, and increases the search space for deltas). One complication is that if we're omitting some objects, we can't set a delta against a base that we're not sending. So we have to check each object in try_partial_reuse() to make sure we have its delta. About performance, in the worst case we might have interleaved objects that we are sending or not sending, and we'd have as many chunks as objects. But in practice we send big chunks. For instance, packing torvalds/linux on GitHub servers now reused 6.5M objects, but only needed ~50k chunks. Helped-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 8ebf529661 |
pack-bitmap: simplify bitmap_has_oid_in_uninteresting()
Let's refactor bitmap_has_oid_in_uninteresting() using bitmap_walk_contains(). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 59b2829ec5 |
pack-bitmap: uninteresting oid can be outside bitmapped packfile
bitmap_has_oid_in_uninteresting() only used bitmap_position_packfile(), not bitmap_position(). So it wouldn't find objects which weren't in the bitmapped packfile (i.e., ones where we extended the bitmap to handle loose objects, or objects in other packs). As we could reuse a delta against such an object it is suboptimal not to use bitmap_position(), so let's use it instead of bitmap_position_packfile(). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 40d18ff8c6 |
pack-bitmap: introduce bitmap_walk_contains()
We will use this helper function in a following commit to tell us if an object is packed. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
5 years ago |
Jeff King | 3a37876b5d |
pack-objects: drop packlist index_pos optimization
Once upon a time, the code to add an object to our packing list in
pack-objects all lived in a single function. It computed the position
within the hash table once, then used it to check if the object was
already present, and if not, to add it.
Later, in
|
6 years ago |
René Scharfe | dad3f0607b |
tag: factor out get_tagged_oid()
Add a function for accessing the ID of the object referenced by a tag safely, i.e. without causing a segfault when encountering a broken tag where ->tagged is NULL. Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |
Jeff King | d2bc62b1fa |
pack-bitmap: convert khash_sha1 maps into kh_oid_map
All of the users of our khash_sha1 maps actually have a "struct object_id". Let's use the more descriptive type. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |
Jeff King | 4ed43d16d7 |
khash: drop broken oid_map typedef
Commit |
6 years ago |
Jeff King | 3df28caefb |
pack-objects: convert packlist_find() to use object_id
We take a raw hash pointer, but most of our callers have a "struct object_id" already. Let's switch to taking the full struct, which will let us continue removing uses of raw sha1 buffers. There are two callers that do need special attention: - in rebuild_existing_bitmaps(), we need to switch to nth_packed_object_oid(). This incurs an extra hash copy over pointing straight to the mmap'd sha1, but it shouldn't be measurable compared to the rest of the operation. - in can_reuse_delta() we already spent the effort to copy the sha1 into a "struct object_id", but now we just have to do so a little earlier in the function (we can't easily convert that function's callers because they may be pointing at mmap'd REF_DELTA blocks). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |
Jeff King | 4828ce9871 |
pack-revindex: open index if necessary
We can't create a pack revindex if we haven't actually looked at the index. Normally we would never get as far as creating a revindex without having already been looking in the pack, so this code never bothered to double-check that pack->index_data had been loaded. But with the new multi-pack-index feature, many code paths might not load the individual pack .idx at all (they'd find objects via the midx and then open the .pack, but not its index). This can't yet be triggered in practice, because a bug in the midx code means we accidentally open up the individual .idx files anyway. But in preparation for fixing that, let's have the revindex code check that everything it needs has been loaded. In most cases this will just be a quick noop. But note that this does introduce a possibility of error (if we have to open the index and it's corrupt), so load_pack_revindex() now returns a result code, and callers need to handle the error. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |
brian m. carlson | 3c7714485d |
pack-bitmap: switch hash tables to use struct object_id
Instead of storing unsigned char pointers in the hash tables, switch to storing instances of struct object_id. Update several internal functions and one external function to take pointers to struct object_id. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |
brian m. carlson | 9941e920e0 |
pack-bitmap: switch hard-coded constants to the_hash_algo
Switch two hard-coded uses of 20 to references to the_hash_algo. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |
brian m. carlson | 0dd4924891 |
pack-bitmap: replace sha1_to_hex
Replace the uses of sha1_to_hex in the pack bitmap code with hash_to_hex to allow the use of SHA-256 as well. Rename a few variables since they are no longer limited to SHA-1. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> |
6 years ago |