kernel/git - git - PowerEL Git System

Commit Graph

Author	SHA1	Message	Date
Victoria Dye	037f8ea6d9	unpack-trees: fix sparse directory recursion check Ensure 'is_sparse_directory_entry()' receives a valid 'name_entry ' if one exists in the list of tree(s) being unpacked in 'unpack_callback()'. Currently, 'is_sparse_directory_entry()' is called with the first 'name_entry' in the 'names' list of entries on 'unpack_callback()'. However, this entry may be empty even when other elements of 'names' are not (such as when switching from an orphan branch back to a "normal" branch). As a result, 'is_sparse_directory_entry()' could incorrectly indicate that a sparse directory is not* actually sparse because the name of the index entry does not match the (empty) 'name_entry' path. Fix the issue by using the existing 'name_entry p' value in 'unpack_callback()', which points to the first non-empty entry in 'names'. Because 'p' is 'const', also update 'is_sparse_directory_entry()'s 'name_entry ' argument to be 'const'. Finally, add a regression test case. Reported-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	2 years ago
Eric Sunshine	0e66bc1b21	t: detect and signal failure within loop Failures within `for` and `while` loops can go unnoticed if not detected and signaled manually since the loop itself does not abort when a contained command fails, nor will a failure necessarily be detected when the loop finishes since the loop returns the exit code of the last command it ran on the final iteration, which may not be the command which failed. Therefore, detect and signal failures manually within loops using the idiom `\|\| return 1` (or `\|\| exit 1` within subshells). Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	2 years ago
Eric Sunshine	625ff5c320	t1092: fix buggy sparse "blame" test This test wants to verify that `git blame` errors out when asked to blame a file _not_ in the sparse checkout. However, the very first file it asks to blame _is_ present in the checkout, thus `test_must_fail git blame $file` gives an unexpected result (the "blame" succeeds). This problem went unnoticed because the test invokes `test_must_fail git blame $file` in loop but forgets to break out of the loop early upon failure, thus the failure gets swallowed. Fix the test by having it not ask to blame a file present in the sparse checkout, and instead only blame files not present, as intended. While at it, also add the missing `\|\| return 1` which allowed this bug to go unnoticed. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	2 years ago
Shaoxuan Yuan	ede241c715	rm: integrate with sparse-index Enable the sparse index within the `git-rm` command. The `p2000` tests demonstrate a ~92% execution time reduction for 'git rm' using a sparse index. Test HEAD~1 HEAD -------------------------------------------------------------------------- 2000.74: git rm ... (full-v3) 0.41(0.37+0.05) 0.43(0.36+0.07) +4.9% 2000.75: git rm ... (full-v4) 0.38(0.34+0.05) 0.39(0.35+0.05) +2.6% 2000.76: git rm ... (sparse-v3) 0.57(0.56+0.01) 0.05(0.05+0.00) -91.2% 2000.77: git rm ... (sparse-v4) 0.57(0.55+0.02) 0.03(0.03+0.00) -94.7% ---- Also, normalize a behavioral difference of `git-rm` under sparse-index. See related discussion [1]. `git-rm` a sparse-directory entry within a sparse-index enabled repo behaves differently from a sparse directory within a sparse-checkout enabled repo. For example, in a sparse-index repo, where 'folder1' is a sparse-directory entry, `git rm -r --sparse folder1` provides this: rm 'folder1/' Whereas in a sparse-checkout repo without sparse-index, doing so provides this: rm 'folder1/0/0/0' rm 'folder1/0/1' rm 'folder1/a' Because `git rm` a sparse-directory entry does not need to expand the index, therefore we should accept the current behavior, which is faster than "expand the sparse-directory entry to match the sparse-checkout situation". Modify a previous test so such difference is not considered as an error. [1] https://github.com/ffyuanda/git/pull/6#discussion_r934861398 Helped-by: Victoria Dye <vdye@github.com> Helped-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Shaoxuan Yuan	bcf96cfca6	rm: expand the index only when necessary Remove the `ensure_full_index()` method so `git-rm` does not always expand the index when the expansion is unnecessary, i.e. when <pathspec> does not have any possibilities to match anything outside of sparse-checkout definition. Expand the index when the <pathspec> needs an expanded index, i.e. the <pathspec> contains wildcard that may need a full-index or the <pathspec> is simply outside of sparse-checkout definition. Notice that the test 'rm pathspec expands index when necessary' in t1092 is testing this code change behavior, though it will be marked as 'test_expect_success' only in the next patch, where we officially mark `command_requires_full_index = 0`, so the index does not expand unless we tell it to do so. Notice that because we also want `ensure_full_index` to record the stdout and stderr from Git command, a corresponding modification is also included in this patch. The reason we want the "sparse-index-out" and "sparse-index-err", is that we need to make sure there is no error from Git command itself, so we can rely on the `test_region` result and determine if the index is expanded or not. Helped-by: Victoria Dye <vdye@github.com> Helped-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Shaoxuan Yuan	ba808251aa	t1092: add tests for `git-rm` Add tests for `git-rm`, make sure it behaves as expected when <pathspec> is both inside or outside of sparse-checkout definition. Helped-by: Victoria Dye <vdye@github.com> Helped-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	b15207b8cf	unpack-trees: unpack new trees as sparse directories If 'unpack_single_entry()' is unpacking a new directory tree (that is, one not already present in the index) into a sparse index, unpack the tree as a sparse directory rather than traversing its contents and unpacking each file individually. This helps keep the sparse index as collapsed as possible in cases such as 'git reset --hard' restoring a outside-of-cone directory removed with 'git rm -r --sparse'. Without this patch, 'unpack_single_entry()' will only unpack a directory into the index as a sparse directory (rather than traversing into it and unpacking its files one-by-one) if an entry with the same name already exists in the index. This patch allows sparse directory unpacking without a matching index entry when the following conditions are met: 1. the directory's path is outside the sparse cone, and 2. there are no children of the directory in the index If a directory meets these requirements (as determined by 'is_new_sparse_dir()'), 'unpack_single_entry()' unpacks the sparse directory index entry and propagates the decision back up to 'unpack_callback()' to prevent unnecessary tree traversal into the unpacked directory. Reported-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	49ff3cb90f	checkout: fix nested sparse directory diff in sparse index Add the 'recursive' diff flag to the local changes reporting done by 'git checkout' in 'show_local_changes()'. Without the flag enabled, unexpanded sparse directories will not be recursed into to report the diff of each file's contents, resulting in the reported local changes including "modified" sparse directories. The same issue was found and fixed for 'git status' in `2c521b0e49` (status: fix nested sparse directory diff in sparse index, 2022-03-01) Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Shaoxuan Yuan	b91a2b6594	mv: add check_dir_in_index() and solve general dir check issue Originally, moving a <source> directory which is not on-disk due to its existence outside of sparse-checkout cone, "giv mv" command errors out with "bad source". Add a helper check_dir_in_index() function to see if a directory name exists in the index. Also add a SKIP_WORKTREE_DIR bit to mark such directories. Change the checking logic, so that such <source> directory makes "giv mv" command warns with "advise_on_updating_sparse_paths()" instead of "bad source"; also user now can supply a "--sparse" flag so this operation can be carried out successfully. Helped-by: Victoria Dye <vdye@github.com> Helped-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Acked-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Shaoxuan Yuan	1143cc01b7	t1092: mv directory from out-of-cone to in-cone Add test for "mv: add check_dir_in_index() and solve general dir check issue" in this series. This change tests the following: 1. mv <source> as a directory on the sparse index boundary (where it would be a sparse directory in a sparse index). 2. mv <source> as a directory which is deeper than the boundary (so the sparse index would expand in the cache_name_pos() method). These tests can be written now for correctness, but later the first case can be updated to use the 'ensure_not_expanded' helper in t1092. Suggested-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> Acked-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	598b1e7d09	sparse-checkout: integrate with sparse index When modifying the sparse-checkout definition, the sparse-checkout builtin calls update_sparsity() to modify the SKIP_WORKTREE bits of all cache entries in the index. Before, we needed the index to be fully expanded in order to ensure we had the full list of files necessary that match the new patterns. Insert a call to reset_sparse_directories() that expands sparse directories that are within the new pattern list, but only far enough that every necessary file path now exists as a cache entry. The remaining logic within update_sparsity() will modify the SKIP_WORKTREE bits appropriately. This allows us to disable command_requires_full_index within the sparse-checkout builtin. Add tests that demonstrate that we are not expanding to a full index unnecessarily. We can see the improved performance in the p2000 test script: Test HEAD~1 HEAD ------------------------------------------------------------------------ 2000.24: git ... (sparse-v3) 2.14(1.55+0.58) 1.57(1.03+0.53) -26.6% 2000.25: git ... (sparse-v4) 2.20(1.62+0.57) 1.58(0.98+0.59) -28.2% These reductions of 26-28% are small compared to most examples, but the time is dominated by writing a new copy of the base repository to the worktree and then deleting it again. The fact that the previous index expansion was such a large portion of the time is telling how important it is to complete this sparse index integration. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	8846847a14	t1092: stress test 'git sparse-checkout set' The 'sparse-index contents' test checks that the sparse index has the correct set of sparse directories in the index after modifying the cone mode patterns using 'git sparse-checkout set'. Add to the coverage here by adding more complicated scenarios that were not previously tested. In order to check paths that do not exist at HEAD, we need to modify the test_sparse_checkout_set helper slightly: 1. Add the --skip-checks argument to the 'set' command to avoid failures when passing paths that do not exist at HEAD. 2. When looking for the non-existence of sparse directories for the paths in $CONE_DIRS, allow the rev-list command to fail because the path does not exist at HEAD. This allows us to add some interesting test cases. Helped-by: Victoria Dye <vdye@github.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	baa73e2b75	t1092: refactor 'sparse-index contents' test Before expanding this test with more involved cases, first extract the repeated logic into a new test_sparse_checkout_set helper. This helper checks that 'git sparse-checkout set ...' succeeds and then verifies that certain directories have sparse directory entries in the sparse index. It also verifies that the in-cone directories are _not_ sparse directory entries in the sparse index. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	0f329b9ae4	unpack-trees: preserve index sparsity When unpacking trees, set the default sparsity of the resultant index based on repo settings and 'is_sparse_index_allowed()'. Normally, when executing 'unpack_trees', the output index is marked sparse when (and only when) it unpacks a sparse directory. However, an index may be "sparse" even if it contains no sparse directories - when all files fall inside the sparse-checkout definition or otherwise have SKIP_WORKTREE disabled. Therefore, the output index may be marked "full" even when it is "sparse", resulting in unnecessary 'ensure_full_index' calls when writing to disk. Avoid this by setting the "default" index sparsity to match what is expected for the repository. As a consequence of this fix, the (non-merge) 'read-tree' performed when applying a stash with untracked files no longer expands the index. Update the corresponding test in 't1092'. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	874cf2a604	stash: apply stash using 'merge_ort_nonrecursive()' Update 'stash' to use 'merge_ort_nonrecursive()' to apply a stash to the current working tree. When 'git stash apply' was converted from its shell script implementation to a builtin in `8a0fc8d19d` (stash: convert apply to builtin, 2019-02-25), 'merge_recursive_generic()' was used to merge a stash into the working tree as part of 'git stash (apply\|pop)'. However, with the single merge base used in 'do_apply_stash()', the commit wrapping done by 'merge_recursive_generic()' is not only unnecessary, but misleading (the real merge base is labeled "constructed merge base"). Therefore, a non-recursive merge of the working tree, stashed tree, and stash base tree is more appropriate. There are two options for a non-recursive merge-then-update-worktree function: 'merge_trees()' and 'merge_ort_nonrecursive()'. Use 'merge_ort_nonrecursive()' to align with the default merge strategy used by 'git merge' (`6a5fb96672` (Change default merge backend from recursive to ort, 2021-08-04)) and, because merge-ort does not operate in-place on the index, avoid unnecessary index expansion. Update tests in 't1092' verifying index expansion for 'git stash' accordingly. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	491df5f679	read-cache: set sparsity when index is new When the index read in 'do_read_index()' does not exist on-disk, mark the index "sparse" if the executing command does not require a full index and sparse index is otherwise enabled. Some commands (such as 'git stash -u') implicitly create a new index (when the 'GIT_INDEX_FILE' variable points to a non-existent file) and perform some operation on it. However, when this index is created, it isn't created with the same sparsity settings as the repo index. As a result, while these indexes may be sparse during the operation, they are always expanded before being written to disk. We can avoid that expansion by defaulting the index to "sparse", in which case it will only be expanded if the full index is needed. Note that the function 'set_new_index_sparsity()' is created despite having only a single caller because additional callers will be added in a subsequent patch. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	3a58792ade	stash: integrate with sparse index Enable sparse index in 'git stash' by disabling 'command_requires_full_index'. With sparse index enabled, some subcommands of 'stash' work without expanding the index, e.g., 'git stash', 'git stash list', 'git stash drop', etc. Others ensure the index is expanded either directly (as in the case of 'git stash [pop\|apply]', where the call to 'merge_recursive_generic()' in 'do_apply_stash()' triggers the expansion), or in a command called internally by stash (e.g., 'git update-index' in 'git stash -u'). So, in addition to enabling sparse index, add tests to 't1092' demonstrating which variants of 'git stash' expand the index, and which do not. Finally, add the option to skip writing 'untracked.txt' in 'ensure_not_expanded', and use that option to successfully apply stashed untracked files without a conflict in 'untracked.txt'. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	eae937059b	stash: expand sparse-checkout compatibility testing Add tests verifying expected 'git stash' behavior in 't1092-sparse-checkout-compatibility'. These cases establish the expected behavior of 'git stash' in a sparse-checkout and verify consistency both with and without a sparse index. Although no sparse index compatibility has been integrated into 'git stash' yet, the tests are all 'expect_success' - we don't want the cone-mode sparse-checkout behavior to change depending on whether it is using a sparse index or not. Therefore, we expect these tests to continue passing once sparse index is integrated with 'git stash'. Additionally, add performance test cases for 'git stash' both with and without untracked files. Note that, unlike the other tests in 'p2000-sparse-operations.sh', the tests added for 'stash' are combination operations. This is done to ensure the stash/unstash is not blocked by the modification of '$SPARSE_CONE/a' performed as part of 'test_perf_on_all'. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	124b05b230	rev-parse: integrate with sparse index It is not obvious that the 'git rev-parse' builtin would use the sparse index, but it is possible to parse paths out of the index using the ":<path>" syntax. The 'git rev-parse' output is only the OID of the object found at that location, but otherwise behaves similarly to 'git show :<path>'. This includes the failure conditions on directories and the error messages depending on whether a path is in the worktree or not. The only code change required is to change the command_requires_full_index setting in builtin/rev-parse.c, and we can re-use many existing 'git show' tests for the rev-parse case. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	4925adb4da	object-name: diagnose trees in index properly When running 'git show :<path>' where '<path>' is a directory, then there is a subtle difference between a full checkout and a sparse checkout. The error message from diagnose_invalid_index_path() reports whether the path is on disk or not. The full checkout will have the directory on disk, but the path will not be in the index. The sparse checkout could have the directory not exist, specifically when that directory is outside of the sparse-checkout cone. In the case of a sparse index, we have yet another state: the path can be a sparse directory in the index. In this case, the error message from diagnose_invalid_index_path() would erroneously say "path '<path>' is in the index, but not at stage 0", which is false. Add special casing around sparse directory entries so we get to the correct error message. This requires two checks in order to get parity with the normal sparse-checkout case. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	561287d342	object-name: reject trees found in the index The get_oid_with_context_1() method is used when parsing revision arguments. One particular case is to take a ":<path>" string and search the index for the given path. In the case of a sparse index, this might find a sparse directory entry, in which case the contained object is a tree. In the case of a full index, this search within the index would fail. In order to maintain identical return state as in a full index, inspect the discovered cache entry to see if it is a sparse directory and reject it. This requires being careful around the only_to_die option to be sure we die only at the correct time. This changes the behavior of 'git show :<sparse-dir>', but does not bring it entirely into alignment with a full index case. It specifically hits the wrong error message within diagnose_invalid_index_path(). That error message will be corrected in a future change. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	a37d14422a	show: integrate with the sparse index The 'git show' command can take an input to request the state of an object in the index. This can lead to parsing the index in order to load a specific file entry. Without the change presented here, a sparse index would expand to a full one, taking much longer than usual to access a simple file. There is one behavioral change that happens here, though: we now can find a sparse directory entry within the index! Commands that previously failed because we could not find an entry in the worktree or index now succeed because we _do_ find an entry in the index. There might be more work to do to make other situations succeed when looking for an indexed tree, perhaps by looking at or updating the cache-tree extension as needed. These situations include having a full index or asking for a directory that is within the sparse-checkout cone (and hence is not a sparse directory entry in the index). For now, we demonstrate how the sparse index integration is extremely simple for files outside of the cone as well as directories within the cone. A later change will resolve this behavior around sparse directories. Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	a9e0a49dc4	t1092: add compatibility tests for 'git show' Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	bfc763df77	unpack-trees: increment cache_bottom for sparse directories Correct tracking of the 'cache_bottom' for cases where sparse directories are present in the index. BACKGROUND ---------- The 'unpack_trees_options.cache_bottom' is a variable that tracks the in-progress "bottom" of the cache as 'unpack_trees()' iterates through the contents of the index. Most importantly, this value informs the sequential return values of 'next_cache_entry()' which, in the "diff cache" usage of 'unpack_callback()', are either unpacked as-is or are passed into the diff machinery. The 'cache_bottom' is intended to track the position of the first entry in the index that has not yet been diffed or unpacked. It is advanced in two main ways: either it is incremented when an index entry is marked as "used" (in 'mark_ce_used()'), indicating that it was unpacked or diffed, or when a directory is unpacked, in which case it is increased by an amount equaling the number of index entries inside that tree. In `17a1bb570b` (unpack-trees: preserve cache_bottom, 2021-07-14), it was identified that sparse directories posed a problem to the above 'cache_bottom' advancement logic - because a sparse directory was both an index entry that could be "used" and a directory that can be unpacked, the 'cache_bottom' would be incremented too many times. To solve this problem, the 'mark_ce_used()' advancement of 'cache_bottom' was skipped for sparse directories. INCORRECT CACHE_BOTTOM TRACKING ------------------------------- Skipping the 'cache_bottom' advancement for sparse directories in 'mark_ce_used()' breaks down in two cases: 1. When the 'unpack_trees()' operation is not a "cache diff" (because the directory contents-based incrementing of 'cache_bottom' does not happen). 2. When a cache diff is performed with a pathspec (because 'unpack_index_entry()' will unpack a sparse directory not matched by the pathspec without performing the directory contents-based increment). The former luckily does not appear to affect 'git' behavior, likely because 'cache_bottom' is largely unused (non-"cache diff" 'unpack_trees()' uses 'find_index_entry()' - rather than 'next_cache_entry()' - to find the index entries to unpack). The latter, however, causes 'cache_bottom' to "lag behind" its intended position by an amount equal to the number of sparse directories unpacked so far with 'unpack_index_entry()'. If a repository is structured such that any sparse directories are ordered lexicographically after any pathspec-matching directories, though, this issue won't present any adverse behavior. This was the case with the 't1092-sparse-checkout-compatibility.sh' tests before the addition of the 'before/' sparse directory (ordered before the in-cone 'deep/' directory), therefore sidestepping the issue. Once the 'before/' directory was added, though, 'cache_bottom' began to lag behind its intended position, causing 'next_cache_entry()' to return index entries it had already processed and, ultimately, an incorrect diff. CORRECTING CACHE_BOTTOM ----------------------- The problems observed in 't1092' come from 'cache_bottom' lagging behind in cases where the cache tree-based advancement doesn't occur. To solve this, then, the fix in `17a1bb570b` is "reversed"; rather than skipping 'cache_bottom' advancement in 'mark_ce_used()', we skip the directory contents-based advancement for sparse directories. Now, every index entry can be accounted for in 'cache_bottom': * if you're working with a single index entry, 'cache_bottom' is incremented in 'mark_ce_used()' * if you're working with a directory that contains index entries (but is not one itself), 'cache_bottom' is incremented by the number of entries in that directory. Finally, change the 'test_expect_failure' tests in 't1092' failing due to this bug back to 'test_expect_success'. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	c3a9cecc7f	t1092: add sparse directory before cone in test repo Add a sparse directory 'before/' containing files 'a' and 'b' to the test repo used in 't/t1092-sparse-checkout-compatibility.sh'. This is meant to ensure that no sparse index integrations rely on the in-cone path(s) being lexicographically first in the repo. Unfortunately, some existing tests do not handle this repo architecture properly: * 'add outside sparse cone' * 'status/add: outside sparse cone' * 'reset with pathspecs inside sparse definition' All three of these are due to the incorrect handling of the 'unpack_trees_options.cache_bottom' when performing a cache diff via 'unpack_trees'. This will be corrected in a future patch; in the meantime, mark the tests with 'test_expect_failure'. Finally, update the 'ls-files' and 'root directory cannot be sparse' tests to include the 'before/' directory in their expected index contents. Co-authored-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	f27c170f64	read-tree: make three-way merge sparse-aware Enable use of 'merged_sparse_dir' in 'threeway_merge'. As with two-way merge, the contents of each conflicted sparse directory are merged without referencing the index, avoiding sparse index expansion. As with two-way merge, the 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --merge with edit/edit conflicts in sparse directories' confirms that three-way merges with edit/edit changes (both with and without conflicts) inside a sparse directory result in the correct index state or error message. To ensure the index is not unnecessarily expanded, add three-way merge cases to 'sparse index is not expanded: read-tree'. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	ab81047a6c	read-tree: make two-way merge sparse-aware Enable two-way merge with 'git read-tree' without expanding the sparse index. When in a sparse index, a two-way merge will trivially succeed as long as there are not changes to the same sparse directory in multiple trees (i.e., sparse directory-level "edit-edit" conflicts). If there are such conflicts, the merge will fail despite the possibility that individual files could merge cleanly. In order to resolve these "edit-edit" conflicts, "conflicted" sparse directories are - rather than rejected - merged by traversing their associated trees by OID. For each child of the sparse directory: 1. Files are merged as normal (see Documentation/git-read-tree.txt for details). 2. Subdirectories are treated as sparse directories and merged in 'twoway_merge'. If there are no conflicts, they are merged according to the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory is recursively traversed and merged. This process allows sparse directories to be individually merged at the necessary depth without expanding a full index. The 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --merge with edit/edit conflicts in sparse directories' tests two-way merges with 1) changes inside sparse directories that do not conflict and 2) changes that do conflict (with the correct file(s) reported in the error message). Additionally, add two-way merge cases to 'sparse index is not expanded: read-tree' to confirm that the index is not expanded regardless of whether edit/edit conflicts are present in a sparse directory. Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	7497039241	read-tree: narrow scope of index expansion for '--prefix' When 'git read-tree' is provided with a prefix, expand the index only if the prefix is equivalent to a sparse directory or contained within one. If the index is not expanded in these cases, 'ce_in_traverse_path' will indicate that the relevant sparse directory is not in the prefix/traverse path, skipping past it and not unpacking the appropriate tree(s). If the prefix is in-cone, its sparse subdirectories (if any) will be traversed correctly without index expansion. The behavior of 'git read-tree' with prefixes 1) inside of cone, 2) equal to a sparse directory, and 3) inside a sparse directory are all tested as part of the 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --prefix', ensuring that the sparse index case works the way it did prior to this change as well as matching non-sparse index sparse-checkout. Helped-by: Elijah Newren <newren@gmail.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	2c66a7c8ce	read-tree: integrate with sparse index Enable use of sparse index in 'git read-tree'. The integration in this patch is limited only to usage of 'read-tree' that does not need additional functional changes for the sparse index to behave as expected (i.e., produce the same user-facing results as a non-sparse index sparse-checkout). To ensure no unexpected behavior occurs, the index is explicitly expanded when: * '--no-sparse-checkout' is specified (because it disables sparse-checkout) * '--prefix' is specified (if the prefix is inside a sparse directory, the prefixed tree cannot be properly traversed) * two or more <tree-ish> arguments are specified ('twoway_merge' and 'threeway_merge' do not yet support merging sparse directories) Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	14bf38cfcf	read-tree: expand sparse checkout test coverage Add tests focused on how 'git read-tree' behaves in sparse checkouts. Extra emphasis is placed on interactions with files outside the sparse cone, e.g. merges with out-of-cone conflicts. Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	2c521b0e49	status: fix nested sparse directory diff in sparse index Enable the 'recursive' diff option for the diff executed as part of 'git status'. Without the 'recursive' enabled, 'git status' reports index changes incorrectly when the following conditions were met: * sparse index is enabled * there is a difference between the index and HEAD in a file inside a subdirectory of a sparse directory * the sparse directory index entry is not expanded in-core Because it is not recursive by default, the diff in 'git status' reports changes only at the level of files and directories that are immediate children of a sparse directory, rather than recursing into directories with changes to identify the modified file(s). As a result, 'git status' reports the immediate subdirectory itself as "modified". Example: $ git init $ mkdir -p sparse/sub $ echo test >sparse/sub/foo $ git add . $ git commit -m "commit 1" $ echo somethingelse >sparse/sub/foo $ git add . $ git commit -a -m "commit 2" $ git sparse-checkout set --cone --sparse-index 'sparse' $ git reset --soft HEAD~1 $ git status On branch master You are in a sparse checkout. Changes to be committed: (use "git restore --staged <file>..." to unstage) modified: sparse/sub Enabling the 'recursive' diff option in 'wt_status_collect_changes_index' corrects this issue by allowing the diff to recurse into subdirectories of sparse directories to find modified files. Given the same repository setup as the example above, the corrected result of `git status` is: $ git status On branch master You are in a sparse checkout. Changes to be committed: (use "git restore --staged <file>..." to unstage) modified: sparse/sub/foo Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	287fd17e3a	sparse-index: prevent repo root from becoming sparse Prevent the repository root from being collapsed into a sparse directory by treating an empty path as "inside the sparse-checkout". When collapsing a sparse index (e.g. in 'git sparse-checkout reapply'), the root directory typically could not become a sparse directory due to the presence of in-cone root-level files and directories. However, if no such in-cone files or directories were present, there was no explicit check signaling that the "repository root path" (an empty string, in the case of 'convert_to_sparse(...)') was in-cone, and a sparse directory index entry would be created from the repository root directory. The documentation in Documentation/git-sparse-checkout.txt explicitly states that the files in the root directory are expected to be in-cone for a cone-mode sparse-checkout. Collapsing the root into a sparse directory entry violates that assumption, as sparse directory entries are expected to be outside the sparse cone and have SKIP_WORKTREE enabled. This invalid state in turn causes issues with commands that interact with the index, e.g. 'git status'. Treating an empty (root) path as in-cone prevents the creation of a root sparse directory in 'convert_to_sparse(...)'. Because the repository root is otherwise never compared with sparse patterns (in both cone-mode and non-cone sparse-checkouts), the new check does not cause additional changes to how sparse patterns are applied. Helped-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Elijah Newren	af6a51875a	repo_read_index: clear SKIP_WORKTREE bit from files present in worktree The fix is short (~30 lines), but the description is not. Sorry. There is a set of problems caused by files in what I'll refer to as the "present-despite-SKIP_WORKTREE" state. This commit aims to not just fix these problems, but remove the entire class as a possibility -- for those using sparse checkouts. But first, we need to understand the problems this class presents. A quick outline: * Problems * User facing issues * Problem space complexity * Maintenance and code correctness challenges * SKIP_WORKTREE expectations in Git * Suggested solution * Pros/Cons of suggested solution * Notes on testcase modifications === User facing issues === There are various ways for users to get files to be present in the working copy despite having the SKIP_WORKTREE bit set for that file in the index. This may come from: * various git commands not really supporting the SKIP_WORKTREE bit[1,2] * users grabbing files from elsewhere and writing them to the worktree (perhaps even cached in their editor) * users attempting to "abort" a sparse-checkout operation with a not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the working tree is not atomic)[3]. Once users have present-despite-SKIP_WORKTREE files, any modifications users make to these files will be ignored, possibly to users' confusion. Further: * these files will degrade performance for the sparse-index case due to requiring the index to be expanded (see commit `55dfcf9591` ("sparse-checkout: clear tracked sparse dirs", 2021-09-08) for why we try to delete entire directories outside the sparse cone). * these files will not be updated by by standard commands (switch/checkout/pull/merge/rebase will leave them alone unless conflicts happen -- and even then, the conflicted file may be written somewhere else to avoid overwriting the SKIP_WORKTREE file that is present and in the way) * there is nothing in Git that users can use to discover such files (status, diff, grep, etc. all ignore it) * there is no reasonable mechanism to "recover" from such a condition (neither `git sparse-checkout reapply` nor `git reset --hard` will correct it). So, not only are users modifications ignored, but the files get progressively more stale over time. At some point in the future, they may change their sparseness specification or disable sparse-checkouts. At that time, all present-despite-SKIP_WORKTREE files will show up as having lots of modifications because they represent a version from a different branch or commit. These might include user-made local changes from days before, but the only way to tell is to have users look through them all closely. If these users come to others for help, there will be no logs that explain the issue; it's just a mysterious list of changes. Users might adamantly claim (correctly, as it turns out) that they didn't modify these files, while others presume they did. [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/ [2] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/ [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/ === Problem space complexity === SKIP_WORKTREE has been part of Git for over a decade. Duy did lots of work on it initially, and several others have since come along and put lots of work into it. Stolee spent most of 2021 on the sparse-index, with lots of bugfixes along the way including to non-sparse-index cases as we are still trying to get sparse checkouts to behave reasonably. Basically every codepath throughout the treat needs to be aware of an additional type of file: tracked-but-not-present. The extra type results in lots of extra testcases and lots of extra code everywhere. But, the sad thing is that we actually have more than one extra type. We have tracked, tracked-but-not-present (SKIP_WORKTREE), and tracked-but-promised-to-not-be-present-but-is-present-anyway (present-despite-SKIP_WORKTREE). Two types is a monumental amount of effort to support, and adding a third feels a bit like insanity[4]. [4] Some examples of which can be seen at https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/ === Maintenance and code correctness challenges === Matheus' patches to grep stalled for nearly a year, in part because of complications of how to handle sparse-checkouts appropriately in all cases[5][6] (with trying to sanely figure out how to sanely handle present-despite-SKIP_WORKTREE files being one of the complications). His rm/add follow-ups also took months because of those kinds of issues[7]. The corner cases with things like submodules and SKIP_WORKTREE with the addition of present-despite-SKIP_WORKTREE start becoming really complex[8]. We've had to add ugly logic to merge-ort to attempt to handle present-despite-SKIP_WORKTREE files[9], and basically just been forced to give up in merge-recursive knowing full well that we'll sometimes silently discard user modifications. Despite stash essentially being a merge, it needed extra code (beyond what was in merge-ort and merge-recursive) to manually tweak SKIP_WORKTREE bits in order to avoid a few different bugs that'd result in an early abort with a partial stash application[10]. [5] See https://lore.kernel.org/git/5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br/#t and the dates on the thread; also Matheus and I had several conversations off-list trying to resolve the issues over that time [6] ...it finally kind of got unstuck after https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/ [7] See for example https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/#t and quotes like "The core functionality of sparse-checkout has always been only partially implemented", a statement I still believe is true today. [8] https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/ [9] See commit `66b209b86a` ("merge-ort: implement CE_SKIP_WORKTREE handling with conflicted entries", 2021-03-20) [10] See commit `ba359fd507` ("stash: fix stash application in sparse-checkouts", 2020-12-01) === SKIP_WORKTREE expectations in Git === A couple quotes: * From [11] (before the "sparse-checkout" command existed): If it needs too many special cases, hacks, and conditionals, then it is not worth the complexity---if it is easier to write a correct code by allowing Git to populate working tree files, it is perfectly fine to do so. In a sense, the sparse checkout "feature" itself is a hack by itself, and that is why I think this part should be "best effort" as well. * From the git-sparse-checkout manual (still present today): THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN THE FUTURE. [11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/ === Suggested solution === SKIP_WORKTREE was written to allow sparse-checkouts, in particular, as the name of the option implies, to allow the file to NOT be in the worktree but consider it to be unchanged rather than deleted. The suggests a simple solution: present-despite-SKIP_WORKTREE files should not exist, for those using sparse-checkouts. Enforce this at index loading time by checking if core.sparseCheckout is true; if so, check files in the index with the SKIP_WORKTREE bit set to verify that they are absent from the working tree. If they are present, unset the bit (in memory, though any commands that write to the index will record the update). Users can, of course, can get the SKIP_WORKTREE bit back such as by running `git sparse-checkout reapply` (if they have ensured the file is unmodified and doesn't match the specified sparsity patterns). === Pros/Cons of suggested solution === Pros: * Solves the user visible problems reported above, which I've been complaining about for nearly a year but couldn't find a solution to. * Helps prevent slow performance degradation with a sparse-index. * Much easier behavior in sparse-checkouts for users to reason about * Very simple, ~30 lines of code. * Significantly simplifies some ugly testcases, and obviates the need to test an entire class of potential issues. * Reduces code complexity, reasoning, and maintenance. Avoids disagreements about weird corner cases[12]. * It has been reported that some users might be (ab)using SKIP_WORKTREE as a let-me-modify-but-keep-the-file-in-the-worktree mechanism[13, and a few other similar references]. These users know of multiple caveats and shortcomings in doing so; perhaps not surprising given the "SKIP_WORKTREE expecations" section above. However, these users use `git update-index --skip-worktree`, and not `git sparse-checkout` or core.sparseCheckout=true. As such, these users would be unaffected by this change and can continue abusing the system as before. [12] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/ [13] https://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree Cons: * When core.sparseCheckout is enabled, this adds a performance cost to reading the index. I'll defer discussion of this cost to a subsequent patch, since I have some optimizations to add. === Notes on testcase modifications === The good: * t1011: Compare to two cases above it ('read-tree will not throw away dirty changes, non-sparse'); since the file is present, it should match the non-sparse case now * t1092: sparse-index & sparse-checkout now match full-worktree behavior in more cases! Yaay for consistency! * t6428, t7012: look at how much simpler the tests become! Merge and stash can just fail early telling the user there's a file in the way, instead of not noticing until it's about to write a file and then have to implement sudden crash avoidance. Hurray for sanity! * t7817: sparse behavior better matches full tree behavior. Hurray for sanity! The confusing: * t3705: These changes were ONLY needed on Windows, but they don't hurt other platforms. Let's discuss each individually: * core.sparseCheckout should be false by default. Nothing in this testcase toggles that until many, many tests later. However, early tests (#5 in particular) were testing `update-index --skip-worktree` behavior in a non-sparse-checkout, but the Windows tests in CI were behaving as if core.sparseCheckout=true had been specified somewhere. I do not have access to a Windows machine. But I just manually did what should have been a no-op and turned the config off. And it fixed the test. * I have no idea why the leftover .gitattributes file from this test was causing failures for test #18 on Windows, but only with these changes of mine. Test #18 was checking for empty stderr, and specifically wanted to know that some error completely unrelated to file endings did not appear. The leftover .gitattributes file thus caused some spurious stderr unrelated to the thing being checked. Since other tests did not intend to test normalization, just proactively remove the .gitattributes file. I'm certain this is cleaner and better, I'm just unsure why/how this didn't trigger problems before. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	b9ca5e2657	update-index: reduce scope of index expansion in do_reupdate Replace unconditional index expansion in 'do_reupdate()' with one scoped to only where a full index is needed. A full index is only required in 'do_reupdate()' when a sparse directory in the index differs from HEAD; in that case, the index is expanded and the operation restarted. Because the index should only be expanded if a sparse directory is modified, add a test ensuring the index is not expanded when differences only exist within the sparse cone. Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	c35e9f5ecd	update-index: integrate with sparse index Enable use of the sparse index with `update-index`. Most variations of `update-index` work without explicitly expanding the index or making any other updates in or outside of `update-index.c`. The one usage requiring additional changes is `--cacheinfo`; if a file inside a sparse directory was specified, the index would not be expanded until after the cache tree is invalidated, leading to a mismatch between the index and cache tree. This scenario is handled by rearranging `add_index_entry_with_check`, allowing `index_name_stage_pos` to expand the index before attempting to invalidate the relevant cache tree path, avoiding cache tree/index corruption. Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	e015d4d961	update-index: add tests for sparse-checkout compatibility Introduce tests for a variety of `git update-index` use cases, including performance scenarios. Tests are intended to exercise `update-index` with options that change the commands interaction with the index (e.g., `--again`) and with files/directories inside and outside a sparse checkout cone. Of note is that these tests clearly establish the behavior of `git update-index --add` with untracked, outside-of-cone files. Unlike `git add`, which fails with an error when provided with such files, `update-index` succeeds in adding them to the index. Additionally, the `skip-worktree` flag is not automatically added to the new entry. Although this is pre-existing behavior, there are a couple of reasons to avoid changing it in favor of consistency with e.g. `git add`: * `update-index` is low-level command for modifying the index; while it can perform operations similar to those of `add`, it traditionally has fewer "guardrails" preventing a user from doing something they may not want to do (in this case, adding an outside-of-cone, non-`skip-worktree` file to the index) * `update-index` typically only exits with an error code if it is incapable of performing an operation (e.g., if an internal function call fails); adding a new file outside the sparse checkout definition is still a valid operation, albeit an inadvisable one * `update-index` does not implicitly set flags (e.g., `skip-worktree`) when creating new index entries with `--add`; if flags need to be updated, options like `--[no-]skip-worktree` allow a user to intentionally set them All this to say that, while there are valid reasons to consider changing the treatment of outside-of-cone files in `update-index`, there are also sufficient reasons for leaving it as-is. Co-authored-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	35682ada44	checkout-index: integrate with sparse index Add repository settings to allow usage of the sparse index. When using the `--all` option, sparse directories are ignored by default due to the `skip-worktree` flag, so there is no need to expand the index. If `--ignore-skip-worktree-bits` is specified, the index is expanded in order to check out all files. When checking out individual files, existing behavior in a full index is to exit with an error if a directory is specified (as the directory name will not match an index entry). However, it is possible in a sparse index to match a directory name to a sparse directory index entry, but checking out that sparse directory still results in an error on checkout. To reduce some potential confusion for users, `checkout_file(...)` explicitly exits with an informative error if provided with a sparse directory name. The test corresponding to this scenario verifies the error message, which now differs between sparse index and non-sparse index checkouts. Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	88078f543b	checkout-index: add --ignore-skip-worktree-bits option Update `checkout-index` to no longer refresh files that have the `skip-worktree` bit set, exiting with an error if `skip-worktree` filenames are directly provided to `checkout-index`. The newly-added `--ignore-skip-worktree-bits` option provides a mechanism to replicate the old behavior, checking out all files specified (even those with `skip-worktree` enabled). The ability to toggle whether files should be checked-out based on `skip-worktree` already exists in `git checkout` and `git restore` (both of which have an `--ignore-skip-worktree-bits` option). The change to, by default, ignore `skip-worktree` files is especially helpful for sparse-checkout; it prevents inadvertent creation of files outside the sparse definition on disk and eliminates the need to expand a sparse index when using the `--all` option. Internal usage of `checkout-index` in `git stash` and `git filter-branch` do not make explicit use of files with `skip-worktree` enabled, so `--ignore-skip-worktree-bits` is not added to them. Helped-by: Elijah Newren <newren@gmail.com> Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	b553ef6749	checkout-index: expand sparse checkout compatibility tests Add tests to cover `checkout-index`, with a focus on cases interesting in a sparse checkout (e.g., files specified outside sparse checkout definition). New tests are intended to serve as a baseline for existing and/or expected behavior and performance when integrating `checkout-index` with the sparse index. Note that the test 'checkout-index --all' is marked as 'test_expect_failure', indicating that `update-index --all` will be modified in a subsequent patch to behave as the test expects. Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	1e9e10e048	clean: integrate with sparse index Remove full index requirement for `git clean` and test to ensure the index is not expanded in `git clean`. Add to existing test for `git clean` to verify cleanup of untracked files in sparse directories is consistent between sparse index and non-sparse index checkouts. Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	bb01b26dec	reset: fix validation in sparse index test Update t1092 test 'reset with pathspecs outside sparse definition' to verify index contents. The use of `rev-parse` verifies the contents of HEAD, not the index, providing no real validation of the reset results. Conversely, `ls-files` reports the contents of the index (OIDs, flags, filenames), which are then compared across checkouts to ensure compatible index states. Fixes 741a2c9ffa (reset: expand test coverage for sparse checkouts, 2021-09-27). Signed-off-by: Victoria Dye <vdye@github.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	3a9a6ac5d9	t1092: replace 'read-cache --table' with 'ls-files --sparse' Now that 'git ls-files --sparse' exists, we can use it to verify the state of a sparse index instead of 'test-tool read-cache table'. Replace these usages within t1092-sparse-checkout-compatibility.sh. The important changes are due to the different output format. We need to use the '--stage' output to get a file mode and OID, but it also includes a stage value and drops the object type. This leads to some differences in how we handle looking for specific entries. Some places where we previously looked for no 'tree' entries, we can instead directly compare the output across the repository with a sparse index and the one without. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	78087097b8	ls-files: add --sparse option Existing callers to 'git ls-files' are expecting file names, not directories. It is best to expand a sparse index to show all of the contained files in this case. However, expert users may want to inspect the contents of the index itself including which directories are sparse. Add a --sparse option to allow users to request this information. During testing, I noticed that options such as --modified did not affect the output when the files in question were outside the sparse-checkout definition. Tests are added to document this preexisting behavior and how it remains unchanged with the sparse index and the --sparse option. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	5a4e0547e2	fetch/pull: use the sparse index The 'git fetch' and 'git pull' commands parse the index in order to determine if submodules exist. Without command_requires_full_index=0, this will expand a sparse index, causing slow performance even when there is no new data to fetch. The .gitmodules file will never be inside a sparse directory entry, and even if it was, the index_name_pos() method would expand the sparse index if needed as we search for the path by name. These commands do not iterate over the index, which is the typical thing we are careful about when integrating with the sparse index. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Lessley Dennington	add4c864b6	blame: enable and test the sparse index Enable the sparse index for the 'git blame' command. The index was already not expanded with this command, so the most interesting thing to do is to add tests that verify that 'git blame' behaves correctly when the sparse index is enabled and that its performance improves. More specifically, these cases are: 1. The index is not expanded for 'blame' when given paths in the sparse checkout cone at multiple levels. 2. Performance measurably improves for 'blame' with sparse index when given paths in the sparse checkout cone at multiple levels. The `p2000` tests demonstrate a ~60% execution time reduction when running 'blame' for a file two levels deep and and a ~30% execution time reduction for a file three levels deep. Test before after ---------------------------------------------------------------- 2000.62: git blame f2/f4/a (full-v3) 0.31 0.32 +3.2% 2000.63: git blame f2/f4/a (full-v4) 0.29 0.31 +6.9% 2000.64: git blame f2/f4/a (sparse-v3) 0.55 0.23 -58.2% 2000.65: git blame f2/f4/a (sparse-v4) 0.57 0.23 -59.6% 2000.66: git blame f2/f4/f3/a (full-v3) 0.77 0.85 +10.4% 2000.67: git blame f2/f4/f3/a (full-v4) 0.78 0.81 +3.8% 2000.68: git blame f2/f4/f3/a (sparse-v3) 1.07 0.72 -32.7% 2000.99: git blame f2/f4/f3/a (sparse-v4) 1.05 0.73 -30.5% We do not include paths outside the sparse checkout cone because blame does not support blaming files that are not present in the working directory. This is true in both sparse and full checkouts. Signed-off-by: Lessley Dennington <lessleydennington@gmail.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Lessley Dennington	51ba65b5c3	diff: enable and test the sparse index Enable the sparse index within the 'git diff' command. Its implementation already safely integrates with the sparse index because it shares code with the 'git status' and 'git checkout' commands that were already integrated. For more details see: `d76723ee53` (status: use sparse-index throughout, 2021-07-14) `1ba5f45132` (checkout: stop expanding sparse indexes, 2021-06-29) The most interesting thing to do is to add tests that verify that 'git diff' behaves correctly when the sparse index is enabled. These cases are: 1. The index is not expanded for 'diff' and 'diff --staged' 2. 'diff' and 'diff --staged' behave the same in full checkout, sparse checkout, and sparse index repositories in the following partially-staged scenarios (i.e. the index, HEAD, and working directory differ at a given path): 1. Path is within sparse-checkout cone 2. Path is outside sparse-checkout cone 3. A merge conflict exists for paths outside sparse-checkout cone The `p2000` tests demonstrate a ~44% execution time reduction for 'git diff' and a ~86% execution time reduction for 'git diff --staged' using a sparse index: Test before after ------------------------------------------------------------- 2000.30: git diff (full-v3) 0.33 0.34 +3.0% 2000.31: git diff (full-v4) 0.33 0.35 +6.1% 2000.32: git diff (sparse-v3) 0.53 0.31 -41.5% 2000.33: git diff (sparse-v4) 0.54 0.29 -46.3% 2000.34: git diff --cached (full-v3) 0.07 0.07 +0.0% 2000.35: git diff --cached (full-v4) 0.07 0.08 +14.3% 2000.36: git diff --cached (sparse-v3) 0.28 0.04 -85.7% 2000.37: git diff --cached (sparse-v4) 0.23 0.03 -87.0% Co-authored-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Lessley Dennington <lessleydennington@gmail.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Lessley Dennington	338e2a9acc	diff: replace --staged with --cached in t1092 tests Replace uses of the synonym --staged in t1092 tests with --cached (which is the real and preferred option). This will allow consistency in the new tests to be added with the upcoming change to enable the sparse index for diff. Signed-off-by: Lessley Dennington <lessleydennington@gmail.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	8c5de0d265	unpack-trees: use traverse_path instead of name The sparse_dir_matches_path() method compares a cache entry that is a sparse directory entry against a 'struct traverse_info info' and a 'struct name_entry p' to see if the cache entry has exactly the right name for those other inputs. This method was introduced in `523506d` (unpack-trees: unpack sparse directory entries, 2021-07-14), but included a significant mistake. The path comparisons used 'info->name' instead of 'info->traverse_path'. Since 'info->name' only stores a single tree entry name while 'info->traverse_path' stores the full path from root, this method does not work when 'info' is in a subdirectory of a directory. Replacing the right strings and their corresponding lengths make the method work properly. The previous change included a failing test that exposes this issue. That test now passes. The critical detail is that as we go deep into unpack_trees(), the logic for merging a sparse directory entry with a tree entry during 'git checkout' relies on this sparse_dir_matches_path() in order to avoid calling traverse_trees_recursive() during unpack_callback() in this hunk: if (!is_sparse_directory_entry(src[0], names, info) && traverse_trees_recursive(n, dirmask, mask & ~dirmask, names, info) < 0) { return -1; } For deep paths, the short-circuit never occurred and traverse_trees_recursive() was being called incorrectly and that was causing other strange issues. Specifically, the error message from the now-passing test previously included this: error: Your local changes to the following files would be overwritten by checkout: deep/deeper1/deepest2/a deep/deeper1/deepest3/a Please commit your changes or stash them before you switch branches. Aborting These messages occurred because the 'current' cache entry in twoway_merge() was showing as NULL because the index did not contain entries for the paths contained within the sparse directory entries. We instead had 'oldtree' given as the entry at HEAD and 'newtree' as the entry in the target tree. This led to reject_merge() listing these paths. Now that sparse_dir_matches_path() works the same for deep paths as it does for shallow depths, the rest of the logic kicks in to properly handle modifying the sparse directory entries as designed. Reported-by: Gustave Granroth <gus.gran@gmail.com> Reported-by: Mike Marcelais <michmarc@exchange.microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Derrick Stolee	1b38efc7a0	t1092: add deeper changes during a checkout Extend the repository data in the setup of t1092 to include more directories within two parent directories. This reproduces a bug found by users of the sparse index feature with suitably-complicated sparse-checkout definitions. Add a failing test that fails in its first 'git checkout deepest' run in the sparse index case with this error: error: Your local changes to the following files would be overwritten by checkout: deep/deeper1/deepest2/a deep/deeper1/deepest3/a Please commit your changes or stash them before you switch branches. Aborting The next change will fix this error, and that fix will make it clear why the extra depth is necessary for revealing this bug. The assignment of the sparse-checkout definition to include deep/deeper1/deepest as a sibling directory is important to ensure that deep/deeper1 is not a sparse directory entry, but deep/deeper1/deepest2 is. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Victoria Dye	4d1cfc1351	reset: make --mixed sparse-aware Remove the `ensure_full_index` guard on `read_from_tree` and update `git reset --mixed` to ensure it can use sparse directory index entries wherever possible. Sparse directory entries are reset using `diff_tree_oid`, which requires `change` and `add_remove` functions to process the internal contents of the sparse directory. The `recursive` diff option handles cases in which `reset --mixed` must diff/merge files that are nested multiple levels deep in a sparse directory. The use of pathspecs with `git reset --mixed` introduces scenarios in which internal contents of sparse directories may be matched by the pathspec. In order to reset all files in the repo that may match the pathspec, the following conditions on the pathspec require index expansion before performing the reset: * "magic" pathspecs * wildcard pathspecs that do not match only in-cone files or entire sparse directories * literal pathspecs matching something outside the sparse checkout definition Helped-by: Elijah Newren <newren@gmail.com> Signed-off-by: Victoria Dye <vdye@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago

1 2 3

107 Commits (fdbfac60fd889d4e55244958ce7febd61cb53f9d)