kernel/git - git - PowerEL Git System

Commit Graph

Author	SHA1	Message	Date
Atharva Raykar	b3c5f5cb04	submodule: move core cmd_update() logic to C This patch completes the conversion past the flag parsing of `submodule update` by introducing a helper subcommand called `submodule--helper update`. The behaviour of `submodule update` should remain the same after this patch. Prior to this patch, `submodule update` was implemented by piping the output of `update-clone` (which clones all missing submodules, then prints relevant information for all submodules) into `run-update-procedure` (which reads the information and updates the submodule tree). With `submodule--helper update`, we iterate over the submodules and update the submodule tree in the same process. This reuses most of existing code structure, except that `update_submodule()` now updates the submodule tree (instead of printing submodule information to be consumed by another process). Recursing on a submodule is done by calling a subprocess that launches `submodule--helper update`, with a modified `--recursive-prefix` and `--prefix` parameter. Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	c9911c9358	submodule--helper: teach update_data more options Refactor 'struct update_data' to hold the parsed args needed by "git submodule--helper update" and refactor "update-clone" and "run-update-procedure" (the functions that will be combined to form "update") to use these options. For "run-update-procedure", 'struct update_data' already holds its args, so only arg parsing code needs to be updated. For "update-clone", move its args from 'struct submodule_update_clone' into 'struct update_data', and replace them with a pointer to 'struct update_data'. Its other members hold the submodule iteration state of "update-clone", so those are unchanged. Incidentally, since we reformat the designated initializers of the affected structs, also reformat MODULE_CLONE_DATA_INIT for consistency. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	55b3f12cb5	submodule update: use die_message() Use die_message() to print the "fatal: " prefix instead of doing it in git-submodule.sh and remove a now-unnecessary exit code from "git submodule--helper run-update-procedure". Also, since die_message() adds the newline for us, replace an invocation of die_with_status() with printf + exit invocations that do not add a newline, but are otherwise identical to die_with_status(). Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	c9d2562493	submodule--helper update-clone: check for --filter and --init "git submodule update --filter" also requires the "--init" option. Teach update-clone to do this usage check in C and remove the check from git-submodule.sh. In addition, change update-clone's usage string so that it teaches users about "git submodule update" instead of "git submodule--helper update-clone" (the string is copied from git-submodule.sh). This should be more helpful to users since they don't invoke update-clone directly. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	97cb977c82	submodule--helper: remove ensure-core-worktree Move the logic of "git submodule--helper ensure-core-worktree" into run-update-procedure, and since this makes the ensure-core-worktree command obsolete, remove it. As a result, the order of two operations in git-submodule.sh is reversed: 'set the value of core.worktree' now happens after the call to "git submodule--helper relative-path". This is safe - "relative-path" does not depend on the value of core.worktree. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	29a5e9e1ff	submodule--helper update-clone: learn --init Teach "git submodule--helper update-clone" the --init flag and remove the corresponding shell code. When the `--init` flag is passed to the subcommand, we do not spawn a new subprocess and call `submodule--helper init` on the submodule paths, because the Git machinery is not able to pick up the configuration changes introduced by that init call. So we instead run the `init_submodule_cb()` callback over each submodule in the same process. [1] https://lore.kernel.org/git/CAP8UFD0NCQ5w_3GtT_xHr35i7h8BuLX4UcHNY6VHPGREmDVObA@mail.gmail.com/ Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	1012a5cbc3	submodule--helper run-update-procedure: learn --remote Teach run-update-procedure to handle --remote instead of parsing --remote in git-submodule.sh. As a result, "git submodule--helper [print-default-remote\|remote-branch]" have no more callers, so remove them. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Glen Choo	e441966596	submodule--helper run-update-procedure: remove --suboid Teach run-update-procedure to determine the oid of the submodule's HEAD instead of doing it in git-submodule.sh. Signed-off-by: Glen Choo <chooglen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Josh Steadmon	f05da2b48b	clone, submodule: pass partial clone filters to submodules When cloning a repo with a --filter and with --recurse-submodules enabled, the partial clone filter only applies to the top-level repo. This can lead to unexpected bandwidth and disk usage for projects which include large submodules. For example, a user might wish to make a partial clone of Gerrit and would run: `git clone --recurse-submodules --filter=blob:5k https://gerrit.googlesource.com/gerrit`. However, only the superproject would be a partial clone; all the submodules would have all blobs downloaded regardless of their size. With this change, the same filter can also be applied to submodules, meaning the expected bandwidth and disk savings apply consistently. To avoid changing default behavior, add a new clone flag, `--also-filter-submodules`. When this is set along with `--filter` and `--recurse-submodules`, the filter spec is passed along to git-submodule and git-submodule--helper, such that submodule clones also have the filter applied. This applies the same filter to the superproject and all submodules. Users who need to customize the filter per-submodule would need to clone with `--no-recurse-submodules` and then manually initialize each submodule with the proper filter. Applying filters to submodules should be safe thanks to Jonathan Tan's recent work [1, 2, 3] eliminating the use of alternates as a method of accessing submodule objects, so any submodule object access now triggers a lazy fetch from the submodule's promisor remote if the accessed object is missing. This patch is a reworked version of [4], which was created prior to Jonathan Tan's work. [1]: `8721e2e` (Merge branch 'jt/partial-clone-submodule-1', 2021-07-16) [2]: `11e5d0a` (Merge branch 'jt/grep-wo-submodule-odb-as-alternate', 2021-09-20) [3]: `162a13b` (Merge branch 'jt/no-abuse-alternate-odb-for-submodules', 2021-10-25) [4]: https://lore.kernel.org/git/52bf9d45b8e2b72ff32aa773f2415bf7b2b86da2.1563322192.git.steadmon@google.com/ Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	3 years ago
Ævar Arnfjörð Bjarmason	162410f8a0	git-submodule: remove unused is_zero_oid() function The is_zero_oid() function in git-submodule.sh has not been used since `e83e3333b5` (submodule: port submodule subcommand 'summary' from shell to C, 2020-08-13), so we can remove it. This was the last user of the sane_egrep() function in git-sh-setup.sh. I'm not removing it in case some out-of-tree user relied on it. Per the discussion that can be found upthread of [1]. 1. https://lore.kernel.org/git/87tuiwjfvi.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Atharva Raykar	c51f8f94e5	submodule--helper: run update procedures from C Add a new submodule--helper subcommand `run-update-procedure` that runs the update procedure if the SHA1 of the submodule does not match what the superproject expects. This is an intermediate change that works towards total conversion of `submodule update` from shell to C. Specific error codes are returned so that the shell script calling the subcommand can take a decision on the control flow, and preserve the error messages across subsequent recursive calls of `cmd_update`. This change is more focused on doing a faithful conversion, so for now we are not too concerned with trying to reduce subprocess spawns. Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Atharva Raykar	a6226fd772	submodule--helper: convert the bulk of cmd_add() to C Introduce the 'add' subcommand to `submodule--helper.c` that does all the work 'submodule add' past the parsing of flags. We also remove the constness of the sm_path field of the `add_data` struct. This is needed so that it can be modified by normalize_path_copy(). As with the previous conversions, this is meant to be a faithful conversion with no modification to the behaviour of `submodule add`. Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Mentored-by: Christian Couder <christian.couder@gmail.com> Helped-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Based-on-patch-by: Shourya Shukla <periperidip@gmail.com> Based-on-patch-by: Prathamesh Chavan <pc44800@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Atharva Raykar	a452128a36	submodule--helper: introduce add-config subcommand Add a new "add-config" subcommand to `git submodule--helper` with the goal of converting part of the shell code in git-submodule.sh related to `git submodule add` into C code. This new subcommand sets the configuration variables of a newly added submodule, by registering the url in local git config, as well as the submodule name and path in the .gitmodules file. It also sets 'submodule.<name>.active' to "true" if the submodule path has not already been covered by any pathspec specified in 'submodule.active'. This is meant to be a faithful conversion from shell to C, although we add comments to areas that could be improved in future patches, after the conversion has settled. Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Based-on-patch-by: Shourya Shukla <periperidip@gmail.com> Based-on-patch-by: Prathamesh Chavan <pc44800@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Atharva Raykar	8c8195e9c3	submodule--helper: introduce add-clone subcommand Let's add a new "add-clone" subcommand to `git submodule--helper` with the goal of converting part of the shell code in git-submodule.sh related to `git submodule add` into C code. This new subcommand clones the repository that is to be added, and checks out to the appropriate branch. This is meant to be a faithful conversion that leaves the behaviour of 'cmd_add()' script unchanged. Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Based-on-patch-by: Shourya Shukla <periperidip@gmail.com> Based-on-patch-by: Prathamesh Chavan <pc44800@gmail.com> Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Atharva Raykar	0008d12284	submodule: prefix die messages with 'fatal' The standard `die()` function that is used in C code prefixes all the messages passed to it with 'fatal: '. This does not happen with the `die` used in 'git-submodule.sh'. Let's prefix each of the shell die messages with 'fatal: ' so that when they are converted to C code, the error messages stay the same as before the conversion. Note that the shell version of `die` exits with error code 1, while the C version exits with error code 128. In practice, this does not change any behaviour, as no functionality in 'submodule add' and 'submodule update' relies on the value of the exit code. Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Kaartic Sivaraam	1cf823d8f0	submodule: remove unnecessary `prefix` based option logic Over time when parts of submodule have been ported from shell to builtin, many instances of the submodule helper have been added. Also added with them are some unnecessary option passing logic that are based on the `prefix` shell variable which never gets set in their code flows. On analysis, the only shell functions which have a valid usage for the `prefix` shell variable are: - cmd_update: which is the only function which sets the variable and thus uses it properly - cmd_init: which uses the variable via a call from cmd_update So, remove the unnecessary option parsing logic based on the `prefix` shell variable. Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Nicholas Clark	62af4bdd42	submodule update: silence underlying fetch with "--quiet" Commands such as $ git submodule update --quiet --init --depth=1 involving shallow clones, call the shell function fetch_in_submodule, which in turn invokes git fetch. Pass the --quiet option onward there. Signed-off-by: Nicholas Clark <nick@ccl4.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Jeff King	66d36b94af	submodule: fix fetch_in_submodule logic Commit `1c1518071c` (submodule: use "fetch" logic instead of custom remote discovery, 2020-11-14) rewrote the logic in fetch_in_submodule to do: elif test "$2" -ne "" But this is nonsense in shell: -ne is for numeric comparisons. This should be "=" or more idiomatically: elif test -n "$2" But once we fix that, many tests start failing. Because that commit introduced another problem. The caller that passes 3 arguments looks like this: fetch_in_submodule "$sm_path" $depth "$sha1" Note the unquoted $depth parameter. When it isn't set, the function will see only 2 arguments, and the function has no idea if what it sees in $2 is an option to go on the command line, or a refspec to pass on stdin. In the old code before that commit: fetch_in_submodule () ( sanitize_submodule_env && cd "$1" && - case "$2" in - '') - git fetch ;; - *) - shift - git fetch $(get_default_remote) "$@" ;; - esac we treated those the same, so it didn't matter. But in the new logic (with my fix above): + if test $# -eq 3 + then + echo "$3" \| git fetch --stdin "$2" + elif test -n "$n" + then + git fetch "$2" + else + git fetch + fi we use the number of parameters to distinguish the two. Let's insist that the caller pass an empty string for positional parameter two if they want to have a third parameter after it. But that still leaves one problem. In the --stdin block, we unconditionally pass "$2" to git-fetch, even if it's the empty string. Rather than add another conditional, we can use :+ parameter expansion to include it only if it's non-empty. In fact, we can do the same for the elif, too, simplifying it further. Technically this is overkill, since we know the --depth parameter will not have whitespace (and indeed, most callers do not bother quoting it), but it doesn't hurt for the function to be careful. It's somewhat amazing that no tests were failing. I think what happened is that: - the 3-arg form rarely triggered; any call with a non-empty $depth and a $sha1 would work, but one with an empty $depth would only have 2 arguments - because of the wrong arguments to "test", the shell would complain and exit non-zero. So we never ran the middle conditional at all - that left every call running "git fetch" with no arguments. A well-written test could have detected the distinction here, but in practice omitting --depth just means fetching more commits, and fetching everything (rather than a single sha1) works as long as the commit in question is reachable Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Ævar Arnfjörð Bjarmason	a89a2fbfcc	parse-remote: remove this now-unused library The previous two commits removed the last use of a function in this library, but most of it had been dead code for a while[1][2]. Only the "get_default_remote" function was still being used. Even though we had a manual page for this library it was never intended (or I expect, actually) used outside of git.git. Let's just remove it, if anyone still cares about a function here they can pull them into their own project[3]. 1. Last use of error_on_missing_default_upstream(): `d03ebd411c` ("rebase: remove the rebase.useBuiltin setting", 2019-03-18) 2. Last use of get_remote_merge_branch(): `49eb8d39c7` ("Remove contrib/examples/*", 2018-03-25) 3. https://lore.kernel.org/git/87a6vmhdka.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Ævar Arnfjörð Bjarmason	e63f7b0acb	submodule: remove sh function in favor of helper Remove the now-redundant "get_default_remote" function by converting its last user to the "print-default-remote" helper. As can be seen in `13424764db` ("submodule: port submodule subcommand 'sync' from shell to C", 2018-01-15) this helper is already used internally by the C code for submodule remote name discovery. The "get_default_remote" function in "git-parse-remote.sh" will be removed in a follow-up change. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Ævar Arnfjörð Bjarmason	1c1518071c	submodule: use "fetch" logic instead of custom remote discovery Replace a use of the get_default_remote() function with an invocation of "git fetch" The "fetch" command already has logic to discover the remote for the current branch. However, before it learned to accept a custom refspec and use its idea of the default remote, it wasn't possible to get rid of some equivalent of the "get_default_remote" invocation here. As it turns out the recently added "--stdin" option to fetch[1] gives us a way to do that. Let's use it instead. While I'm at it simplify the "fetch_in_submodule" function. It wasn't necessary to pass "$@" to "fetch" since we'd only ever provide one SHA-1 as an argument in the previous "*" codepath (in addition to "--depth=N"). Rewrite the function to more narrowly reflect its use-case. 1. https://lore.kernel.org/git/87eekwf87n.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Theodore Dubois	3ad0401e9e	submodule update: silence underlying merge/rebase with "--quiet" Commands such as $ git pull --rebase --recurse-submodules --quiet produce non-quiet output from the merge or rebase. Pass the --quiet option down when invoking "rebase" and "merge". Also fix the parsing of git submodule update -v. When `e84c3cf3` (git-submodule.sh: accept verbose flag in cmd_update to be non-quiet, 2018-08-14) taught "git submodule update" to take "--quiet", it apparently did not know how ${GIT_QUIET:+--quiet} works, and reviewers seem to have missed that setting the variable to "0", rather than unsetting it, still results in "--quiet" being passed to underlying commands. Signed-off-by: Theodore Dubois <tbodt@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	4 years ago
Prathamesh Chavan	e83e3333b5	submodule: port submodule subcommand 'summary' from shell to C Convert submodule subcommand 'summary' to a builtin and call it via 'git-submodule.sh'. The shell version had to call $diff_cmd twice, once to find the modified modules cared by the user and then again, with that list of modules to do various operations for computing the summary of those modules. On the other hand, the C version does not need a second call to $diff_cmd since it reuses the module list from the first call to do the aforementioned tasks. In the C version, we use the combination of setting a child process' working directory to the submodule path and then calling 'prepare_submodule_repo_env()' which also sets the 'GIT_DIR' to '.git', so that we can be certain that those spawned processes will not access the superproject's ODB by mistake. A behavioural difference between the C and the shell version is that the shell version outputs two line feeds after the 'git log' output when run outside of the tests while the C version outputs one line feed in any case. The reason for this is that the shell version calls log with '--pretty=format:<fmt>' whose output is followed by two echo calls; 'format' does not have "terminator" semantics like its 'tformat' counterpart. So, the log output is terminated by a newline only when invoked by the user and not when invoked from the scripts. This results in the one & two line feed differences in the shell version. On the other hand, the C version calls log with '--pretty=<fmt>' which is equivalent to '--pretty:tformat:<fmt>' which is then followed by a 'printf("\n")'. Due to its "terminator" semantics the log output is always terminated by newline and hence one line feed in any case. Also, when we try to pass an option-like argument after a non-option argument, for instance: git submodule summary HEAD --foo-bar (or) git submodule summary HEAD --cached That argument would be treated like a path to the submodule for which the user is requesting a summary. So, the option ends up having no effect. Though, passing '--quiet' is an exception to this: git submodule summary HEAD --quiet While 'summary' doesn't support '--quiet', we don't get an output for the above command as '--quiet' is treated as a path which means we get an output only if a submodule whose path is '--quiet' exists. The error message in case of computing a summary for non-existent submodules in the C version is different from that of the shell version. Since the new error message is not marked for translation, change the 'test_i18ngrep' in t7421.4 to 'grep'. Mentored-by: Christian Couder <chriscool@tuxfamily.org> Mentored-by: Stefan Beller <stefanbeller@gmail.com> Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com> Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Prathamesh Chavan <pc44800@gmail.com> Signed-off-by: Shourya Shukla <shouryashukla.oo@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Shourya Shukla	2964d6e5e1	submodule: port subcommand 'set-branch' from shell to C Convert submodule subcommand 'set-branch' to a builtin and call it via 'git-submodule.sh'. Mentored-by: Christian Couder <chriscool@tuxfamily.org> Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com> Helped-by: Denton Liu <liu.denton@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Shourya Shukla <shouryashukla.oo@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Shourya Shukla	6417cf9c21	submodule: port subcommand 'set-url' from shell to C Convert submodule subcommand 'set-url' to a builtin. Port 'set-url' to 'submodule--helper.c' and call the latter via 'git-submodule.sh'. Signed-off-by: Shourya Shukla <shouryashukla.oo@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Li Xuejiang	65d100c4dd	git-submodule.sh: setup uninitialized variables We have an environment variable `jobs=16` defined in our CI system, and this environment makes our build job failed with the following message: error: pathspec '16' did not match any file(s) known to git The pathspec '16' for Git command is from the environment variable "jobs". This is because "git-submodule" command is implemented in shell script, and environment variables may change its behavior. Set values for uninitialized variables, such as "jobs" and "recommend_shallow" will fix this issue. Helped-by: Jiang Xin <worldhello.net@gmail.com> Signed-off-by: Li Xuejiang <xuejiang@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Emily Shaffer	132f600b06	clone: pass --single-branch during --recurse-submodules Previously, performing "git clone --recurse-submodules --single-branch" resulted in submodules cloning all branches even though the superproject cloned only one branch. Pipe --single-branch through the submodule helper framework to make it to 'clone' later on. Signed-off-by: Emily Shaffer <emilyshaffer@google.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Kyle Meyer	c81638541c	submodule add: show 'add --dry-run' stderr when aborting Unless --force is specified, 'submodule add' checks if the destination path is ignored by calling 'git add --dry-run --ignore-missing', and, if that call fails, aborts with a custom "path is ignored" message (a slight variant of what 'git add' shows). Aborting early rather than letting the downstream 'git add' call fail is done so that the command exits before cloning into the destination path. However, in rare cases where the dry-run call fails for a reason other than the path being ignored---for example, due to a preexisting index.lock file---displaying the "ignored path" error message hides the real source of the failure. Instead of displaying the tailored "ignored path" message, let's report the standard error from the dry run to give the caller more accurate information about failures that are not due to an ignored path. For the ignored path case, this leads to the following change in the error message: The following [-path is-]{+paths are+} ignored by one of your .gitignore files: <destination path> Use -f if you really want to add [-it.-]{+them.+} The new phrasing is a bit awkward, because 'submodule add' is only dealing with one destination path. Alternatively, we could continue to use the tailored message when the exit code is 1 (the expected status for a failure due to an ignored path) and relay the standard error for all other non-zero exits. That, however, risks hiding the message of unrelated failures that share an exit code of 1, so it doesn't seem worth doing just to avoid a clunkier, but still clear, error message. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Johannes Schindelin	0060fd1511	clone --recurse-submodules: prevent name squatting on Windows In addition to preventing `.git` from being tracked by Git, on Windows we also have to prevent `git~1` from being tracked, as the default NTFS short name (also known as the "8.3 filename") for the file name `.git` is `git~1`, otherwise it would be possible for malicious repositories to write directly into the `.git/` directory, e.g. a `post-checkout` hook that would then be executed _during_ a recursive clone. When we implemented appropriate protections in `2b4c6efc82` (read-cache: optionally disallow NTFS .git variants, 2014-12-16), we had analyzed carefully that the `.git` directory or file would be guaranteed to be the first directory entry to be written. Otherwise it would be possible e.g. for a file named `..git` to be assigned the short name `git~1` and subsequently, the short name generated for `.git` would be `git~2`. Or `git~3`. Or even `~9999999` (for a detailed explanation of the lengths we have to go to protect `.gitmodules`, see the commit message of `e7cb0b4455` (is_ntfs_dotgit: match other .git files, 2018-05-11)). However, by exploiting two issues (that will be addressed in a related patch series close by), it is currently possible to clone a submodule into a non-empty directory: - On Windows, file names cannot end in a space or a period (for historical reasons: the period separating the base name from the file extension was not actually written to disk, and the base name/file extension was space-padded to the full 8/3 characters, respectively). Helpfully, when creating a directory under the name, say, `sub.`, that trailing period is trimmed automatically and the actual name on disk is `sub`. This means that while Git thinks that the submodule names `sub` and `sub.` are different, they both access `.git/modules/sub/`. - While the backslash character is a valid file name character on Linux, it is not so on Windows. As Git tries to be cross-platform, it therefore allows backslash characters in the file names stored in tree objects. Which means that it is totally possible that a submodule `c` sits next to a file `c\..git`, and on Windows, during recursive clone a file called `..git` will be written into `c/`, of course _before_ the submodule is cloned. Note that the actual exploit is not quite as simple as having a submodule `c` next to a file `c\..git`, as we have to make sure that the directory `.git/modules/b` already exists when the submodule is checked out, otherwise a different code path is taken in `module_clone()` that does _not_ allow a non-empty submodule directory to exist already. Even if we will address both issues nearby (the next commit will disallow backslash characters in tree entries' file names on Windows, and another patch will disallow creating directories/files with trailing spaces or periods), it is a wise idea to defend in depth against this sort of attack vector: when submodules are cloned recursively, we now _require_ the directory to be empty, addressing CVE-2019-1349. Note: the code path we patch is shared with the code path of `git submodule update --init`, which must not expect, in general, that the directory is empty. Hence we have to introduce the new option `--force-init` and hand it all the way down from `git submodule` to the actual `git submodule--helper` process that performs the initial clone. Reported-by: Nicolas Joly <Nicolas.Joly@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>	5 years ago
Denton Liu	26b061007c	submodule: teach set-url subcommand Currently, in the event that a submodule's upstream URL changes, users have to manually alter the URL in the .gitmodules file then run `git submodule sync`. Let's make that process easier. Teach submodule the set-url subcommand which will automatically change the `submodule.$name.url` property in the .gitmodules file and then run `git submodule sync` to complete the process. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Nguyễn Thái Ngọc Duy	a282f5a906	submodule foreach: fix "<command> --quiet" not being respected Robin reported that git submodule foreach --quiet git pull --quiet origin is not really quiet anymore [1]. "git pull" behaves as if --quiet is not given. This happens because parseopt in submodule--helper will try to parse both --quiet options as if they are foreach's options, not git-pull's. The parsed options are removed from the command line. So when we do pull later, we execute just this git pull origin When calling submodule helper, adding "--" in front of "git pull" will stop parseopt for parsing options that do not really belong to submodule--helper foreach. PARSE_OPT_KEEP_UNKNOWN is removed as a safety measure. parseopt should never see unknown options or something has gone wrong. There are also a couple usage string update while I'm looking at them. While at it, I also add "--" to other subcommands that pass "$@" to submodule--helper. "$@" in these cases are paths and less likely to be --something-like-this. But the point still stands, git-submodule has parsed and classified what are options, what are paths. submodule--helper should never consider paths passed by git-submodule to be options even if they look like one. The test case is also contributed by Robin. [1] it should be quiet before `fc1b9243cd` (submodule: port submodule subcommand 'foreach' from shell to C, 2018-05-10) because parseopt can't accidentally eat options then. Reported-by: Robin H. Johnson <robbat2@gentoo.org> Tested-by: Robin H. Johnson <robbat2@gentoo.org> Signed-off-by: Robin H. Johnson <robbat2@gentoo.org> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Kyle Meyer	e13811189b	submodule: refuse to add repository with no commits When the path given to 'git submodule add' is an existing repository that is not in the index, the repository is passed to 'git add'. If this repository doesn't have a commit checked out, we don't get a useful result: there is no subproject OID to track, and any untracked files in the sub-repository are added as blobs in the top-level repository. To avoid getting into this state, abort if the path is a repository that doesn't have a commit checked out. Note that this check must come before the 'git add --dry-run' check because the next commit will make 'git add' fail when given a repository that doesn't have a commit checked out. Signed-off-by: Kyle Meyer <kyle@kyleam.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Denton Liu	b57e8119e6	submodule: teach set-branch subcommand This teaches git-submodule the set-branch subcommand which allows the branch of a submodule to be set through a porcelain command without having to manually manipulate the .gitmodules file. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Jonathan Tan	bd5e567dc7	submodule: explain first attempt failure clearly When cloning with --recurse-submodules a superproject with at least one submodule with HEAD pointing to an unborn branch, the clone goes something like this: Cloning into 'test'... <messages about cloning of superproject> Submodule '<name>' (<uri>) registered for path '<submodule path>' Cloning into '<submodule path>'... fatal: Couldn't find remote ref HEAD Unable to fetch in submodule path '<submodule path>' <messages about fetching with SHA-1> From <uri> * branch <hash> -> FETCH_HEAD Submodule path '<submodule path>': checked out '<hash>' In other words, first, a fetch is done with no hash arguments (that is, a fetch of HEAD) resulting in a "Couldn't find remote ref HEAD" error; then, a fetch is done given a hash, which succeeds. The fetch given a hash was added in `fb43e31f2b` ("submodule: try harder to fetch needed sha1 by direct fetching sha1", 2016-02-24), and the "Unable to fetch..." message was downgraded from a fatal error to a notice in `e30d833671` ("git-submodule.sh: try harder to fetch a submodule", 2018-05-16). This commit improves the notice to be clearer that we are retrying the fetch, and that the previous messages (in particular, the fatal errors from fetch) do not necessarily indicate that the whole command fails. In other words: - If the HEAD-fetch succeeds and we then have the commit we want, git-submodule prints no explanation. - If the HEAD-fetch succeeds and we do not have the commit we want, but the hash-fetch succeeds, git-submodule prints no explanation. - If the HEAD-fetch succeeds and we do not have the commit we want, but the hash-fetch fails, git-submodule prints a fatal error. - If the HEAD-fetch fails, fetch prints a fatal error, and git-submodule informs the user that it will retry by fetching specific commits by hash. - If the hash-fetch then succeeds, git-submodule prints no explanation (besides the ones already printed). - If the HEAD-fetch then fails, git-submodule prints a fatal error. It could be said that we should just eliminate the HEAD-fetch altogether, but that changes some behavior (in particular, some refs that were opportunistically updated would no longer be), so I have left that alone for now. There is an analogous situation with the fetching code in fetch_finish() and surrounding functions. For now, I have added a NEEDSWORK. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Denton Liu	68cabbfda3	submodule: document default behavior submodule's default behavior wasn't documented in both git-submodule.txt and in the usage text of git-submodule. Document the default behavior similar to how git-remote does it. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Sven van Haastregt	0586a438f6	git-submodule.sh: shorten submodule SHA-1s using rev-parse Until now, `git submodule summary` was always emitting 7-character SHA-1s that have a higher chance of being ambiguous for larger repositories. Use `git rev-parse --short` instead, which will determine suitable short SHA-1 lengths. When a submodule hasn't been initialized with "submodule init" or not cloned, `git rev-parse` would not work in it yet; as a fallback, use the original method of cutting at 7 hexdigits. Signed-off-by: Sven van Haastregt <svenvh@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Stefan Beller	5d124f419d	git-submodule: abort if core.worktree could not be set correctly `74d4731da1` (submodule--helper: replace connect-gitdir-workingtree by ensure-core-worktree, 2018-08-13) forgot to exit the submodule operation if the helper could not ensure that core.worktree is set correctly. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Antonio Ospite	76e9bdc437	submodule: support reading .gitmodules when it's not in the working tree When the .gitmodules file is not available in the working tree, try using the content from the index and from the current branch. This covers the case when the file is part of the repository but for some reason it is not checked out, for example because of a sparse checkout. This makes it possible to use at least the 'git submodule' commands which read the gitmodules configuration file without fully populating the working tree. Writing to .gitmodules will still require that the file is checked out, so check for that before calling config_set_in_gitmodules_file_gently. Add a similar check also in git-submodule.sh::cmd_add() to anticipate the eventual failure of the "git submodule add" command when .gitmodules is not safely writeable; this prevents the command from leaving the repository in a spurious state (e.g. the submodule repository was cloned but .gitmodules was not updated because config_set_in_gitmodules_file_gently failed). Moreover, since config_from_gitmodules() now accesses the global object store, it is necessary to protect all code paths which call the function against concurrent access to the global object store. Currently this only happens in builtin/grep.c::grep_submodules(), so call grep_read_lock() before invoking code involving config_from_gitmodules(). Finally, add t7418-submodule-sparse-gitmodules.sh to verify that reading from .gitmodules succeeds and that writing to it fails when the file is not checked out. NOTE: there is one rare case where this new feature does not work properly yet: nested submodules without .gitmodules in their working tree. This has been documented with a warning and a test_expect_failure item in t7814, and in this case the current behavior is not altered: no config is read. Signed-off-by: Antonio Ospite <ao2@ao2.it> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
brian m. carlson	dda6346877	submodule: make zero-oid comparison hash function agnostic With SHA-256, the length of the all-zeros object ID is longer. Add a function to git-submodule.sh to check if a full hex object ID is the all-zeros value, and use it to check the output we're parsing from git diff-files or diff-index. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Antonio Ospite	b2faad44e2	submodule: use the 'submodule--helper config' command Use the 'submodule--helper config' command in git-submodules.sh to avoid referring explicitly to .gitmodules by the hardcoded file path. This makes it possible to access the submodules configuration in a more controlled way. Signed-off-by: Antonio Ospite <ao2@ao2.it> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Jonathan Nieder	f178c13fda	Revert "Merge branch 'sb/submodule-core-worktree'" This reverts commit `7e25437d35`, reversing changes made to `00624d608c`. v2.19.0-rc0~165^2~1 (submodule: ensure core.worktree is set after update, 2018-06-18) assumes an "absorbed" submodule layout, where the submodule's Git directory is in the superproject's .git/modules/ directory and .git in the submodule worktree is a .git file pointing there. In particular, it uses $GIT_DIR/modules/$name to find the submodule to find out whether it already has core.worktree set, and it uses connect_work_tree_and_git_dir if not, resulting in fatal: could not open sub/.git for writing The context behind that patch: v2.19.0-rc0~165^2~2 (submodule: unset core.worktree if no working tree is present, 2018-06-12) unsets core.worktree when running commands like "git checkout --recurse-submodules" to switch to a branch without the submodule. If a user then uses "git checkout --no-recurse-submodules" to switch back to a branch with the submodule and runs "git submodule update", this patch is needed to ensure that commands using the submodule directly are aware of the path to the worktree. It is late in the release cycle, so revert the whole 3-patch series. We can try again later for 2.20. Reported-by: Allan Sandfeld Jensen <allan.jensen@qt.io> Helped-by: Stefan Beller <sbeller@google.com> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Stefan Beller	ee69b2a90c	submodule--helper: introduce new update-module-mode helper This chews off a bit of the shell part of the update command in git-submodule.sh. When writing the C code, keep in mind that the submodule--helper part will go away eventually and we want to have a C function that is able to determine the submodule update strategy, it as a nicety, make determine_submodule_update_strategy accessible for arbitrary repositories. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Stefan Beller	74d4731da1	submodule--helper: replace connect-gitdir-workingtree by ensure-core-worktree `e98317508c` (submodule: ensure core.worktree is set after update, 2018-06-18) was overly aggressive in calling connect_work_tree_and_git_dir as that ensures both the 'core.worktree' configuration is set as well as setting up correct gitlink file pointing at the git directory. We do not need to check for the gitlink in this part of the cmd_update in git-submodule.sh, as the initial call to update-clone will have ensured that. So we can reduce the work to only (check and potentially) set the 'core.worktree' setting. While at it move the check from shell to C as that proves to be useful in a follow up patch, as we do not need the 'name' in shell now. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Stefan Beller	e84c3cf3dc	git-submodule.sh: accept verbose flag in cmd_update to be non-quiet In `a56771a668` (builtin/pull: respect verbosity settings in submodules, 2018-01-25), we made sure to pass on both quiet and verbose flag from builtin/pull.c to the submodule shell script. However git-submodule doesn't understand a verbose flag, which results in a bug when invoking git pull --recurse-submodules -v [...] There are a few different approaches to fix this bug: 1) rewrite 'argv_push_verbosity' or its caller in builtin/pull.c to cap opt_verbosity at 0. Then 'argv_push_verbosity' would only add '-q' if any. 2) Have a flag in 'argv_push_verbosity' that specifies if we allow adding -q or -v (or both). 3) Add -v to git-submodule.sh and make it a no-op (1) seems like a maintenance burden: What if we add code after the submodule operations or move submodule operations higher up, then we have altered the opt_verbosity setting further down the line in builtin/pull.c. (2) seems like it could work reasonably well without more regressions (3) seems easiest to implement as well as actually is a feature with the last-one-wins rule of passing flags to Git commands. Reported-by: Jochen Kühner Signed-off-by: Stefan Beller <sbeller@google.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Stefan Beller	9eca701f69	git-submodule.sh: rename unused variables The 'mode' variable is not used in cmd_update for its original purpose, rename it to 'dummy' as it only serves the purpose to abort quickly documenting this knowledge. The variable 'stage' is also not used any more in cmd_update, so remove it. This went unnoticed as first each function used the commonly used submodule listing, which was converted in `74703a1e4d` (submodule: rewrite `module_list` shell function in C, 2015-09-02). When cmd_update was using its own function starting in `48308681b0` (git submodule update: have a dedicated helper for cloning, 2016-02-29), its removal was missed. A later patch in this series also touches the communication between the submodule helper and git-submodule.sh, but let's have this as a preparatory patch, as it eases the next patch, which stores the raw data instead of the line printed for this communication. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Stefan Beller	ff03d9306c	git-submodule.sh: align error reporting for update mode to use path All other error messages in cmd_update are reporting the submodule based on its path, so let's do that for invalid update modes, too. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Stefan Beller	e98317508c	submodule: ensure core.worktree is set after update Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Jeff King	0383bbb901	submodule-config: verify submodule names as paths Submodule "names" come from the untrusted .gitmodules file, but we blindly append them to $GIT_DIR/modules to create our on-disk repo paths. This means you can do bad things by putting "../" into the name (among other things). Let's sanity-check these names to avoid building a path that can be exploited. There are two main decisions: 1. What should the allowed syntax be? It's tempting to reuse verify_path(), since submodule names typically come from in-repo paths. But there are two reasons not to: a. It's technically more strict than what we need, as we really care only about breaking out of the $GIT_DIR/modules/ hierarchy. E.g., having a submodule named "foo/.git" isn't actually dangerous, and it's possible that somebody has manually given such a funny name. b. Since we'll eventually use this checking logic in fsck to prevent downstream repositories, it should be consistent across platforms. Because verify_path() relies on is_dir_sep(), it wouldn't block "foo\..\bar" on a non-Windows machine. 2. Where should we enforce it? These days most of the .gitmodules reads go through submodule-config.c, so I've put it there in the reading step. That should cover all of the C code. We also construct the name for "git submodule add" inside the git-submodule.sh script. This is probably not a big deal for security since the name is coming from the user anyway, but it would be polite to remind them if the name they pick is invalid (and we need to expose the name-checker to the shell anyway for our test scripts). This patch issues a warning when reading .gitmodules and just ignores the related config entry completely. This will generally end up producing a sensible error, as it works the same as a .gitmodules file which is missing a submodule entry (so "submodule update" will barf, but "git clone --recurse-submodules" will print an error but not abort the clone. There is one minor oddity, which is that we print the warning once per malformed config key (since that's how the config subsystem gives us the entries). So in the new test, for example, the user would see three warnings. That's OK, since the intent is that this case should never come up outside of malicious repositories (and then it might even benefit the user to see the message multiple times). Credit for finding this vulnerability and the proof of concept from which the test script was adapted goes to Etienne Stalmans. Signed-off-by: Jeff King <peff@peff.net>	7 years ago
Casey Fitzpatrick	a0ef29341a	submodule: add --dissociate option to add/update commands Add --dissociate option to add and update commands, both clone helper commands that already have the --reference option --dissociate pairs with. Signed-off-by: Casey Fitzpatrick <kcghost@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Casey Fitzpatrick	6d33e1c282	submodule: add --progress option to add command The '--progress' was introduced in `72c5f88311` (clone: pass --progress decision to recursive submodules, 2016-09-22) to fix the progress reporting of the clone command. Also add the progress option to the 'submodule add' command. The update command already supports the progress flag, but it is not documented. Signed-off-by: Casey Fitzpatrick <kcghost@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago

1 2 3 4 5 ...

410 Commits (2088a0c0cd61ab6c98064e68619e1d931a4619e2)