From 3eeed876a909c45695f2f3a3edd4141be331a3be Mon Sep 17 00:00:00 2001 From: Andrew Carter Date: Tue, 4 Feb 2025 23:38:56 -0600 Subject: [PATCH 01/43] docs: indicate http.sslCertType and sslKeyType 0a01d41ee4 (http: add support for different sslcert and sslkey types., 2023-03-20) added useful SSL config options, but did not document them. Signed-off-by: Andrew Carter Signed-off-by: Junio C Hamano --- Documentation/config/http.txt | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/Documentation/config/http.txt b/Documentation/config/http.txt index 162b33fc52..ed11ddcd02 100644 --- a/Documentation/config/http.txt +++ b/Documentation/config/http.txt @@ -216,6 +216,21 @@ http.sslBackend:: This option is ignored if cURL lacks support for choosing the SSL backend at runtime. +http.sslCertType:: + Type of client certificate used when fetching or pushing over HTTPS. + "PEM", "DER" are supported when using openssl or gnutls backends. "P12" + is supported on "openssl", "schannel", "securetransport", and gnutls 8.11+. + See also libcurl `CURLOPT_SSLCERTTYPE`. Can be overridden by the + `GIT_SSL_CERT_TYPE` environment variable. + +http.sslKeyType:: + Type of client private key used when fetching or pushing over HTTPS. (e.g. + "PEM", "DER", or "ENG"). Only applicable when using "openssl" backend. "DER" + is not supported with openssl. Particularly useful when set to "ENG" for + authenticating with PKCS#11 tokens, with a PKCS#11 URL in sslCert option. + See also libcurl `CURLOPT_SSLKEYTYPE`. Can be overridden by the + `GIT_SSL_KEY_TYPE` environment variable. + http.schannelCheckRevoke:: Used to enforce or disable certificate revocation checks in cURL when http.sslBackend is set to "schannel". Defaults to `true` if From 62898b8f5e86b67dbc8f9092ae9ee0819823a0be Mon Sep 17 00:00:00 2001 From: Usman Akinyemi Date: Mon, 10 Feb 2025 23:40:30 +0530 Subject: [PATCH 02/43] builtin/update-server-info: remove the_repository global variable Remove the_repository global variable in favor of the repository argument that gets passed in "builtin/update-server-info.c". When `-h` is passed to the command outside a Git repository, the `run_builtin()` will call the `cmd_update_server_info()` function with `repo` set to NULL and then early in the function, "parse_options()" call will give the options help and exit, without having to consult much of the configuration file. So it is safe to omit reading the config when `repo` argument the caller gave us is NULL. Mentored-by: Christian Couder Signed-off-by: Usman Akinyemi Signed-off-by: Junio C Hamano --- builtin/update-server-info.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/builtin/update-server-info.c b/builtin/update-server-info.c index 47a3f0bdd9..d7467290a8 100644 --- a/builtin/update-server-info.c +++ b/builtin/update-server-info.c @@ -1,4 +1,3 @@ -#define USE_THE_REPOSITORY_VARIABLE #include "builtin.h" #include "config.h" #include "gettext.h" @@ -13,7 +12,7 @@ static const char * const update_server_info_usage[] = { int cmd_update_server_info(int argc, const char **argv, const char *prefix, - struct repository *repo UNUSED) + struct repository *repo) { int force = 0; struct option options[] = { @@ -21,11 +20,12 @@ int cmd_update_server_info(int argc, OPT_END() }; - git_config(git_default_config, NULL); + if (repo) + repo_config(repo, git_default_config, NULL); argc = parse_options(argc, argv, prefix, options, update_server_info_usage, 0); if (argc > 0) usage_with_options(update_server_info_usage, options); - return !!update_server_info(the_repository, force); + return !!update_server_info(repo, force); } From 832f56f06a5910b2f41a2e0dd8b382b6c709de07 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Tue, 11 Feb 2025 09:20:07 -0800 Subject: [PATCH 03/43] doc: centrally document various ways tospell `true` and `false` We do not seem to centrally document exhaustively ways to spell Boolean values. The description in the Environment Variables of git(1) section assumes that the reader is already familiar with how "Boolean valued configuration variables" are specified, without referring to anything, so there is no way for the readers to find out more. The description of `bool` in the section on "--type " in "git config --help" might be the place to do so, but it is not telling us all that much. The description of Boolean valued placeholders in the pretty formats section of "git log --help" enumerates the possible values with "etc." implying there may be other synonyms; shrink the list of samples and instead refer to the canonical and authoritative source of truth, which now is git-config(1). Signed-off-by: Junio C Hamano --- Documentation/git-config.txt | 4 +++- Documentation/git.txt | 5 +++-- Documentation/pretty-formats.txt | 8 ++++---- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/Documentation/git-config.txt b/Documentation/git-config.txt index 7f81fbbea8..e904c074bf 100644 --- a/Documentation/git-config.txt +++ b/Documentation/git-config.txt @@ -213,7 +213,9 @@ See also <>. + Valid ``'s include: + -- 'bool': canonicalize values as either "true" or "false". +- 'bool': canonicalize values `true`, `yes`,`on`, and positive + numbers as "true", and values `false`, `no`, `off` and `0` as + "false". - 'int': canonicalize values as simple decimal numbers. An optional suffix of 'k', 'm', or 'g' will cause the value to be multiplied by 1024, 1048576, or 1073741824 upon input. diff --git a/Documentation/git.txt b/Documentation/git.txt index 4489e2297a..57295c0413 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -472,8 +472,9 @@ Environment Variables --------------------- Various Git commands pay attention to environment variables and change their behavior. The environment variables marked as "Boolean" take -their values the same way as Boolean valued configuration variables, e.g. -"true", "yes", "on" and positive numbers are taken as "yes". +their values the same way as Boolean valued configuration variables, i.e., +"true", "yes", "on" and positive numbers are taken as "yes", while "false", +"no", "off", and "0" are taken as "no". Here are the variables: diff --git a/Documentation/pretty-formats.txt b/Documentation/pretty-formats.txt index 8ee940b6a4..07475de8c3 100644 --- a/Documentation/pretty-formats.txt +++ b/Documentation/pretty-formats.txt @@ -339,10 +339,10 @@ insert an empty string unless we are traversing reflog entries (e.g., by decoration format if `--decorate` was not already provided on the command line. -The boolean options accept an optional value `[=]`. The values -`true`, `false`, `on`, `off` etc. are all accepted. See the "boolean" -sub-section in "EXAMPLES" in linkgit:git-config[1]. If a boolean -option is given with no value, it's enabled. +The boolean options accept an optional value `[=]`. The +values taken by `--type=bool` git-config[1], like `yes` and `off`, +are all accepted. Giving a boolean option without `=` is +equivalent to giving it with `=true`. If you add a `+` (plus sign) after '%' of a placeholder, a line-feed is inserted immediately before the expansion if and only if the From 45761988ac01b99f9a81ad6ec884bef3c2d8e402 Mon Sep 17 00:00:00 2001 From: Elijah Newren Date: Tue, 11 Feb 2025 21:01:52 +0000 Subject: [PATCH 04/43] doc: clarify the intent of the renormalize option in the merge machinery The -X renormalize (or merge.renormalize config) option is intended to reduce conflicts due to normalization of newer versions of history. It does so by renormalizing files that it is about to do a three-way content merge on. Some folks thought it would renormalize all files throughout the tree, and the previous wording wasn't clear enough to dispell that misconception. Update the docs to make it clear that the merge machinery will only apply renormalization to files which need a three-way content merge. (Technically, the merge machinery also does renormalization on modify/delete conflicts, in order to see if the modification was merely a normalization; if so, it can accept the delete and not report a conflict. But it's not clear that this piece needs to be explained to users, and trying to distinguish it might feel like splitting hairs and overcomplicating the explanation, so we leave it out.) Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/config/merge.txt | 3 ++- Documentation/gitattributes.txt | 4 ++-- Documentation/merge-strategies.txt | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Documentation/config/merge.txt b/Documentation/config/merge.txt index 8851b6cede..5806c1eb46 100644 --- a/Documentation/config/merge.txt +++ b/Documentation/config/merge.txt @@ -69,7 +69,8 @@ merge.renormalize:: Tell Git that canonical representation of files in the repository has changed over time (e.g. earlier commits record text files with CRLF line endings, but recent ones use LF line - endings). In such a repository, Git can convert the data + endings). In such a repository, for each file where a + three-way content merge is needed, Git can convert the data recorded in commits to a canonical form before performing a merge to reduce unnecessary conflicts. For more information, see section "Merging branches with differing checkin/checkout diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt index e6150595af..9ae8117300 100644 --- a/Documentation/gitattributes.txt +++ b/Documentation/gitattributes.txt @@ -701,8 +701,8 @@ where the attribute is not in place would normally cause merge conflicts. To prevent these unnecessary merge conflicts, Git can be told to run a -virtual check-out and check-in of all three stages of a file when -resolving a three-way merge by setting the `merge.renormalize` +virtual check-out and check-in of all three stages of each file that +needs a three-way content merge, by setting the `merge.renormalize` configuration variable. This prevents changes caused by check-in conversion from causing spurious merge conflicts when a converted file is merged with an unconverted file. diff --git a/Documentation/merge-strategies.txt b/Documentation/merge-strategies.txt index 5fc54ec060..a5dc95a378 100644 --- a/Documentation/merge-strategies.txt +++ b/Documentation/merge-strategies.txt @@ -56,7 +56,7 @@ ignore-cr-at-eol;; renormalize;; This runs a virtual check-out and check-in of all three stages - of a file when resolving a three-way merge. This option is + of any file which needs a three-way merge. This option is meant to be used when merging branches with different clean filters or end-of-line normalization rules. See "Merging branches with differing checkin/checkout attributes" in From 9d16f8958467be0162441820d56eb7697453e120 Mon Sep 17 00:00:00 2001 From: David Aguilar Date: Tue, 11 Feb 2025 22:04:13 -0800 Subject: [PATCH 05/43] xdiff: move sign comparison warning guard into each file Allow each file to fix the warnings guarded by the macro separately by moving the definition from the shared xinclude.h into each file that needs it. xmerge.c and xprepare.c do not contain any signed vs. unsigned comparisons so the definition was not included in these files. Signed-off-by: David Aguilar Signed-off-by: Junio C Hamano --- xdiff/xemit.c | 2 ++ xdiff/xhistogram.c | 2 ++ xdiff/xinclude.h | 2 -- xdiff/xpatience.c | 3 +++ xdiff/xutils.c | 2 ++ 5 files changed, 9 insertions(+), 2 deletions(-) diff --git a/xdiff/xemit.c b/xdiff/xemit.c index 75f0fe4986..2b394a4806 100644 --- a/xdiff/xemit.c +++ b/xdiff/xemit.c @@ -20,6 +20,8 @@ * */ +#define DISABLE_SIGN_COMPARE_WARNINGS + #include "xinclude.h" static long xdl_get_rec(xdfile_t *xdf, long ri, char const **rec) { diff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c index 16a8fe2f3f..3d2b190fa6 100644 --- a/xdiff/xhistogram.c +++ b/xdiff/xhistogram.c @@ -41,6 +41,8 @@ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ +#define DISABLE_SIGN_COMPARE_WARNINGS + #include "xinclude.h" #define MAX_PTR UINT_MAX diff --git a/xdiff/xinclude.h b/xdiff/xinclude.h index 7e56542526..a4285ac0eb 100644 --- a/xdiff/xinclude.h +++ b/xdiff/xinclude.h @@ -23,8 +23,6 @@ #if !defined(XINCLUDE_H) #define XINCLUDE_H -#define DISABLE_SIGN_COMPARE_WARNINGS - #include "git-compat-util.h" #include "xmacros.h" #include "xdiff.h" diff --git a/xdiff/xpatience.c b/xdiff/xpatience.c index a2d8955537..b0ba421b28 100644 --- a/xdiff/xpatience.c +++ b/xdiff/xpatience.c @@ -19,6 +19,9 @@ * Davide Libenzi * */ + +#define DISABLE_SIGN_COMPARE_WARNINGS + #include "xinclude.h" /* diff --git a/xdiff/xutils.c b/xdiff/xutils.c index 9e36f24875..169edff5ab 100644 --- a/xdiff/xutils.c +++ b/xdiff/xutils.c @@ -20,6 +20,8 @@ * */ +#define DISABLE_SIGN_COMPARE_WARNINGS + #include "xinclude.h" From 0d31bab479820c1536893f6fbc0dbb1ef1637eb0 Mon Sep 17 00:00:00 2001 From: David Aguilar Date: Tue, 11 Feb 2025 22:04:14 -0800 Subject: [PATCH 06/43] xdiff: avoid signed vs. unsigned comparisons in xdiffi.c The loop iteration variable is non-negative and only used in comparisons against other size_t values. Signed-off-by: David Aguilar Signed-off-by: Junio C Hamano --- xdiff/xdiffi.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c index 4685ba6137..8889b8b62a 100644 --- a/xdiff/xdiffi.c +++ b/xdiff/xdiffi.c @@ -19,7 +19,6 @@ * Davide Libenzi * */ -#define DISABLE_SIGN_COMPARE_WARNINGS #include "xinclude.h" @@ -1014,7 +1013,7 @@ static void xdl_mark_ignorable_lines(xdchange_t *xscr, xdfenv_t *xe, long flags) static int record_matches_regex(xrecord_t *rec, xpparam_t const *xpp) { regmatch_t regmatch; - int i; + size_t i; for (i = 0; i < xpp->ignore_regex_nr; i++) if (!regexec_buf(xpp->ignore_regex[i], rec->ptr, rec->size, 1, From 46fb0843536f2e9281c45042b024098a38236b17 Mon Sep 17 00:00:00 2001 From: David Aguilar Date: Tue, 11 Feb 2025 22:04:15 -0800 Subject: [PATCH 07/43] xdiff: avoid signed vs. unsigned comparisons in xemit.c The unsigned `ignored` variable causes expressions to promote to unsigned. Use a signed value to make comparisons use the same types. Signed-off-by: David Aguilar Signed-off-by: Junio C Hamano --- xdiff/xemit.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/xdiff/xemit.c b/xdiff/xemit.c index 2b394a4806..f8e3f25b03 100644 --- a/xdiff/xemit.c +++ b/xdiff/xemit.c @@ -20,8 +20,6 @@ * */ -#define DISABLE_SIGN_COMPARE_WARNINGS - #include "xinclude.h" static long xdl_get_rec(xdfile_t *xdf, long ri, char const **rec) { @@ -56,7 +54,7 @@ xdchange_t *xdl_get_hunk(xdchange_t **xscr, xdemitconf_t const *xecfg) xdchange_t *xch, *xchp, *lxch; long max_common = 2 * xecfg->ctxlen + xecfg->interhunkctxlen; long max_ignorable = xecfg->ctxlen; - unsigned long ignored = 0; /* number of ignored blank lines */ + long ignored = 0; /* number of ignored blank lines */ /* remove ignorable changes that are too far before other changes */ for (xchp = *xscr; xchp && xchp->ignore; xchp = xchp->next) { From 2dc6cf247e907009950a46c400cbd5efc563a3a2 Mon Sep 17 00:00:00 2001 From: David Aguilar Date: Tue, 11 Feb 2025 22:04:16 -0800 Subject: [PATCH 08/43] xdiff: avoid signed vs. unsigned comparisons in xhistogram.c The comparisons all involve unsigned variables. Cast the comparison to unsigned to eliminate the mismatch. Signed-off-by: David Aguilar Signed-off-by: Junio C Hamano --- xdiff/xhistogram.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/xdiff/xhistogram.c b/xdiff/xhistogram.c index 3d2b190fa6..040d81e0bc 100644 --- a/xdiff/xhistogram.c +++ b/xdiff/xhistogram.c @@ -41,8 +41,6 @@ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ -#define DISABLE_SIGN_COMPARE_WARNINGS - #include "xinclude.h" #define MAX_PTR UINT_MAX @@ -108,7 +106,7 @@ static int scanA(struct histindex *index, int line1, int count1) unsigned int chain_len; struct record **rec_chain, *rec; - for (ptr = LINE_END(1); line1 <= ptr; ptr--) { + for (ptr = LINE_END(1); (unsigned int)line1 <= ptr; ptr--) { tbl_idx = TABLE_HASH(index, 1, ptr); rec_chain = index->records + tbl_idx; rec = *rec_chain; @@ -183,14 +181,14 @@ static int try_lcs(struct histindex *index, struct region *lcs, int b_ptr, be = bs; rc = rec->cnt; - while (line1 < as && line2 < bs + while ((unsigned int)line1 < as && (unsigned int)line2 < bs && CMP(index, 1, as - 1, 2, bs - 1)) { as--; bs--; if (1 < rc) rc = XDL_MIN(rc, CNT(index, as)); } - while (ae < LINE_END(1) && be < LINE_END(2) + while (ae < (unsigned int)LINE_END(1) && be < (unsigned int)LINE_END(2) && CMP(index, 1, ae + 1, 2, be + 1)) { ae++; be++; @@ -315,7 +313,7 @@ redo: if (count1 <= 0 && count2 <= 0) return 0; - if (LINE_END(1) >= MAX_PTR) + if ((unsigned int)LINE_END(1) >= MAX_PTR) return -1; if (!count1) { From 13b67f15c13d2e45215da8950f31ef27645733c3 Mon Sep 17 00:00:00 2001 From: David Aguilar Date: Tue, 11 Feb 2025 22:04:17 -0800 Subject: [PATCH 09/43] xdiff: avoid signed vs. unsigned comparisons in xpatience.c The loop iteration variable is non-negative and used in comparisons against a size_t value. Use size_t to eliminate the mismatch. Signed-off-by: David Aguilar Signed-off-by: Junio C Hamano --- xdiff/xpatience.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/xdiff/xpatience.c b/xdiff/xpatience.c index b0ba421b28..82f663004e 100644 --- a/xdiff/xpatience.c +++ b/xdiff/xpatience.c @@ -20,8 +20,6 @@ * */ -#define DISABLE_SIGN_COMPARE_WARNINGS - #include "xinclude.h" /* @@ -78,7 +76,7 @@ struct hashmap { static int is_anchor(xpparam_t const *xpp, const char *line) { - int i; + size_t i; for (i = 0; i < xpp->anchors_nr; i++) { if (!strncmp(line, xpp->anchors[i], strlen(xpp->anchors[i]))) return 1; From a3b56f5f431d2421b575f329d401361e3196b467 Mon Sep 17 00:00:00 2001 From: David Aguilar Date: Tue, 11 Feb 2025 22:04:18 -0800 Subject: [PATCH 10/43] xdiff: avoid signed vs. unsigned comparisons in xutils.c The comparisons all involve comparisons against unsigned values. Signed-off-by: David Aguilar Signed-off-by: Junio C Hamano --- xdiff/xutils.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/xdiff/xutils.c b/xdiff/xutils.c index 169edff5ab..444a108f87 100644 --- a/xdiff/xutils.c +++ b/xdiff/xutils.c @@ -20,8 +20,6 @@ * */ -#define DISABLE_SIGN_COMPARE_WARNINGS - #include "xinclude.h" @@ -377,7 +375,7 @@ static int xdl_format_hunk_hdr(long s1, long c1, long s2, long c2, nb += 3; if (func && funclen) { buf[nb++] = ' '; - if (funclen > sizeof(buf) - nb - 1) + if ((size_t)funclen > sizeof(buf) - nb - 1) funclen = sizeof(buf) - nb - 1; memcpy(buf + nb, func, funclen); nb += funclen; @@ -439,7 +437,7 @@ void* xdl_alloc_grow_helper(void *p, long nr, long *alloc, size_t size) { void *tmp = NULL; size_t n = ((LONG_MAX - 16) / 2 >= *alloc) ? 2 * *alloc + 16 : LONG_MAX; - if (nr > n) + if ((size_t)nr > n) n = nr; if (SIZE_MAX / size >= n) tmp = xdl_realloc(p, n * size); From f23179924bf4ee0e888cfbe911d9bd472918bcb4 Mon Sep 17 00:00:00 2001 From: Taylor Blau Date: Thu, 13 Feb 2025 15:25:50 -0500 Subject: [PATCH 11/43] Makefile: remove accidental recipe prefix in conditional Back in 728b9ac0c3 (Makefile(s): avoid recipe prefix in conditional statements, 2024-04-08), we prepared our Makefiles for a forthcoming change in upstream Make that would ban the recipe prefix within a conditional statement by replacing tabs (the prefix) with eight spaces. In b9d6f64393 (compat/zlib: allow use of zlib-ng as backend, 2025-01-28), a handful of recipe prefix characters were introduced in a conditional statement ('ifdef ZLIB_NG'), causing 'make' to fail on my system, which uses GNU Make 4.4.90. Remove the recipe prefix characters by replacing them with the same script as is mentioned in 728b9ac0c3. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano --- Makefile | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 1853e6ddfa..82e51a80bd 100644 --- a/Makefile +++ b/Makefile @@ -1690,16 +1690,16 @@ IMAP_SEND_LDFLAGS += $(OPENSSL_LINK) $(OPENSSL_LIBSSL) $(LIB_4_CRYPTO) ifdef ZLIB_NG BASIC_CFLAGS += -DHAVE_ZLIB_NG - ifdef ZLIB_NG_PATH + ifdef ZLIB_NG_PATH BASIC_CFLAGS += -I$(ZLIB_NG_PATH)/include EXTLIBS += $(call libpath_template,$(ZLIB_NG_PATH)/$(lib)) - endif + endif EXTLIBS += -lz-ng else - ifdef ZLIB_PATH + ifdef ZLIB_PATH BASIC_CFLAGS += -I$(ZLIB_PATH)/include EXTLIBS += $(call libpath_template,$(ZLIB_PATH)/$(lib)) - endif + endif EXTLIBS += -lz endif From 5309c1e9fb399c390ed36ef476e91f76f6746fa9 Mon Sep 17 00:00:00 2001 From: Adam Dinwoodie Date: Sat, 15 Feb 2025 21:19:03 +0000 Subject: [PATCH 12/43] Makefile: set default goals in makefiles Explicitly set the default goal at the very top of various makefiles. This is already present in some makefiles, but not all of them. In particular, this corrects a regression introduced in a38edab7c8 (Makefile: generate doc versions via GIT-VERSION-GEN, 2024-12-06). That commit added some config files as build targets for the Documentation directory, and put the target configuration in a sensible place. Unfortunately, that sensible place was above any other build target definitions, meaning the default goal changed to being those configuration files only, rather than the HTML and man page documentation. Signed-off-by: Adam Dinwoodie Helped-by: Junio C Hamano Acked-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- Documentation/Makefile | 5 ++++- contrib/credential/libsecret/Makefile | 3 +++ contrib/credential/osxkeychain/Makefile | 1 + contrib/credential/wincred/Makefile | 3 ++- contrib/diff-highlight/Makefile | 3 ++- contrib/diff-highlight/t/Makefile | 5 ++++- contrib/mw-to-git/Makefile | 5 ++++- contrib/mw-to-git/t/Makefile | 3 ++- contrib/persistent-https/Makefile | 5 ++++- contrib/subtree/t/Makefile | 5 ++++- git-gui/Makefile | 1 + git-gui/po/glossary/Makefile | 3 +++ t/interop/Makefile | 5 ++++- t/perf/Makefile | 5 ++++- templates/Makefile | 5 ++++- 15 files changed, 46 insertions(+), 11 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index 3392e1ce7e..22c476fae5 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -1,3 +1,6 @@ +# The default target of this Makefile is... +all:: + # Import tree-wide shared Makefile behavior and libraries include ../shared.mak @@ -236,7 +239,7 @@ DEFAULT_EDITOR_SQ = $(subst ','\'',$(DEFAULT_EDITOR)) ASCIIDOC_EXTRA += -a 'git-default-editor=$(DEFAULT_EDITOR_SQ)' endif -all: html man +all:: html man html: $(DOC_HTML) diff --git a/contrib/credential/libsecret/Makefile b/contrib/credential/libsecret/Makefile index 3e67552cc5..97ce9c92fb 100644 --- a/contrib/credential/libsecret/Makefile +++ b/contrib/credential/libsecret/Makefile @@ -1,3 +1,6 @@ +# The default target of this Makefile is... +all:: + MAIN:=git-credential-libsecret all:: $(MAIN) diff --git a/contrib/credential/osxkeychain/Makefile b/contrib/credential/osxkeychain/Makefile index 238f5f8c36..0948297e20 100644 --- a/contrib/credential/osxkeychain/Makefile +++ b/contrib/credential/osxkeychain/Makefile @@ -1,3 +1,4 @@ +# The default target of this Makefile is... all:: git-credential-osxkeychain CC = gcc diff --git a/contrib/credential/wincred/Makefile b/contrib/credential/wincred/Makefile index 6e992c0866..5b795fc9fe 100644 --- a/contrib/credential/wincred/Makefile +++ b/contrib/credential/wincred/Makefile @@ -1,4 +1,5 @@ -all: git-credential-wincred.exe +# The default target of this Makefile is... +all:: git-credential-wincred.exe -include ../../../config.mak.autogen -include ../../../config.mak diff --git a/contrib/diff-highlight/Makefile b/contrib/diff-highlight/Makefile index f2be7cc924..33c2ccc9f7 100644 --- a/contrib/diff-highlight/Makefile +++ b/contrib/diff-highlight/Makefile @@ -1,4 +1,5 @@ -all: diff-highlight +# The default target of this Makefile is... +all:: diff-highlight PERL_PATH = /usr/bin/perl -include ../../config.mak diff --git a/contrib/diff-highlight/t/Makefile b/contrib/diff-highlight/t/Makefile index 5ff5275496..2a98541477 100644 --- a/contrib/diff-highlight/t/Makefile +++ b/contrib/diff-highlight/t/Makefile @@ -1,3 +1,6 @@ +# The default target of this Makefile is... +all:: + -include ../../../config.mak.autogen -include ../../../config.mak @@ -6,7 +9,7 @@ SHELL_PATH ?= $(SHELL) SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH)) T = $(wildcard t[0-9][0-9][0-9][0-9]-*.sh) -all: test +all:: test test: $(T) .PHONY: help clean all test $(T) diff --git a/contrib/mw-to-git/Makefile b/contrib/mw-to-git/Makefile index 4e603512a3..497ac434d6 100644 --- a/contrib/mw-to-git/Makefile +++ b/contrib/mw-to-git/Makefile @@ -12,6 +12,9 @@ # # make install +# The default target of this Makefile is... +all:: + GIT_MEDIAWIKI_PM=Git/Mediawiki.pm SCRIPT_PERL=git-remote-mediawiki.perl SCRIPT_PERL+=git-mw.perl @@ -27,7 +30,7 @@ INSTLIBDIR=$(shell $(MAKE) -C $(GIT_ROOT_DIR)/ \ DESTDIR_SQ = $(subst ','\'',$(DESTDIR)) INSTLIBDIR_SQ = $(subst ','\'',$(INSTLIBDIR)) -all: build +all:: build test: all $(MAKE) -C t diff --git a/contrib/mw-to-git/t/Makefile b/contrib/mw-to-git/t/Makefile index f422203fa0..6c9f377caa 100644 --- a/contrib/mw-to-git/t/Makefile +++ b/contrib/mw-to-git/t/Makefile @@ -8,7 +8,8 @@ # ## Test git-remote-mediawiki -all: test +# The default target of this Makefile is... +all:: test -include ../../../config.mak.autogen -include ../../../config.mak diff --git a/contrib/persistent-https/Makefile b/contrib/persistent-https/Makefile index 52b84ba3d4..691737e76b 100644 --- a/contrib/persistent-https/Makefile +++ b/contrib/persistent-https/Makefile @@ -12,10 +12,13 @@ # See the License for the specific language governing permissions and # limitations under the License. +# The default target of this Makefile is... +all:: + BUILD_LABEL=$(shell cut -d" " -f3 ../../GIT-VERSION-FILE) TAR_OUT=$(shell go env GOOS)_$(shell go env GOARCH).tar.gz -all: git-remote-persistent-https git-remote-persistent-https--proxy \ +all:: git-remote-persistent-https git-remote-persistent-https--proxy \ git-remote-persistent-http git-remote-persistent-https--proxy: git-remote-persistent-https diff --git a/contrib/subtree/t/Makefile b/contrib/subtree/t/Makefile index 093399c788..2a85f5ee84 100644 --- a/contrib/subtree/t/Makefile +++ b/contrib/subtree/t/Makefile @@ -3,6 +3,9 @@ # Copyright (c) 2005 Junio C Hamano # +# The default target of this Makefile is... +all:: + -include ../../../config.mak.autogen -include ../../../config.mak @@ -31,7 +34,7 @@ TSVN = $(sort $(wildcard t91[0-9][0-9]-*.sh)) TGITWEB = $(sort $(wildcard t95[0-9][0-9]-*.sh)) THELPERS = $(sort $(filter-out $(T),$(wildcard *.sh))) -all: $(DEFAULT_TEST_TARGET) +all:: $(DEFAULT_TEST_TARGET) test: pre-clean $(TEST_LINT) $(MAKE) aggregate-results-and-cleanup diff --git a/git-gui/Makefile b/git-gui/Makefile index 667c39ed56..6c5a12bc32 100644 --- a/git-gui/Makefile +++ b/git-gui/Makefile @@ -1,3 +1,4 @@ +# The default target of this Makefile is... all:: # Define V=1 to have a more verbose compile. diff --git a/git-gui/po/glossary/Makefile b/git-gui/po/glossary/Makefile index 749aa2e7ec..e656b0d2b0 100644 --- a/git-gui/po/glossary/Makefile +++ b/git-gui/po/glossary/Makefile @@ -1,3 +1,6 @@ +# The default target of this Makefile is... +update-po:: + PO_TEMPLATE = git-gui-glossary.pot ALL_POFILES = $(wildcard *.po) diff --git a/t/interop/Makefile b/t/interop/Makefile index 6911c2915a..4ff4ed0616 100644 --- a/t/interop/Makefile +++ b/t/interop/Makefile @@ -1,3 +1,6 @@ +# The default target of this Makefile is... +all:: + # Import tree-wide shared Makefile behavior and libraries include ../../shared.mak @@ -8,7 +11,7 @@ SHELL_PATH ?= $(SHELL) SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH)) T = $(sort $(wildcard i[0-9][0-9][0-9][0-9]-*.sh)) -all: $(T) +all:: $(T) $(T): @echo "*** $@ ***"; '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS) diff --git a/t/perf/Makefile b/t/perf/Makefile index e4808aebed..9b3090c4ed 100644 --- a/t/perf/Makefile +++ b/t/perf/Makefile @@ -1,10 +1,13 @@ +# The default target of this Makefile is... +all:: + # Import tree-wide shared Makefile behavior and libraries include ../../shared.mak -include ../../config.mak export GIT_TEST_OPTIONS -all: test-lint perf +all:: test-lint perf perf: pre-clean ./run diff --git a/templates/Makefile b/templates/Makefile index bd1e9e30c1..722755338d 100644 --- a/templates/Makefile +++ b/templates/Makefile @@ -1,3 +1,6 @@ +# The default target of this Makefile is... +all:: + # Import tree-wide shared Makefile behavior and libraries include ../shared.mak @@ -23,7 +26,7 @@ PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH)) DESTDIR_SQ = $(subst ','\'',$(DESTDIR)) template_instdir_SQ = $(subst ','\'',$(template_instdir)) -all: boilerplates.made custom +all:: boilerplates.made custom # Put templates that can be copied straight from the source # in a file direc--tory--file in the source. They will be From 044b6f04f23d6c7e3c3750c9829db96b71470874 Mon Sep 17 00:00:00 2001 From: Meet Soni Date: Sat, 15 Feb 2025 14:15:39 +0530 Subject: [PATCH 13/43] refspec: clarify function naming and documentation Rename `match_name_with_pattern()` to `match_refname_with_pattern()` to better reflect its purpose and improve documentation comment clarity. The previous function name and parameter names were inconsistent, making it harder to understand their roles in refspec matching. - Rename parameters: - `key` -> `pattern` (globbing pattern to match) - `name` -> `refname` (refname to check) - `value` -> `replacement` (replacement mapping pattern) Signed-off-by: Meet Soni Signed-off-by: Junio C Hamano --- refspec.c | 34 +++++++++++++++++----------------- refspec.h | 9 +++++---- remote.c | 6 +++--- 3 files changed, 25 insertions(+), 24 deletions(-) diff --git a/refspec.c b/refspec.c index 3d6cf4dc92..4cb80b5208 100644 --- a/refspec.c +++ b/refspec.c @@ -269,28 +269,28 @@ void refspec_ref_prefixes(const struct refspec *rs, } } -int match_name_with_pattern(const char *key, const char *name, - const char *value, char **result) +int match_refname_with_pattern(const char *pattern, const char *refname, + const char *replacement, char **result) { - const char *kstar = strchr(key, '*'); + const char *kstar = strchr(pattern, '*'); size_t klen; size_t ksuffixlen; size_t namelen; int ret; if (!kstar) - die(_("key '%s' of pattern had no '*'"), key); - klen = kstar - key; + die(_("pattern '%s' has no '*'"), pattern); + klen = kstar - pattern; ksuffixlen = strlen(kstar + 1); - namelen = strlen(name); - ret = !strncmp(name, key, klen) && namelen >= klen + ksuffixlen && - !memcmp(name + namelen - ksuffixlen, kstar + 1, ksuffixlen); - if (ret && value) { + namelen = strlen(refname); + ret = !strncmp(refname, pattern, klen) && namelen >= klen + ksuffixlen && + !memcmp(refname + namelen - ksuffixlen, kstar + 1, ksuffixlen); + if (ret && replacement) { struct strbuf sb = STRBUF_INIT; - const char *vstar = strchr(value, '*'); + const char *vstar = strchr(replacement, '*'); if (!vstar) - die(_("value '%s' of pattern has no '*'"), value); - strbuf_add(&sb, value, vstar - value); - strbuf_add(&sb, name + klen, namelen - klen - ksuffixlen); + die(_("replacement '%s' has no '*'"), replacement); + strbuf_add(&sb, replacement, vstar - replacement); + strbuf_add(&sb, refname + klen, namelen - klen - ksuffixlen); strbuf_addstr(&sb, vstar + 1); *result = strbuf_detach(&sb, NULL); } @@ -301,7 +301,7 @@ static int refspec_match(const struct refspec_item *refspec, const char *name) { if (refspec->pattern) - return match_name_with_pattern(refspec->src, name, NULL, NULL); + return match_refname_with_pattern(refspec->src, name, NULL, NULL); return !strcmp(refspec->src, name); } @@ -352,7 +352,7 @@ static int refspec_find_negative_match(struct refspec *rs, struct refspec_item * const char *key = refspec->dst ? refspec->dst : refspec->src; const char *value = refspec->src; - if (match_name_with_pattern(key, needle, value, &expn_name)) + if (match_refname_with_pattern(key, needle, value, &expn_name)) string_list_append_nodup(&reversed, expn_name); } else if (refspec->matching) { /* For the special matching refspec, any query should match */ @@ -397,7 +397,7 @@ void refspec_find_all_matches(struct refspec *rs, if (!refspec->dst || refspec->negative) continue; if (refspec->pattern) { - if (match_name_with_pattern(key, needle, value, result)) + if (match_refname_with_pattern(key, needle, value, result)) string_list_append_nodup(results, *result); } else if (!strcmp(needle, key)) { string_list_append(results, value); @@ -426,7 +426,7 @@ int refspec_find_match(struct refspec *rs, struct refspec_item *query) if (!refspec->dst || refspec->negative) continue; if (refspec->pattern) { - if (match_name_with_pattern(key, needle, value, result)) { + if (match_refname_with_pattern(key, needle, value, result)) { query->force = refspec->force; return 0; } diff --git a/refspec.h b/refspec.h index f62f83a7ee..e2b5cc54ef 100644 --- a/refspec.h +++ b/refspec.h @@ -75,11 +75,12 @@ void refspec_ref_prefixes(const struct refspec *rs, int refname_matches_negative_refspec_item(const char *refname, struct refspec *rs); /* - * Checks whether a name matches a pattern and optionally generates a result. - * Returns 1 if the name matches the pattern, 0 otherwise. + * Checks if a refname matches a globbing refspec pattern. + * If replacement is provided, computes the corresponding mapped refname. + * Returns 1 if refname matches pattern, 0 otherwise. */ -int match_name_with_pattern(const char *key, const char *name, - const char *value, char **result); +int match_refname_with_pattern(const char *pattern, const char *refname, + const char *replacement, char **result); /* * Queries a refspec for a match and updates the query item. diff --git a/remote.c b/remote.c index c1cf363cca..c9c1384c1d 100644 --- a/remote.c +++ b/remote.c @@ -1322,9 +1322,9 @@ static char *get_ref_match(const struct refspec *rs, const struct ref *ref, const char *dst_side = item->dst ? item->dst : item->src; int match; if (direction == FROM_SRC) - match = match_name_with_pattern(item->src, ref->name, dst_side, &name); + match = match_refname_with_pattern(item->src, ref->name, dst_side, &name); else - match = match_name_with_pattern(dst_side, ref->name, item->src, &name); + match = match_refname_with_pattern(dst_side, ref->name, item->src, &name); if (match) { matching_refs = i; break; @@ -1942,7 +1942,7 @@ static struct ref *get_expanded_map(const struct ref *remote_refs, if (strchr(ref->name, '^')) continue; /* a dereference item */ - if (match_name_with_pattern(refspec->src, ref->name, + if (match_refname_with_pattern(refspec->src, ref->name, refspec->dst, &expn_name) && !ignore_symref_update(expn_name, &scratch)) { struct ref *cpy = copy_ref(ref); From 344a107b557604d3f958e7bf7ebf0901290c50d5 Mon Sep 17 00:00:00 2001 From: Phillip Wood Date: Tue, 18 Feb 2025 16:24:35 +0000 Subject: [PATCH 14/43] merge-tree --stdin: flush stdout to avoid deadlock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit If a process tries to read the output from "git merge-tree --stdin" before it closes merge-tree's stdin then it deadlocks. This happens because merge-tree does not flush its output before trying to read another line of input and means that it is not possible to cherry-pick a sequence of commits using "git merge-tree --stdin". Fix this by calling maybe_flush_or_die() before trying to read the next line of input. Flushing the output after each merge does not seem to affect the performance, any difference is lost in the noise even after increasing the number of runs. $ git rev-list --merges --parents -n100 origin/master | sed 's/^[^ ]* //' >/tmp/merges $ hyperfine -L flush 0,1 --warmup 1 --runs 30 \ 'GIT_FLUSH={flush} ./git merge-tree --stdin Acked-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/merge-tree.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builtin/merge-tree.c b/builtin/merge-tree.c index 9a6c8b4e4c..57f4340fab 100644 --- a/builtin/merge-tree.c +++ b/builtin/merge-tree.c @@ -18,6 +18,7 @@ #include "tree.h" #include "config.h" #include "strvec.h" +#include "write-or-die.h" static int line_termination = '\n'; @@ -623,6 +624,7 @@ int cmd_merge_tree(int argc, } else { die(_("malformed input line: '%s'."), buf.buf); } + maybe_flush_or_die(stdout, "stdout"); if (result < 0) die(_("merging cannot continue; got unclean result of %d"), result); From 1b0e5f4499a0c099d99b00a2a6a3edb45ae98660 Mon Sep 17 00:00:00 2001 From: Phillip Wood Date: Tue, 18 Feb 2025 16:24:36 +0000 Subject: [PATCH 15/43] merge-tree: remove redundant code real_merge() only ever returns "0" or "1" as it dies if the merge status is less than zero. Therefore the check for "result < 0" is redundant and the result variable is not needed. The return value of real_merge() is ignored because exit status of "git merge-tree --stdin" is "0" for both successful and conflicted merges (the status of each merge is written to stdout). The return type of real_merge() is not changed as it is used for the program's exit status when "--stdin" is not given. Signed-off-by: Phillip Wood Acked-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/merge-tree.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/builtin/merge-tree.c b/builtin/merge-tree.c index 57f4340fab..3c73482f2b 100644 --- a/builtin/merge-tree.c +++ b/builtin/merge-tree.c @@ -601,7 +601,6 @@ int cmd_merge_tree(int argc, line_termination = '\0'; while (strbuf_getline_lf(&buf, stdin) != EOF) { struct strbuf **split; - int result; const char *input_merge_base = NULL; split = strbuf_split(&buf, ' '); @@ -618,16 +617,14 @@ int cmd_merge_tree(int argc, if (input_merge_base && split[2] && split[3] && !split[4]) { strbuf_rtrim(split[2]); strbuf_rtrim(split[3]); - result = real_merge(&o, input_merge_base, split[2]->buf, split[3]->buf, prefix); + real_merge(&o, input_merge_base, split[2]->buf, split[3]->buf, prefix); } else if (!input_merge_base && !split[2]) { - result = real_merge(&o, NULL, split[0]->buf, split[1]->buf, prefix); + real_merge(&o, NULL, split[0]->buf, split[1]->buf, prefix); } else { die(_("malformed input line: '%s'."), buf.buf); } maybe_flush_or_die(stdout, "stdout"); - if (result < 0) - die(_("merging cannot continue; got unclean result of %d"), result); strbuf_list_free(split); } strbuf_release(&buf); From 54cf5d2da897d4ca2ed4872bd9b7e48851e5573e Mon Sep 17 00:00:00 2001 From: Phillip Wood Date: Tue, 18 Feb 2025 16:24:37 +0000 Subject: [PATCH 16/43] merge-tree: only use basic merge config Commit 9c93ba4d0ae (merge-recursive: honor diff.algorithm, 2024-07-13) replaced init_merge_options() with init_basic_merge_config() for use in plumbing commands and init_ui_merge_config() for use in porcelain commands. As "git merge-tree" is a plumbing command it should call init_basic_merge_config() rather than init_ui_merge_config(). The merge ort machinery ignores "diff.algorithm" so the behavior is unchanged by this commit but it future proofs us against any future changes to init_ui_merge_config(). Signed-off-by: Phillip Wood Acked-by: Elijah Newren Signed-off-by: Junio C Hamano --- builtin/merge-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builtin/merge-tree.c b/builtin/merge-tree.c index 3c73482f2b..3ec7127b3a 100644 --- a/builtin/merge-tree.c +++ b/builtin/merge-tree.c @@ -576,7 +576,7 @@ int cmd_merge_tree(int argc, }; /* Init merge options */ - init_ui_merge_options(&o.merge_options, the_repository); + init_basic_merge_options(&o.merge_options, the_repository); /* Parse arguments */ original_argc = argc - 1; /* ignoring argv[0] */ From 3e681a7ccc97b81f9c93e5b4ca6d3a85d9817285 Mon Sep 17 00:00:00 2001 From: Phillip Wood Date: Tue, 18 Feb 2025 16:24:38 +0000 Subject: [PATCH 17/43] merge-tree: improve docs for --stdin Add a section for --stdin in the list of options and document that it implies -z so readers know how to parse the output. Also correct the merge status documentation for --stdin as if the status is less than zero "git merge-tree" dies before printing it. Signed-off-by: Phillip Wood Acked-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/git-merge-tree.txt | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/Documentation/git-merge-tree.txt b/Documentation/git-merge-tree.txt index 0b6a8a19b1..efb16b4f27 100644 --- a/Documentation/git-merge-tree.txt +++ b/Documentation/git-merge-tree.txt @@ -40,6 +40,11 @@ After the merge completes, a new toplevel tree object is created. See OPTIONS ------- +--stdin:: + Read the commits to merge from the standard input rather than + the command-line. See <> below for more + information. Implies `-z`. + -z:: Do not quote filenames in the section, and end each filename with a NUL character rather than @@ -116,8 +121,6 @@ This is an integer status followed by a NUL character. The integer status is: 0: merge had conflicts 1: merge was clean - <0: something prevented the merge from running (e.g. access to repository - objects denied by filesystem) [[OIDTLT]] OID of toplevel tree @@ -235,6 +238,7 @@ with linkgit:git-merge[1]: * any messages that would have been printed to stdout (the <>) +[[INPUT]] INPUT FORMAT ------------ 'git merge-tree --stdin' input format is fully text based. Each line From 6a9ae8101525360b8f79ed20d2f483616bd39c90 Mon Sep 17 00:00:00 2001 From: Phillip Wood Date: Tue, 18 Feb 2025 16:24:39 +0000 Subject: [PATCH 18/43] merge-tree: fix link formatting in html docs In the html documentation the link to the "OUTPUT" section is surrounded by square brackets. Fix this by adding explicit link text to the cross reference. Signed-off-by: Phillip Wood Acked-by: Elijah Newren Signed-off-by: Junio C Hamano --- Documentation/git-merge-tree.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Documentation/git-merge-tree.txt b/Documentation/git-merge-tree.txt index efb16b4f27..cf0578f9b5 100644 --- a/Documentation/git-merge-tree.txt +++ b/Documentation/git-merge-tree.txt @@ -49,7 +49,8 @@ OPTIONS Do not quote filenames in the section, and end each filename with a NUL character rather than newline. Also begin the messages section with a NUL character - instead of a newline. See <> below for more information. + instead of a newline. See <> below for more + information. --name-only:: In the Conflicted file info section, instead of writing a list From c2d96bc42cfc358178078d8c2c65d550671d3ed0 Mon Sep 17 00:00:00 2001 From: M Hickford Date: Sun, 16 Feb 2025 21:02:41 +0000 Subject: [PATCH 19/43] doc: use 'title' consistently The first line of a commit message is variously called 'title' or 'subject'. Prefer 'title' unless discussing email. Signed-off-by: M Hickford Signed-off-by: Junio C Hamano --- Documentation/git-commit.txt | 8 ++++---- Documentation/git-rebase.txt | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/git-commit.txt b/Documentation/git-commit.txt index 602e2f1200..70a5e4a181 100644 --- a/Documentation/git-commit.txt +++ b/Documentation/git-commit.txt @@ -98,8 +98,8 @@ OPTIONS replaces the log message of __ with its own log message but makes no changes to the content of __. + -The commit created by plain `--fixup=` has a subject -composed of "fixup!" followed by the subject line from __, +The commit created by plain `--fixup=` has a title +composed of "fixup!" followed by the title of __, and is recognized specially by `git rebase --autosquash`. The `-m` option may be used to supplement the log message of the created commit, but the additional commentary will be thrown away once the @@ -107,7 +107,7 @@ commit, but the additional commentary will be thrown away once the `git rebase --autosquash`. + The commit created by `--fixup=amend:` is similar but its -subject is instead prefixed with "amend!". The log message of +title is instead prefixed with "amend!". The log message of __ is copied into the log message of the "amend!" commit and opened in an editor so it can be refined. When `git rebase --autosquash` squashes the "amend!" commit into __, the @@ -128,7 +128,7 @@ See linkgit:git-rebase[1] for details. `--squash=`:: Construct a commit message for use with `git rebase --autosquash`. - The commit message subject line is taken from the specified + The commit message title is taken from the specified commit with a prefix of "squash! ". Can be used with additional commit message options (`-m`/`-c`/`-C`/`-F`). See linkgit:git-rebase[1] for details. diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt index b18cdbc023..8dc5938b51 100644 --- a/Documentation/git-rebase.txt +++ b/Documentation/git-rebase.txt @@ -599,11 +599,11 @@ See also INCOMPATIBLE OPTIONS below. --no-autosquash:: Automatically squash commits with specially formatted messages into previous commits being rebased. If a commit message starts with - "squash! ", "fixup! " or "amend! ", the remainder of the subject line + "squash! ", "fixup! " or "amend! ", the remainder of the title is taken as a commit specifier, which matches a previous commit if it - matches the subject line or the hash of that commit. If no commit + matches the title or the hash of that commit. If no commit matches fully, matches of the specifier with the start of commit - subjects are considered. + titles are considered. + In the rebase todo list, the actions of squash, fixup and amend commits are changed from `pick` to `squash`, `fixup` or `fixup -C`, respectively, and they @@ -613,7 +613,7 @@ be used to review and edit the todo list before proceeding. The recommended way to create commits with squash markers is by using the `--squash`, `--fixup`, `--fixup=amend:` or `--fixup=reword:` options of linkgit:git-commit[1], which take the target commit as an argument and -automatically fill in the subject line of the new commit from that. +automatically fill in the title of the new commit from that. + Setting configuration variable `rebase.autoSquash` to true enables auto-squashing by default for interactive rebase. The `--no-autosquash` From a620046b29d3a9b8a0337c0396441c26ac84ebe9 Mon Sep 17 00:00:00 2001 From: "brian m. carlson" Date: Mon, 17 Feb 2025 17:57:59 +0000 Subject: [PATCH 20/43] diff: don't crash with empty argument to -G or -S The pickaxe options, -G and -S, need either a regex or a string to look through the history for. An empty value isn't very useful since it would either match everything or nothing, and what's worse, we presently crash with a BUG like so when the user provides one: BUG: diffcore-pickaxe.c:241: should have needle under -G or -S Since it's not very nice of us to crash and this wouldn't do anything useful anyway, let's simply inform the user that they must provide a non-empty argument and exit with an error if they provide an empty one instead. Reported-by: Jared Van Bortel Signed-off-by: brian m. carlson Acked-by: Elijah Newren Signed-off-by: Junio C Hamano --- diff.c | 4 ++++ t/t4209-log-pickaxe.sh | 16 ++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/diff.c b/diff.c index d28b4114c8..5a26300ec4 100644 --- a/diff.c +++ b/diff.c @@ -5493,6 +5493,8 @@ static int diff_opt_pickaxe_regex(const struct option *opt, BUG_ON_OPT_NEG(unset); options->pickaxe = arg; options->pickaxe_opts |= DIFF_PICKAXE_KIND_G; + if (arg && !*arg) + return error(_("-G requires a non-empty argument")); return 0; } @@ -5504,6 +5506,8 @@ static int diff_opt_pickaxe_string(const struct option *opt, BUG_ON_OPT_NEG(unset); options->pickaxe = arg; options->pickaxe_opts |= DIFF_PICKAXE_KIND_S; + if (arg && !*arg) + return error(_("-S requires a non-empty argument")); return 0; } diff --git a/t/t4209-log-pickaxe.sh b/t/t4209-log-pickaxe.sh index a675ace081..0e2f80a268 100755 --- a/t/t4209-log-pickaxe.sh +++ b/t/t4209-log-pickaxe.sh @@ -93,6 +93,22 @@ test_expect_success 'usage: --no-pickaxe-regex' ' test_cmp expect actual ' +test_expect_success 'usage: -G and -S with empty argument' ' + cat >expect <<-\EOF && + error: -S requires a non-empty argument + EOF + + test_expect_code 129 git log -S "" 2>actual && + test_cmp expect actual && + + cat >expect <<-\EOF && + error: -G requires a non-empty argument + EOF + + test_expect_code 129 git log -G "" 2>actual && + test_cmp expect actual +' + test_log expect_initial --grep initial test_log expect_nomatch --grep InItial test_log_icase expect_initial --grep InItial From d874d378379f4a2e7fa0a8ab52ff6e407f4da343 Mon Sep 17 00:00:00 2001 From: Peter Oliver Date: Tue, 18 Feb 2025 15:30:42 +0000 Subject: [PATCH 21/43] meson: bump minimum required Perl version to 5.26.0 Commit 702d8c1f3b (Require Perl 5.26.0, 2024-10-23) dropped support for Perl versions older than 5.26.0. The Meson build system, which has been developed in parallel to that commit, hasn't been bumped accordingly and thus still requires Perl 5.8.1 or newer. Fix this by requiring Perl 5.26.0 or newer with Meson. Signed-off-by: Peter Oliver Reviewed-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meson.build b/meson.build index 0df3872c6a..2ffea06aa7 100644 --- a/meson.build +++ b/meson.build @@ -777,7 +777,7 @@ endif # Note that we only set NO_PERL if the Perl features were disabled by the user. # It may not be set when we have found Perl, but only use it to run tests. -perl = find_program('perl', version: '>=5.8.1', dirs: program_path, required: perl_required) +perl = find_program('perl', version: '>=5.26.0', dirs: program_path, required: perl_required) perl_features_enabled = perl.found() and get_option('perl').allowed() if perl_features_enabled build_options_config.set('NO_PERL', '') From 0bf8d1b3954920eb6d9304d187af18fea5f318fd Mon Sep 17 00:00:00 2001 From: Peter Oliver Date: Tue, 18 Feb 2025 15:30:43 +0000 Subject: [PATCH 22/43] meson: fix Perl version check for Meson versions before 1.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Command `perl --version` says, e.g., “This is perl 5, version 26, subversion 0 (v5.26.0)”, which older versions of Meson interpret as version 26. This will be fixed in Meson 1.7.0, but at the time of writing that isn’t yet released. If we run `perl -V:version` we get the unambiguous response “version='5.26.0';”, but we need at least Meson 1.5.0 to be able to do that. Note that Perl are seriously considering dropping the leading 5 entirely in the near future (https://perl.github.io/PPCs/ppc0025-perl-version/), but that shouldn’t affect us. Signed-off-by: Peter Oliver Co-authored-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- meson.build | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/meson.build b/meson.build index 2ffea06aa7..9c5f593771 100644 --- a/meson.build +++ b/meson.build @@ -777,7 +777,22 @@ endif # Note that we only set NO_PERL if the Perl features were disabled by the user. # It may not be set when we have found Perl, but only use it to run tests. -perl = find_program('perl', version: '>=5.26.0', dirs: program_path, required: perl_required) +# +# At the time of writing, executing `perl --version` results in a string +# similar to the following output: +# +# This is perl 5, version 40, subversion 0 (v5.40.0) built for x86_64-linux-thread-multi +# +# Meson picks up the "40" as version number instead of using "v5.40.0" +# due to the regular expression it uses. This got fixed in Meson 1.7.0, +# but meanwhile we have to either use `-V:version` instead of `--version`, +# which we can do starting with Meson 1.5.0 and newer, or we have to +# match against the minor version. +if meson.version().version_compare('>=1.5.0') + perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=5.26.0', version_argument: '-V:version') +else + perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=26') +endif perl_features_enabled = perl.found() and get_option('perl').allowed() if perl_features_enabled build_options_config.set('NO_PERL', '') From 975fc0471abf12cd3fdf8ce3220d2388aca7f759 Mon Sep 17 00:00:00 2001 From: Eliah Kagan Date: Fri, 21 Feb 2025 12:01:36 +0000 Subject: [PATCH 23/43] compat/mingw: rename the symlink, not the target Since 183ea3ea (Merge branch 'ps/mingw-rename', 2024-11-13), a new technique is used on Windows to rename files, where supported. The first step of this technique is to open the file with `CreateFileW`. At that time, `FILE_ATTRIBUTE_NORMAL` was passed as the value of the `dwFlagsAndAttributes` argument. In b30404df [2], this was improved by passing `FILE_FLAG_BACKUP_SEMANTICS`, to support directories as well as regular files. However, neither value of `dwFlagsAndAttributes` is sufficient to open a symbolic link with the correct semantics to rename it. Symlinks on Windows are reparse points. Attempting to open a reparse point with `CreateFileW` dereferences the reparse point and opens the target instead, unless `FILE_FLAG_OPEN_REPARSE_POINT` is included in `dwFlagsAndAttributes`. This is documented for that flag and in the "Symbolic Link Behavior" section of the `CreateFileW` docs [3]. This produces a regression where attempting to rename a symlink on Windows renames its target to the intended new name and location of the symlink. For example, if `symlink` points to `file`, then running git mv symlink symlink-renamed leaves `symlink` in place and unchanged, but renames `file` to `symlink-renamed` [4]. This regression is detectable by existing tests in `t7001-mv.sh`, but the tests must be run by a Windows user with the ability to create symlinks, and the `ln -s` command used to create the initial symlink must also be able to create a real symlink (such as by setting the `MSYS` environment variable to `winsymlinks:nativestrict`). Then these two tests fail if the regression is present, and pass otherwise: 38 - git mv should overwrite file with a symlink 39 - check moved symlink Let's fix this, so that renaming a symlink again renames the symlink itself and leaves the target unchanged, by passing FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT as the `dwFlagsAndAttributes` argument. This is sufficient (and safe) because including `FILE_FLAG_OPEN_REPARSE_POINT` causes no harm even when used to open a file or directory that is not a reparse point. In that case, as noted in [3], this flag is simply ignored. [1]: https://github.com/git-for-windows/git/commit/183ea3eabf81822506d2cd3aa1dc0727099ebccd [2]: https://github.com/git-for-windows/git/commit/b30404dfc04a4b087b630aea4ab88a51cd3a7459 [3]: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew [4]: https://github.com/git-for-windows/git/issues/5436 Signed-off-by: Eliah Kagan Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- compat/mingw.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/compat/mingw.c b/compat/mingw.c index 1d5b211b54..f524c54d06 100644 --- a/compat/mingw.c +++ b/compat/mingw.c @@ -2278,7 +2278,9 @@ repeat: old_handle = CreateFileW(wpold, DELETE, FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE, - NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL); + NULL, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT, + NULL); if (old_handle == INVALID_HANDLE_VALUE) { errno = err_win_to_posix(GetLastError()); return -1; From 2d2a71ce85026edcc40f469678a1035df0dfcf57 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Fri, 21 Feb 2025 10:35:39 -0800 Subject: [PATCH 24/43] The thirteenth batch Signed-off-by: Junio C Hamano --- Documentation/RelNotes/2.49.0.adoc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Documentation/RelNotes/2.49.0.adoc b/Documentation/RelNotes/2.49.0.adoc index 64323ec4a9..83bab70cb6 100644 --- a/Documentation/RelNotes/2.49.0.adoc +++ b/Documentation/RelNotes/2.49.0.adoc @@ -33,6 +33,8 @@ UI, Workflows & Features * "git rev-list --missing=" learned to accept "print-info" that gives known details expected of the missing objects, like path and type. + * Comes with an updated "gitk". + Performance, Internal Implementation, Development Support etc. -------------------------------------------------------------- @@ -217,3 +219,6 @@ Fixes since v2.48 (merge 087740d65a ps/leakfixes-0129 later to maint). (merge 6bba6f604b jp/doc-trailer-config later to maint). (merge f1cc562b77 lo/t7603-path-is-file-update later to maint). + (merge 45761988ac en/doc-renormalize later to maint). + (merge 832f56f06a jc/doc-boolean-synonyms later to maint). + (merge 3eeed876a9 ac/doc-http-ssl-type-config later to maint). From bb60c5213134cda73c31940ec0fb33c361a064eb Mon Sep 17 00:00:00 2001 From: Jacob Keller Date: Fri, 21 Feb 2025 15:47:58 -0800 Subject: [PATCH 25/43] mailmap: fix check-mailmap with full mailmap line I recently had reported to me a crash from a coworker using the recently added sendemail mailmap support: 3724814 Segmentation fault (core dumped) git check-mailmap "bugs@company.xx" This appears to happen because of the NULL pointer name passed into map_user(). Fix this by passing "" instead of NULL so that we have a valid pointer. Signed-off-by: Jacob Keller Signed-off-by: Junio C Hamano --- builtin/check-mailmap.c | 2 +- t/t4203-mailmap.sh | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/builtin/check-mailmap.c b/builtin/check-mailmap.c index df00b5ee13..be2cebe121 100644 --- a/builtin/check-mailmap.c +++ b/builtin/check-mailmap.c @@ -35,7 +35,7 @@ static void check_mailmap(struct string_list *mailmap, const char *contact) mail = ident.mail_begin; maillen = ident.mail_end - ident.mail_begin; } else { - name = NULL; + name = ""; namelen = 0; mail = contact; maillen = strlen(contact); diff --git a/t/t4203-mailmap.sh b/t/t4203-mailmap.sh index 2265ff8872..0efa1ac86c 100755 --- a/t/t4203-mailmap.sh +++ b/t/t4203-mailmap.sh @@ -114,6 +114,18 @@ test_expect_success 'check-mailmap --stdin simple address: no mapping' ' test_cmp expect actual ' +test_expect_success 'check-mailmap name and address: mapping' ' + test_when_finished "rm .mailmap" && + cat >.mailmap <<-EOF && + Bug Reports Bugs + EOF + cat >expect <<-EOF && + + EOF + git check-mailmap "bugs@company.xx" >actual && + test_cmp expect actual +' + test_expect_success 'No mailmap' ' cat >expect <<-EOF && $GIT_AUTHOR_NAME (1): From 5a526e5e18ddb9a7dfc5a2967d21d6154df64a4f Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Tue, 25 Feb 2025 11:54:57 -0800 Subject: [PATCH 26/43] The fourteenth batch Signed-off-by: Junio C Hamano --- Documentation/RelNotes/2.49.0.adoc | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/Documentation/RelNotes/2.49.0.adoc b/Documentation/RelNotes/2.49.0.adoc index 83bab70cb6..96b46d58f4 100644 --- a/Documentation/RelNotes/2.49.0.adoc +++ b/Documentation/RelNotes/2.49.0.adoc @@ -35,6 +35,9 @@ UI, Workflows & Features * Comes with an updated "gitk". + * The documentation of "git commit" and "git rebase" now refer to + commit titles as such, not "subject". + Performance, Internal Implementation, Development Support etc. -------------------------------------------------------------- @@ -204,6 +207,19 @@ Fixes since v2.48 * A thunderbird helper script lost its bashism. (merge 59d26bd961 bc/contrib-thunderbird-patch-inline-fix later to maint). + * The -G/-S options to the "diff" family of commands caused us to hit + a BUG() when they get no values; they have been corrected. + (merge a620046b29 bc/diff-reject-empty-arg-to-pickaxe later to maint). + + * "git merge-tree --stdin" has been improved (including a workaround + for a deadlock). + (merge 6a9ae81015 pw/merge-tree-stdin-deadlock-fix later to maint). + + * Correct the default target in Documentation/Makefile, and + future-proof all Makefiles from similar breakages by declaring the + default target (which happens to be "all") upfront. + (merge 5309c1e9fb ad/set-default-target-in-makefiles later to maint). + * Other code cleanup, docfix, build fix, etc. (merge ddb5287894 jk/t7407-use-test-grep later to maint). (merge 21e1b44865 aj/difftool-config-doc-fix later to maint). From 08bdfd453584e489d5a551aecbdcb77584e1b958 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Wed, 26 Feb 2025 08:55:18 -0800 Subject: [PATCH 27/43] Git 2.49-rc0 Signed-off-by: Junio C Hamano --- Documentation/RelNotes/2.49.0.adoc | 6 ++++++ GIT-VERSION-GEN | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/Documentation/RelNotes/2.49.0.adoc b/Documentation/RelNotes/2.49.0.adoc index 96b46d58f4..2e9aa0d69f 100644 --- a/Documentation/RelNotes/2.49.0.adoc +++ b/Documentation/RelNotes/2.49.0.adoc @@ -220,6 +220,12 @@ Fixes since v2.48 default target (which happens to be "all") upfront. (merge 5309c1e9fb ad/set-default-target-in-makefiles later to maint). + * "git check-mailmap" used to segfault when queried without human + readable name. + (merge bb60c52131 jk/check-mailmap-wo-name-fix later to maint). + + * Support for renaming of symbolic links on Windows has been improved. + * Other code cleanup, docfix, build fix, etc. (merge ddb5287894 jk/t7407-use-test-grep later to maint). (merge 21e1b44865 aj/difftool-config-doc-fix later to maint). diff --git a/GIT-VERSION-GEN b/GIT-VERSION-GEN index 56b45333c1..67fa0401cf 100755 --- a/GIT-VERSION-GEN +++ b/GIT-VERSION-GEN @@ -1,6 +1,6 @@ #!/bin/sh -DEF_VER=v2.48.GIT +DEF_VER=v2.49.0-rc0 LF=' ' From 37e7546b91fb7d7c263d809dbf7d85831c15cc4f Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:07 +0100 Subject: [PATCH 28/43] object-name: introduce `repo_get_oid_with_flags()` Introduce a new function `repo_get_oid_with_flags()`. This function behaves the same as `repo_get_oid()`, except that it takes an extra `flags` parameter that it ends up passing to `get_oid_with_context()`. This function will be used in a subsequent commit. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- object-name.c | 14 ++++++++------ object-name.h | 6 ++++++ 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/object-name.c b/object-name.c index 945d5bdef2..233f3f861e 100644 --- a/object-name.c +++ b/object-name.c @@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx) strbuf_release(&ctx->symlink_path); } -/* - * This is like "get_oid_basic()", except it allows "object ID expressions", - * notably "xyz^" for "parent of xyz" - */ -int repo_get_oid(struct repository *r, const char *name, struct object_id *oid) +int repo_get_oid_with_flags(struct repository *r, const char *name, + struct object_id *oid, unsigned flags) { struct object_context unused; - int ret = get_oid_with_context(r, name, 0, oid, &unused); + int ret = get_oid_with_context(r, name, flags, oid, &unused); object_context_release(&unused); return ret; } +int repo_get_oid(struct repository *r, const char *name, struct object_id *oid) +{ + return repo_get_oid_with_flags(r, name, oid, 0); +} + /* * This returns a non-zero value if the string (built using printf * format and the given arguments) is not a valid object. diff --git a/object-name.h b/object-name.h index 8dba4a47a4..cda4934cd5 100644 --- a/object-name.h +++ b/object-name.h @@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo, void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid, int abbrev_len); +/* + * This is like "get_oid_basic()", except it allows "object ID expressions", + * notably "xyz^" for "parent of xyz". Accepts GET_OID_* flags. + */ +int repo_get_oid_with_flags(struct repository *r, const char *str, + struct object_id *oid, unsigned flags); int repo_get_oid(struct repository *r, const char *str, struct object_id *oid); __attribute__((format (printf, 2, 3))) int get_oidf(struct object_id *oid, const char *fmt, ...); From 46a2b52240ef687ad4c9d2aec4317796704c991a Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:08 +0100 Subject: [PATCH 29/43] object-name: allow skipping ambiguity checks in `get_oid()` family When reading an object ID via `get_oid_basic()` or any of its related functions we perform a check whether the object ID is ambiguous, which can be the case when a reference with the same name exists. While the check is generally helpful, there are cases where it only adds to the runtime overhead without providing much of a benefit. Add a new flag that allows us to disable the check. The flag will be used in a subsequent commit. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- hash.h | 23 ++++++++++++----------- object-name.c | 4 +++- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/hash.h b/hash.h index 4367acfec5..5e3c462dc5 100644 --- a/hash.h +++ b/hash.h @@ -193,17 +193,18 @@ struct object_id { int algo; /* XXX requires 4-byte alignment */ }; -#define GET_OID_QUIETLY 01 -#define GET_OID_COMMIT 02 -#define GET_OID_COMMITTISH 04 -#define GET_OID_TREE 010 -#define GET_OID_TREEISH 020 -#define GET_OID_BLOB 040 -#define GET_OID_FOLLOW_SYMLINKS 0100 -#define GET_OID_RECORD_PATH 0200 -#define GET_OID_ONLY_TO_DIE 04000 -#define GET_OID_REQUIRE_PATH 010000 -#define GET_OID_HASH_ANY 020000 +#define GET_OID_QUIETLY 01 +#define GET_OID_COMMIT 02 +#define GET_OID_COMMITTISH 04 +#define GET_OID_TREE 010 +#define GET_OID_TREEISH 020 +#define GET_OID_BLOB 040 +#define GET_OID_FOLLOW_SYMLINKS 0100 +#define GET_OID_RECORD_PATH 0200 +#define GET_OID_ONLY_TO_DIE 04000 +#define GET_OID_REQUIRE_PATH 010000 +#define GET_OID_HASH_ANY 020000 +#define GET_OID_SKIP_AMBIGUITY_CHECK 040000 #define GET_OID_DISAMBIGUATORS \ (GET_OID_COMMIT | GET_OID_COMMITTISH | \ diff --git a/object-name.c b/object-name.c index 233f3f861e..85444dbb15 100644 --- a/object-name.c +++ b/object-name.c @@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len, int fatal = !(flags & GET_OID_QUIETLY); if (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) { - if (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) { + if (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) && + repo_settings_get_warn_ambiguous_refs(r) && + warn_on_object_refname_ambiguity) { refs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0); if (refs_found > 0) { warning(warn_msg, len, str); From 3c20bf0c850561482e3da79311d3de1fcfb36091 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:09 +0100 Subject: [PATCH 30/43] builtin/update-ref: skip ambiguity checks when parsing object IDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Most of the commands in git-update-ref(1) accept an old and/or new object ID to update a specific reference to. These object IDs get parsed via `repo_get_oid()`, which not only handles plain object IDs, but also those that have a suffix like "~" or "^2". More surprisingly though, it even knows to resolve arbitrary revisions, despite the fact that its manpage does not mention this fact even once. One consequence of this is that we also check for ambiguous references: when parsing a full object ID where the DWIM mechanism would also cause us to resolve it as a branch, we'd end up printing a warning. While this check makes sense to have in general, it is arguably less useful in the context of git-update-ref(1). This is due to multiple reasons: - The manpage is explicitly structured around object IDs. So if we see a fully blown object ID, the intent should be quite clear in general. - The command is part of our plumbing layer and not a tool that users would generally use in interactive workflows. As such, the warning will likely not be visible to anybody in the first place. - Users can and should use the fully-qualified refname in case there is any potential for ambiguity. And given that this command is part of our plumbing layer, one should always try to be as defensive as possible and use fully-qualified refnames. Furthermore, this check can be quite expensive when updating lots of references via `--stdin`, because we try to read multiple references per object ID that we parse according to the DWIM rules. This effect can be seen both with the "files" and "reftable" backend. The issue is not unique to git-update-ref(1), but was also an issue in git-cat-file(1), where it was addressed by disabling the ambiguity check in 25fba78d36b (cat-file: disable object/refname ambiguity check for batch mode, 2013-07-12). Disable the warning in git-update-ref(1), which provides a significant speedup with both backends. The user-visible outcome is unchanged even when ambiguity exists, except that we don't show the warning anymore. The following benchmark creates 10000 new references with a 100000 preexisting refs with the "files" backend: Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~) Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms] Range (min … max): 461.9 ms … 479.3 ms 10 runs Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms] Range (min … max): 384.9 ms … 405.7 ms 10 runs Summary update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~) And with the "reftable" backend: Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~) Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms] Range (min … max): 142.7 ms … 150.8 ms 19 runs Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms] Range (min … max): 61.1 ms … 66.6 ms 41 runs Summary update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~) Note that the absolute improvement with both backends is roughly in the same ballpark, but the relative improvement for the "reftable" backend is more significant because writing the new table to disk is faster in the first place. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/update-ref.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/builtin/update-ref.c b/builtin/update-ref.c index 4d35bdc4b4..1d541e13ad 100644 --- a/builtin/update-ref.c +++ b/builtin/update-ref.c @@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end, (*next)++; *next = parse_arg(*next, &arg); if (arg.len) { - if (repo_get_oid(the_repository, arg.buf, oid)) + if (repo_get_oid_with_flags(the_repository, arg.buf, oid, + GET_OID_SKIP_AMBIGUITY_CHECK)) goto invalid; } else { /* Without -z, an empty value means all zeros: */ @@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end, *next += arg.len; if (arg.len) { - if (repo_get_oid(the_repository, arg.buf, oid)) + if (repo_get_oid_with_flags(the_repository, arg.buf, oid, + GET_OID_SKIP_AMBIGUITY_CHECK)) goto invalid; } else if (flags & PARSE_SHA1_ALLOW_EMPTY) { /* With -z, treat an empty value as all zeros: */ @@ -299,7 +301,8 @@ static void parse_cmd_symref_update(struct ref_transaction *transaction, die("symref-update %s: expected old value", refname); if (!strcmp(old_arg, "oid")) { - if (repo_get_oid(the_repository, old_target, &old_oid)) + if (repo_get_oid_with_flags(the_repository, old_target, &old_oid, + GET_OID_SKIP_AMBIGUITY_CHECK)) die("symref-update %s: invalid oid: %s", refname, old_target); have_old_oid = 1; @@ -772,7 +775,8 @@ int cmd_update_ref(int argc, refname = argv[0]; value = argv[1]; oldval = argv[2]; - if (repo_get_oid(the_repository, value, &oid)) + if (repo_get_oid_with_flags(the_repository, value, &oid, + GET_OID_SKIP_AMBIGUITY_CHECK)) die("%s: not a valid SHA1", value); } @@ -783,7 +787,8 @@ int cmd_update_ref(int argc, * must not already exist: */ oidclr(&oldoid, the_repository->hash_algo); - else if (repo_get_oid(the_repository, oldval, &oldoid)) + else if (repo_get_oid_with_flags(the_repository, oldval, &oldoid, + GET_OID_SKIP_AMBIGUITY_CHECK)) die("%s: not a valid old SHA1", oldval); } From 2ff58dec493ab5bebb6943b814461ba4e9937e15 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:10 +0100 Subject: [PATCH 31/43] refs: introduce function to batch refname availability checks The `refs_verify_refname_available()` functions checks whether a reference update can be committed or whether it would conflict with either a prefix or suffix thereof. This function needs to be called once per reference that one wants to check, which requires us to redo a couple of checks every time the function is called. Introduce a new function `refs_verify_refnames_available()` that does the same, but for a list of references. For now, the new function uses the exact same implementation, except that we loop through all refnames provided by the caller. This will be tuned in subsequent commits. The existing `refs_verify_refname_available()` function is reimplemented on top of the new function. As such, the diff is best viewed with the `--ignore-space-change option`. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs.c | 172 +++++++++++++++++++++++++++++++++------------------------ refs.h | 12 ++++ 2 files changed, 111 insertions(+), 73 deletions(-) diff --git a/refs.c b/refs.c index e1a6a2d189..03fa238d96 100644 --- a/refs.c +++ b/refs.c @@ -2475,19 +2475,16 @@ int ref_transaction_commit(struct ref_transaction *transaction, return ret; } -int refs_verify_refname_available(struct ref_store *refs, - const char *refname, - const struct string_list *extras, - const struct string_list *skip, - unsigned int initial_transaction, - struct strbuf *err) +int refs_verify_refnames_available(struct ref_store *refs, + const struct string_list *refnames, + const struct string_list *extras, + const struct string_list *skip, + unsigned int initial_transaction, + struct strbuf *err) { - const char *slash; - const char *extra_refname; struct strbuf dirname = STRBUF_INIT; struct strbuf referent = STRBUF_INIT; - struct object_id oid; - unsigned int type; + struct string_list_item *item; int ret = -1; /* @@ -2497,79 +2494,91 @@ int refs_verify_refname_available(struct ref_store *refs, assert(err); - strbuf_grow(&dirname, strlen(refname) + 1); - for (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) { - /* - * Just saying "Is a directory" when we e.g. can't - * lock some multi-level ref isn't very informative, - * the user won't be told *what* is a directory, so - * let's not use strerror() below. - */ - int ignore_errno; - /* Expand dirname to the new prefix, not including the trailing slash: */ - strbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len); + for_each_string_list_item(item, refnames) { + const char *refname = item->string; + const char *extra_refname; + struct object_id oid; + unsigned int type; + const char *slash; - /* - * We are still at a leading dir of the refname (e.g., - * "refs/foo"; if there is a reference with that name, - * it is a conflict, *unless* it is in skip. - */ - if (skip && string_list_has_string(skip, dirname.buf)) - continue; + strbuf_reset(&dirname); - if (!initial_transaction && - !refs_read_raw_ref(refs, dirname.buf, &oid, &referent, - &type, &ignore_errno)) { - strbuf_addf(err, _("'%s' exists; cannot create '%s'"), - dirname.buf, refname); - goto cleanup; - } + for (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) { + /* + * Just saying "Is a directory" when we e.g. can't + * lock some multi-level ref isn't very informative, + * the user won't be told *what* is a directory, so + * let's not use strerror() below. + */ + int ignore_errno; - if (extras && string_list_has_string(extras, dirname.buf)) { - strbuf_addf(err, _("cannot process '%s' and '%s' at the same time"), - refname, dirname.buf); - goto cleanup; - } - } + /* Expand dirname to the new prefix, not including the trailing slash: */ + strbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len); - /* - * We are at the leaf of our refname (e.g., "refs/foo/bar"). - * There is no point in searching for a reference with that - * name, because a refname isn't considered to conflict with - * itself. But we still need to check for references whose - * names are in the "refs/foo/bar/" namespace, because they - * *do* conflict. - */ - strbuf_addstr(&dirname, refname + dirname.len); - strbuf_addch(&dirname, '/'); - - if (!initial_transaction) { - struct ref_iterator *iter; - int ok; - - iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0, - DO_FOR_EACH_INCLUDE_BROKEN); - while ((ok = ref_iterator_advance(iter)) == ITER_OK) { - if (skip && - string_list_has_string(skip, iter->refname)) + /* + * We are still at a leading dir of the refname (e.g., + * "refs/foo"; if there is a reference with that name, + * it is a conflict, *unless* it is in skip. + */ + if (skip && string_list_has_string(skip, dirname.buf)) continue; - strbuf_addf(err, _("'%s' exists; cannot create '%s'"), - iter->refname, refname); - ref_iterator_abort(iter); - goto cleanup; + if (!initial_transaction && + !refs_read_raw_ref(refs, dirname.buf, &oid, &referent, + &type, &ignore_errno)) { + strbuf_addf(err, _("'%s' exists; cannot create '%s'"), + dirname.buf, refname); + goto cleanup; + } + + if (extras && string_list_has_string(extras, dirname.buf)) { + strbuf_addf(err, _("cannot process '%s' and '%s' at the same time"), + refname, dirname.buf); + goto cleanup; + } } - if (ok != ITER_DONE) - BUG("error while iterating over references"); + /* + * We are at the leaf of our refname (e.g., "refs/foo/bar"). + * There is no point in searching for a reference with that + * name, because a refname isn't considered to conflict with + * itself. But we still need to check for references whose + * names are in the "refs/foo/bar/" namespace, because they + * *do* conflict. + */ + strbuf_addstr(&dirname, refname + dirname.len); + strbuf_addch(&dirname, '/'); + + if (!initial_transaction) { + struct ref_iterator *iter; + int ok; + + iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0, + DO_FOR_EACH_INCLUDE_BROKEN); + while ((ok = ref_iterator_advance(iter)) == ITER_OK) { + if (skip && + string_list_has_string(skip, iter->refname)) + continue; + + strbuf_addf(err, _("'%s' exists; cannot create '%s'"), + iter->refname, refname); + ref_iterator_abort(iter); + goto cleanup; + } + + if (ok != ITER_DONE) + BUG("error while iterating over references"); + } + + extra_refname = find_descendant_ref(dirname.buf, extras, skip); + if (extra_refname) { + strbuf_addf(err, _("cannot process '%s' and '%s' at the same time"), + refname, extra_refname); + goto cleanup; + } } - extra_refname = find_descendant_ref(dirname.buf, extras, skip); - if (extra_refname) - strbuf_addf(err, _("cannot process '%s' and '%s' at the same time"), - refname, extra_refname); - else - ret = 0; + ret = 0; cleanup: strbuf_release(&referent); @@ -2577,6 +2586,23 @@ cleanup: return ret; } +int refs_verify_refname_available(struct ref_store *refs, + const char *refname, + const struct string_list *extras, + const struct string_list *skip, + unsigned int initial_transaction, + struct strbuf *err) +{ + struct string_list_item item = { .string = (char *) refname }; + struct string_list refnames = { + .items = &item, + .nr = 1, + }; + + return refs_verify_refnames_available(refs, &refnames, extras, skip, + initial_transaction, err); +} + struct do_for_each_reflog_help { each_reflog_fn *fn; void *cb_data; diff --git a/refs.h b/refs.h index 09be47afbe..b14ba1f9ff 100644 --- a/refs.h +++ b/refs.h @@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs, unsigned int initial_transaction, struct strbuf *err); +/* + * Same as `refs_verify_refname_available()`, but checking for a list of + * refnames instead of only a single item. This is more efficient in the case + * where one needs to check multiple refnames. + */ +int refs_verify_refnames_available(struct ref_store *refs, + const struct string_list *refnames, + const struct string_list *extras, + const struct string_list *skip, + unsigned int initial_transaction, + struct strbuf *err); + int refs_ref_exists(struct ref_store *refs, const char *refname); int should_autocreate_reflog(enum log_refs_config log_all_ref_updates, From 351f592e1d6668f07529d30f05c04725a0a17b59 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:11 +0100 Subject: [PATCH 32/43] refs/reftable: batch refname availability checks Refactor the "reftable" backend to batch the availability check for refnames. This does not yet have an effect on performance as `refs_verify_refnames_available()` effectively still performs the availability check for each refname individually. But this will be optimized in subsequent commits, where we learn to optimize some parts of the logic when checking multiple refnames for availability. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/reftable-backend.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 7e90e13f74..546861d64c 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, reftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, "ref_transaction_prepare"); struct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT; struct string_list affected_refnames = STRING_LIST_INIT_NODUP; + struct string_list refnames_to_check = STRING_LIST_INIT_NODUP; struct reftable_transaction_data *tx_data = NULL; struct reftable_backend *be; struct object_id head_oid; @@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, * can output a proper error message instead of failing * at a later point. */ - ret = refs_verify_refname_available(ref_store, u->refname, - &affected_refnames, NULL, - transaction->flags & REF_TRANSACTION_FLAG_INITIAL, - err); - if (ret < 0) - goto done; + string_list_append(&refnames_to_check, u->refname); /* * There is no need to write the reference deletion @@ -1379,6 +1375,12 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, } } + ret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL, + transaction->flags & REF_TRANSACTION_FLAG_INITIAL, + err); + if (ret < 0) + goto done; + transaction->backend_data = tx_data; transaction->state = REF_TRANSACTION_PREPARED; @@ -1394,6 +1396,7 @@ done: string_list_clear(&affected_refnames, 0); strbuf_release(&referent); strbuf_release(&head_referent); + string_list_clear(&refnames_to_check, 0); return ret; } From 6c90726bebfd8ec4dd429f1fad9d00112b1b6603 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:12 +0100 Subject: [PATCH 33/43] refs/files: batch refname availability checks for normal transactions Same as the "reftable" backend that we have adapted in the preceding commit to use batched refname availability checks we can also do so for the "files" backend. Things are a bit more intricate here though, as we call `refs_verify_refname_available()` in a set of different contexts: 1. `lock_raw_ref()` when it hits either EEXISTS or EISDIR when creating a new reference, mostly to create a nice, user-readable error message. This is nothing we have to care about too much, as we only hit this code path at most once when we hit a conflict. 2. `lock_raw_ref()` when it _could_ create the lockfile to check whether it is conflicting with any packed refs. In the general case, this code path will be hit once for every (successful) reference update. 3. `lock_ref_oid_basic()`, but it is only executed when copying or renaming references or when expiring reflogs. It will thus not be called in contexts where we have many references queued up. 4. `refs_refname_ref_available()`, but again only when copying or renaming references. It is thus not interesting due to the same reason as the previous case. 5. `files_transaction_finish_initial()`, which is only executed when creating a new repository or migrating references. So out of these, only (2) and (5) are viable candidates to use the batched checks. Adapt `lock_raw_ref()` accordingly by queueing up reference names that need to be checked for availability and then checking them after we have processed all updates. This check is done before we (optionally) lock the `packed-refs` file, which is somewhat flawed because it means that the `packed-refs` could still change after the availability check and thus create an undetected conflict. But unconditionally locking the file would change semantics that users are likely to rely on, so we keep the current locking sequence intact, even if it's suboptmial. The refactoring of `files_transaction_finish_initial()` will be done in the next commit. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/files-backend.c | 42 +++++++++++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 11 deletions(-) diff --git a/refs/files-backend.c b/refs/files-backend.c index 29f08dced4..f798d8dae3 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -678,6 +678,7 @@ static void unlock_ref(struct ref_lock *lock) */ static int lock_raw_ref(struct files_ref_store *refs, const char *refname, int mustexist, + struct string_list *refnames_to_check, const struct string_list *extras, struct ref_lock **lock_p, struct strbuf *referent, @@ -855,16 +856,11 @@ retry: } /* - * If the ref did not exist and we are creating it, - * make sure there is no existing packed ref that - * conflicts with refname: + * If the ref did not exist and we are creating it, we have to + * make sure there is no existing packed ref that conflicts + * with refname. This check is deferred so that we can batch it. */ - if (refs_verify_refname_available( - refs->packed_ref_store, refname, - extras, NULL, 0, err)) { - ret = TRANSACTION_NAME_CONFLICT; - goto error_return; - } + string_list_append(refnames_to_check, refname); } ret = 0; @@ -2569,6 +2565,7 @@ static int lock_ref_for_update(struct files_ref_store *refs, struct ref_update *update, struct ref_transaction *transaction, const char *head_ref, + struct string_list *refnames_to_check, struct string_list *affected_refnames, struct strbuf *err) { @@ -2597,7 +2594,7 @@ static int lock_ref_for_update(struct files_ref_store *refs, lock->count++; } else { ret = lock_raw_ref(refs, update->refname, mustexist, - affected_refnames, + refnames_to_check, affected_refnames, &lock, &referent, &update->type, err); if (ret) { @@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store, size_t i; int ret = 0; struct string_list affected_refnames = STRING_LIST_INIT_NODUP; + struct string_list refnames_to_check = STRING_LIST_INIT_NODUP; char *head_ref = NULL; int head_type; struct files_transaction_backend_data *backend_data; @@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store, struct ref_update *update = transaction->updates[i]; ret = lock_ref_for_update(refs, update, transaction, - head_ref, &affected_refnames, err); + head_ref, &refnames_to_check, + &affected_refnames, err); if (ret) goto cleanup; @@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store, } } + /* + * Verify that none of the loose reference that we're about to write + * conflict with any existing packed references. Ideally, we'd do this + * check after the packed-refs are locked so that the file cannot + * change underneath our feet. But introducing such a lock now would + * probably do more harm than good as users rely on there not being a + * global lock with the "files" backend. + * + * Another alternative would be to do the check after the (optional) + * lock, but that would extend the time we spend in the globally-locked + * state. + * + * So instead, we accept the race for now. + */ + if (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check, + &affected_refnames, NULL, 0, err)) { + ret = TRANSACTION_NAME_CONFLICT; + goto cleanup; + } + if (packed_transaction) { if (packed_refs_lock(refs->packed_ref_store, 0, err)) { ret = TRANSACTION_GENERIC_ERROR; @@ -2972,6 +2991,7 @@ static int files_transaction_prepare(struct ref_store *ref_store, cleanup: free(head_ref); string_list_clear(&affected_refnames, 0); + string_list_clear(&refnames_to_check, 0); if (ret) files_transaction_cleanup(refs, transaction); From 268ea8515cd11bd0f3f8c4d64373121058c3fac2 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:13 +0100 Subject: [PATCH 34/43] refs/files: batch refname availability checks for initial transactions The "files" backend explicitly carves out special logic for its initial transaction so that it can avoid writing out every single reference as a loose reference. While the assumption is that there shouldn't be any preexisting references, we still have to verify that none of the newly written references will conflict with any other new reference in the same transaction. Refactor the initial transaction to use batched refname availability checks. This does not yet have an effect on performance as we still call `refs_verify_refname_available()` in a loop. But this will change in subsequent commits and then impact performance when cloning a repository with many references or when migrating references to the "files" format. This will improve performance when cloning a repository with many references or when migrating references from any format to the "files" format once the availability checks have learned to optimize checks for many references in a subsequent commit. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/files-backend.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/refs/files-backend.c b/refs/files-backend.c index f798d8dae3..ab6f0af550 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs, size_t i; int ret = 0; struct string_list affected_refnames = STRING_LIST_INIT_NODUP; + struct string_list refnames_to_check = STRING_LIST_INIT_NODUP; struct ref_transaction *packed_transaction = NULL; struct ref_transaction *loose_transaction = NULL; @@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs, !is_null_oid(&update->old_oid)) BUG("initial ref transaction with old_sha1 set"); - if (refs_verify_refname_available(&refs->base, update->refname, - &affected_refnames, NULL, 1, err)) { - ret = TRANSACTION_NAME_CONFLICT; - goto cleanup; - } + string_list_append(&refnames_to_check, update->refname); /* * packed-refs don't support symbolic refs, root refs and reflogs, @@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs, } } - if (packed_refs_lock(refs->packed_ref_store, 0, err) || - ref_transaction_commit(packed_transaction, err)) { + if (packed_refs_lock(refs->packed_ref_store, 0, err)) { + ret = TRANSACTION_GENERIC_ERROR; + goto cleanup; + } + + if (refs_verify_refnames_available(&refs->base, &refnames_to_check, + &affected_refnames, NULL, 1, err)) { + packed_refs_unlock(refs->packed_ref_store); + ret = TRANSACTION_NAME_CONFLICT; + goto cleanup; + } + + if (ref_transaction_commit(packed_transaction, err)) { ret = TRANSACTION_GENERIC_ERROR; goto cleanup; } @@ -3167,6 +3175,7 @@ cleanup: ref_transaction_free(packed_transaction); transaction->state = REF_TRANSACTION_CLOSED; string_list_clear(&affected_refnames, 0); + string_list_clear(&refnames_to_check, 0); return ret; } From 9e39a966ecd6e3bdb8fe028ec1869bfb9018b200 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:14 +0100 Subject: [PATCH 35/43] refs: stop re-verifying common prefixes for availability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One of the checks done by `refs_verify_refnames_available()` is whether any of the prefixes of a reference already exists. For example, given a reference "refs/heads/main", we'd check whether "refs/heads" or "refs" already exist, and if so we'd abort the transaction. When updating multiple references at once, this check is performed for each of the references individually. Consequently, because references tend to have common prefixes like "refs/heads/" or refs/tags/", we evaluate the availability of these prefixes repeatedly. Naturally this is a waste of compute, as the availability of those prefixes should in general not change in the middle of a transaction. And if it would, backends would notice at a later point in time. Optimize this pattern by storing prefixes in a `strset` so that we can trivially track those prefixes that we have already checked. This leads to a significant speedup with the "reftable" backend when creating many references that all share a common prefix: Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~) Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms] Range (min … max): 60.6 ms … 69.5 ms 38 runs Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms] Range (min … max): 38.1 ms … 47.3 ms 61 runs Summary update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~) For the "files" backend we see an improvement, but a much smaller one: Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~) Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms] Range (min … max): 387.0 ms … 404.6 ms 10 runs Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms] Range (min … max): 380.8 ms … 392.6 ms 10 runs Summary update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~) This change also leads to a modest improvement when writing references with "initial" semantics, for example when migrating references. The following benchmarks are migrating 1m references from the "reftable" to the "files" backend: Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~) Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms] Range (min … max): 829.6 ms … 845.9 ms 10 runs Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD) Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms] Range (min … max): 753.1 ms … 768.8 ms 10 runs Summary migrate reftable:files (refcount = 1000000, revision = HEAD) ran 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~) And vice versa: Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~) Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms] Range (min … max): 861.6 ms … 883.2 ms 10 runs Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD) Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms] Range (min … max): 787.5 ms … 812.6 ms 10 runs Summary migrate files:reftable (refcount = 1000000, revision = HEAD) ran 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~) The impact here is significantly smaller given that we don't perform any reference reads with "initial" semantics, so the speedup only comes from us doing less string list lookups. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/refs.c b/refs.c index 03fa238d96..957446da9e 100644 --- a/refs.c +++ b/refs.c @@ -2485,6 +2485,7 @@ int refs_verify_refnames_available(struct ref_store *refs, struct strbuf dirname = STRBUF_INIT; struct strbuf referent = STRBUF_INIT; struct string_list_item *item; + struct strset dirnames; int ret = -1; /* @@ -2494,6 +2495,8 @@ int refs_verify_refnames_available(struct ref_store *refs, assert(err); + strset_init(&dirnames); + for_each_string_list_item(item, refnames) { const char *refname = item->string; const char *extra_refname; @@ -2523,6 +2526,14 @@ int refs_verify_refnames_available(struct ref_store *refs, if (skip && string_list_has_string(skip, dirname.buf)) continue; + /* + * If we've already seen the directory we don't need to + * process it again. Skip it to avoid checking checking + * common prefixes like "refs/heads/" repeatedly. + */ + if (!strset_add(&dirnames, dirname.buf)) + continue; + if (!initial_transaction && !refs_read_raw_ref(refs, dirname.buf, &oid, &referent, &type, &ignore_errno)) { @@ -2583,6 +2594,7 @@ int refs_verify_refnames_available(struct ref_store *refs, cleanup: strbuf_release(&referent); strbuf_release(&dirname); + strset_clear(&dirnames); return ret; } From cec2b6f55a805c010d2acc81abf4cbc41b712130 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:15 +0100 Subject: [PATCH 36/43] refs/iterator: separate lifecycle from iteration The ref and reflog iterators have their lifecycle attached to iteration: once the iterator reaches its end, it is automatically released and the caller doesn't have to care about that anymore. When the iterator should be released before it has been exhausted, callers must explicitly abort the iterator via `ref_iterator_abort()`. This lifecycle is somewhat unusual in the Git codebase and creates two problems: - Callsites need to be very careful about when exactly they call `ref_iterator_abort()`, as calling the function is only valid when the iterator itself still is. This leads to somewhat awkward calling patterns in some situations. - It is impossible to reuse iterators and re-seek them to a different prefix. This feature isn't supported by any iterator implementation except for the reftable iterators anyway, but if it was implemented it would allow us to optimize cases where we need to search for specific references repeatedly by reusing internal state. Detangle the lifecycle from iteration so that we don't deallocate the iterator anymore once it is exhausted. Instead, callers are now expected to always call a newly introduce `ref_iterator_free()` function that deallocates the iterator and its internal state. Note that the `dir_iterator` is somewhat special because it does not implement the `ref_iterator` interface, but is only used to implement other iterators. Consequently, we have to provide `dir_iterator_free()` instead of `dir_iterator_release()` as the allocated structure itself is managed by the `dir_iterator` interfaces, as well, and not freed by `ref_iterator_free()` like in all the other cases. While at it, drop the return value of `ref_iterator_abort()`, which wasn't really required by any of the iterator implementations anyway. Furthermore, stop calling `base_ref_iterator_free()` in any of the backends, but instead call it in `ref_iterator_free()`. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 2 + dir-iterator.c | 24 ++++----- dir-iterator.h | 11 ++-- iterator.h | 2 +- refs.c | 7 ++- refs/debug.c | 9 ++-- refs/files-backend.c | 36 ++++--------- refs/iterator.c | 100 ++++++++++++----------------------- refs/packed-backend.c | 27 +++++----- refs/ref-cache.c | 9 ++-- refs/refs-internal.h | 29 ++++------ refs/reftable-backend.c | 34 +++--------- t/helper/test-dir-iterator.c | 1 + 13 files changed, 105 insertions(+), 186 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index f9a2ecbe9c..add9d8600c 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -342,6 +342,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, strbuf_setlen(src, src_len); die(_("failed to iterate over '%s'"), src->buf); } + + dir_iterator_free(iter); } static void clone_local(const char *src_repo, const char *dest_repo) diff --git a/dir-iterator.c b/dir-iterator.c index de619846f2..857e1d9bda 100644 --- a/dir-iterator.c +++ b/dir-iterator.c @@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) if (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) { if (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC) - goto error_out; + return ITER_ERROR; if (iter->levels_nr == 0) - goto error_out; + return ITER_ERROR; } /* Loop until we find an entry that we can give back to the caller. */ @@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) int ret = next_directory_entry(level->dir, iter->base.path.buf, &de); if (ret < 0) { if (iter->flags & DIR_ITERATOR_PEDANTIC) - goto error_out; + return ITER_ERROR; continue; } else if (ret > 0) { if (pop_level(iter) == 0) - return dir_iterator_abort(dir_iterator); + return ITER_DONE; continue; } @@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) } else { if (level->entries_idx >= level->entries.nr) { if (pop_level(iter) == 0) - return dir_iterator_abort(dir_iterator); + return ITER_DONE; continue; } @@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator) if (prepare_next_entry_data(iter, name)) { if (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC) - goto error_out; + return ITER_ERROR; continue; } return ITER_OK; } - -error_out: - dir_iterator_abort(dir_iterator); - return ITER_ERROR; } -int dir_iterator_abort(struct dir_iterator *dir_iterator) +void dir_iterator_free(struct dir_iterator *dir_iterator) { struct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator; + if (!iter) + return; + for (; iter->levels_nr; iter->levels_nr--) { struct dir_iterator_level *level = &iter->levels[iter->levels_nr - 1]; @@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator) free(iter->levels); strbuf_release(&iter->base.path); free(iter); - return ITER_DONE; } struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags) @@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags) return dir_iterator; error_out: - dir_iterator_abort(dir_iterator); + dir_iterator_free(dir_iterator); errno = saved_errno; return NULL; } diff --git a/dir-iterator.h b/dir-iterator.h index 6d438809b6..ccd6a19734 100644 --- a/dir-iterator.h +++ b/dir-iterator.h @@ -28,7 +28,7 @@ * * while ((ok = dir_iterator_advance(iter)) == ITER_OK) { * if (want_to_stop_iteration()) { - * ok = dir_iterator_abort(iter); + * ok = ITER_DONE; * break; * } * @@ -39,6 +39,7 @@ * * if (ok != ITER_DONE) * handle_error(); + * dir_iterator_free(iter); * * Callers are allowed to modify iter->path while they are working, * but they must restore it to its original contents before calling @@ -107,11 +108,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags); */ int dir_iterator_advance(struct dir_iterator *iterator); -/* - * End the iteration before it has been exhausted. Free the - * dir_iterator and any associated resources and return ITER_DONE. On - * error, free the dir_iterator and return ITER_ERROR. - */ -int dir_iterator_abort(struct dir_iterator *iterator); +/* Free the dir_iterator and any associated resources. */ +void dir_iterator_free(struct dir_iterator *iterator); #endif diff --git a/iterator.h b/iterator.h index 0f6900e43a..6b77dcc262 100644 --- a/iterator.h +++ b/iterator.h @@ -12,7 +12,7 @@ #define ITER_OK 0 /* - * The iterator is exhausted and has been freed. + * The iterator is exhausted. */ #define ITER_DONE -1 diff --git a/refs.c b/refs.c index 957446da9e..eeb8fb1021 100644 --- a/refs.c +++ b/refs.c @@ -2485,6 +2485,7 @@ int refs_verify_refnames_available(struct ref_store *refs, struct strbuf dirname = STRBUF_INIT; struct strbuf referent = STRBUF_INIT; struct string_list_item *item; + struct ref_iterator *iter = NULL; struct strset dirnames; int ret = -1; @@ -2561,7 +2562,6 @@ int refs_verify_refnames_available(struct ref_store *refs, strbuf_addch(&dirname, '/'); if (!initial_transaction) { - struct ref_iterator *iter; int ok; iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0, @@ -2573,12 +2573,14 @@ int refs_verify_refnames_available(struct ref_store *refs, strbuf_addf(err, _("'%s' exists; cannot create '%s'"), iter->refname, refname); - ref_iterator_abort(iter); goto cleanup; } if (ok != ITER_DONE) BUG("error while iterating over references"); + + ref_iterator_free(iter); + iter = NULL; } extra_refname = find_descendant_ref(dirname.buf, extras, skip); @@ -2595,6 +2597,7 @@ cleanup: strbuf_release(&referent); strbuf_release(&dirname); strset_clear(&dirnames); + ref_iterator_free(iter); return ret; } diff --git a/refs/debug.c b/refs/debug.c index fbc4df08b4..a9786da4ba 100644 --- a/refs/debug.c +++ b/refs/debug.c @@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator, return res; } -static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void debug_ref_iterator_release(struct ref_iterator *ref_iterator) { struct debug_ref_iterator *diter = (struct debug_ref_iterator *)ref_iterator; - int res = diter->iter->vtable->abort(diter->iter); - trace_printf_key(&trace_refs, "iterator_abort: %d\n", res); - return res; + diter->iter->vtable->release(diter->iter); + trace_printf_key(&trace_refs, "iterator_abort\n"); } static struct ref_iterator_vtable debug_ref_iterator_vtable = { .advance = debug_ref_iterator_advance, .peel = debug_ref_iterator_peel, - .abort = debug_ref_iterator_abort, + .release = debug_ref_iterator_release, }; static struct ref_iterator * diff --git a/refs/files-backend.c b/refs/files-backend.c index ab6f0af550..e97a267ad6 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -915,10 +915,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) return ITER_OK; } - iter->iter0 = NULL; - if (ref_iterator_abort(ref_iterator) != ITER_DONE) - ok = ITER_ERROR; - return ok; } @@ -931,23 +927,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator, return ref_iterator_peel(iter->iter0, peeled); } -static int files_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void files_ref_iterator_release(struct ref_iterator *ref_iterator) { struct files_ref_iterator *iter = (struct files_ref_iterator *)ref_iterator; - int ok = ITER_DONE; - - if (iter->iter0) - ok = ref_iterator_abort(iter->iter0); - - base_ref_iterator_free(ref_iterator); - return ok; + ref_iterator_free(iter->iter0); } static struct ref_iterator_vtable files_ref_iterator_vtable = { .advance = files_ref_iterator_advance, .peel = files_ref_iterator_peel, - .abort = files_ref_iterator_abort, + .release = files_ref_iterator_release, }; static struct ref_iterator *files_ref_iterator_begin( @@ -1378,7 +1368,7 @@ static int should_pack_refs(struct files_ref_store *refs, iter->flags, opts)) refcount++; if (refcount >= limit) { - ref_iterator_abort(iter); + ref_iterator_free(iter); return 1; } } @@ -1386,6 +1376,7 @@ static int should_pack_refs(struct files_ref_store *refs, if (ret != ITER_DONE) die("error while iterating over references"); + ref_iterator_free(iter); return 0; } @@ -1452,6 +1443,7 @@ static int files_pack_refs(struct ref_store *ref_store, packed_refs_unlock(refs->packed_ref_store); prune_refs(refs, &refs_to_prune); + ref_iterator_free(iter); strbuf_release(&err); return 0; } @@ -2299,9 +2291,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) return ITER_OK; } - iter->dir_iterator = NULL; - if (ref_iterator_abort(ref_iterator) == ITER_ERROR) - ok = ITER_ERROR; return ok; } @@ -2311,23 +2300,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED, BUG("ref_iterator_peel() called for reflog_iterator"); } -static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator) +static void files_reflog_iterator_release(struct ref_iterator *ref_iterator) { struct files_reflog_iterator *iter = (struct files_reflog_iterator *)ref_iterator; - int ok = ITER_DONE; - - if (iter->dir_iterator) - ok = dir_iterator_abort(iter->dir_iterator); - - base_ref_iterator_free(ref_iterator); - return ok; + dir_iterator_free(iter->dir_iterator); } static struct ref_iterator_vtable files_reflog_iterator_vtable = { .advance = files_reflog_iterator_advance, .peel = files_reflog_iterator_peel, - .abort = files_reflog_iterator_abort, + .release = files_reflog_iterator_release, }; static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store, @@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store, ret = error(_("failed to iterate over '%s'"), sb.buf); out: + dir_iterator_free(iter); strbuf_release(&sb); strbuf_release(&refname); return ret; diff --git a/refs/iterator.c b/refs/iterator.c index d25e568bf0..d61474cba7 100644 --- a/refs/iterator.c +++ b/refs/iterator.c @@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator, return ref_iterator->vtable->peel(ref_iterator, peeled); } -int ref_iterator_abort(struct ref_iterator *ref_iterator) +void ref_iterator_free(struct ref_iterator *ref_iterator) { - return ref_iterator->vtable->abort(ref_iterator); + if (ref_iterator) { + ref_iterator->vtable->release(ref_iterator); + /* Help make use-after-free bugs fail quickly: */ + ref_iterator->vtable = NULL; + free(ref_iterator); + } } void base_ref_iterator_init(struct ref_iterator *iter, @@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter, iter->flags = 0; } -void base_ref_iterator_free(struct ref_iterator *iter) -{ - /* Help make use-after-free bugs fail quickly: */ - iter->vtable = NULL; - free(iter); -} - struct empty_ref_iterator { struct ref_iterator base; }; -static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator) +static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) { - return ref_iterator_abort(ref_iterator); + return ITER_DONE; } static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED, @@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED, BUG("peel called for empty iterator"); } -static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED) { - base_ref_iterator_free(ref_iterator); - return ITER_DONE; } static struct ref_iterator_vtable empty_ref_iterator_vtable = { .advance = empty_ref_iterator_advance, .peel = empty_ref_iterator_peel, - .abort = empty_ref_iterator_abort, + .release = empty_ref_iterator_release, }; struct ref_iterator *empty_ref_iterator_begin(void) @@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) if (!iter->current) { /* Initialize: advance both iterators to their first entries */ if ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) { + ref_iterator_free(iter->iter0); iter->iter0 = NULL; if (ok == ITER_ERROR) goto error; } if ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) { + ref_iterator_free(iter->iter1); iter->iter1 = NULL; if (ok == ITER_ERROR) goto error; @@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) * entry: */ if ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) { + ref_iterator_free(*iter->current); *iter->current = NULL; if (ok == ITER_ERROR) goto error; @@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) iter->select(iter->iter0, iter->iter1, iter->cb_data); if (selection == ITER_SELECT_DONE) { - return ref_iterator_abort(ref_iterator); + return ITER_DONE; } else if (selection == ITER_SELECT_ERROR) { - ref_iterator_abort(ref_iterator); return ITER_ERROR; } @@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) if (selection & ITER_SKIP_SECONDARY) { if ((ok = ref_iterator_advance(*secondary)) != ITER_OK) { + ref_iterator_free(*secondary); *secondary = NULL; if (ok == ITER_ERROR) goto error; @@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) } error: - ref_iterator_abort(ref_iterator); return ITER_ERROR; } @@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator, return ref_iterator_peel(*iter->current, peeled); } -static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void merge_ref_iterator_release(struct ref_iterator *ref_iterator) { struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator; - int ok = ITER_DONE; - - if (iter->iter0) { - if (ref_iterator_abort(iter->iter0) != ITER_DONE) - ok = ITER_ERROR; - } - if (iter->iter1) { - if (ref_iterator_abort(iter->iter1) != ITER_DONE) - ok = ITER_ERROR; - } - base_ref_iterator_free(ref_iterator); - return ok; + ref_iterator_free(iter->iter0); + ref_iterator_free(iter->iter1); } static struct ref_iterator_vtable merge_ref_iterator_vtable = { .advance = merge_ref_iterator_advance, .peel = merge_ref_iterator_peel, - .abort = merge_ref_iterator_abort, + .release = merge_ref_iterator_release, }; struct ref_iterator *merge_ref_iterator_begin( @@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin( * them. */ if (is_empty_ref_iterator(front)) { - ref_iterator_abort(front); + ref_iterator_free(front); return back; } else if (is_empty_ref_iterator(back)) { - ref_iterator_abort(back); + ref_iterator_free(back); return front; } @@ -350,19 +338,15 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) while ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) { int cmp = compare_prefix(iter->iter0->refname, iter->prefix); - if (cmp < 0) continue; - - if (cmp > 0) { - /* - * As the source iterator is ordered, we - * can stop the iteration as soon as we see a - * refname that comes after the prefix: - */ - ok = ref_iterator_abort(iter->iter0); - break; - } + /* + * As the source iterator is ordered, we + * can stop the iteration as soon as we see a + * refname that comes after the prefix: + */ + if (cmp > 0) + return ITER_DONE; if (iter->trim) { /* @@ -386,9 +370,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) return ITER_OK; } - iter->iter0 = NULL; - if (ref_iterator_abort(ref_iterator) != ITER_DONE) - return ITER_ERROR; return ok; } @@ -401,23 +382,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator, return ref_iterator_peel(iter->iter0, peeled); } -static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator) { struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; - int ok = ITER_DONE; - - if (iter->iter0) - ok = ref_iterator_abort(iter->iter0); + ref_iterator_free(iter->iter0); free(iter->prefix); - base_ref_iterator_free(ref_iterator); - return ok; } static struct ref_iterator_vtable prefix_ref_iterator_vtable = { .advance = prefix_ref_iterator_advance, .peel = prefix_ref_iterator_peel, - .abort = prefix_ref_iterator_abort, + .release = prefix_ref_iterator_release, }; struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0, @@ -453,20 +429,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter, current_ref_iter = iter; while ((ok = ref_iterator_advance(iter)) == ITER_OK) { retval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data); - if (retval) { - /* - * If ref_iterator_abort() returns ITER_ERROR, - * we ignore that error in deference to the - * callback function's return value. - */ - ref_iterator_abort(iter); + if (retval) goto out; - } } out: current_ref_iter = old_ref_iter; if (ok == ITER_ERROR) - return -1; + retval = -1; + ref_iterator_free(iter); return retval; } diff --git a/refs/packed-backend.c b/refs/packed-backend.c index a7b6f74b6e..38a1956d1a 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) return ITER_OK; } - if (ref_iterator_abort(ref_iterator) != ITER_DONE) - ok = ITER_ERROR; - return ok; } @@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator, } } -static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void packed_ref_iterator_release(struct ref_iterator *ref_iterator) { struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator; - int ok = ITER_DONE; - strbuf_release(&iter->refname_buf); free(iter->jump); release_snapshot(iter->snapshot); - base_ref_iterator_free(ref_iterator); - return ok; } static struct ref_iterator_vtable packed_ref_iterator_vtable = { .advance = packed_ref_iterator_advance, .peel = packed_ref_iterator_peel, - .abort = packed_ref_iterator_abort + .release = packed_ref_iterator_release, }; static int jump_list_entry_cmp(const void *va, const void *vb) @@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs, */ iter = packed_ref_iterator_begin(&refs->base, "", NULL, DO_FOR_EACH_INCLUDE_BROKEN); - if ((ok = ref_iterator_advance(iter)) != ITER_OK) + if ((ok = ref_iterator_advance(iter)) != ITER_OK) { + ref_iterator_free(iter); iter = NULL; + } i = 0; @@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs, * the iterator over the unneeded * value. */ - if ((ok = ref_iterator_advance(iter)) != ITER_OK) + if ((ok = ref_iterator_advance(iter)) != ITER_OK) { + ref_iterator_free(iter); iter = NULL; + } cmp = +1; } else { /* @@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs, peel_error ? NULL : &peeled)) goto write_error; - if ((ok = ref_iterator_advance(iter)) != ITER_OK) + if ((ok = ref_iterator_advance(iter)) != ITER_OK) { + ref_iterator_free(iter); iter = NULL; + } } else if (is_null_oid(&update->new_oid)) { /* * The update wants to delete the reference, @@ -1499,9 +1498,7 @@ write_error: get_tempfile_path(refs->tempfile), strerror(errno)); error: - if (iter) - ref_iterator_abort(iter); - + ref_iterator_free(iter); delete_tempfile(&refs->tempfile); return -1; } diff --git a/refs/ref-cache.c b/refs/ref-cache.c index 02f09e4df8..6457e02c1e 100644 --- a/refs/ref-cache.c +++ b/refs/ref-cache.c @@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) if (++level->index == level->dir->nr) { /* This level is exhausted; pop up a level */ if (--iter->levels_nr == 0) - return ref_iterator_abort(ref_iterator); + return ITER_DONE; continue; } @@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator, return peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0; } -static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void cache_ref_iterator_release(struct ref_iterator *ref_iterator) { struct cache_ref_iterator *iter = (struct cache_ref_iterator *)ref_iterator; - free((char *)iter->prefix); free(iter->levels); - base_ref_iterator_free(ref_iterator); - return ITER_DONE; } static struct ref_iterator_vtable cache_ref_iterator_vtable = { .advance = cache_ref_iterator_advance, .peel = cache_ref_iterator_peel, - .abort = cache_ref_iterator_abort + .release = cache_ref_iterator_release, }; struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache, diff --git a/refs/refs-internal.h b/refs/refs-internal.h index 8894b43d1d..7d3bab654b 100644 --- a/refs/refs-internal.h +++ b/refs/refs-internal.h @@ -273,11 +273,11 @@ enum do_for_each_ref_flags { * the next reference and returns ITER_OK. The data pointed at by * refname and oid belong to the iterator; if you want to retain them * after calling ref_iterator_advance() again or calling - * ref_iterator_abort(), you must make a copy. When the iteration has + * ref_iterator_free(), you must make a copy. When the iteration has * been exhausted, ref_iterator_advance() releases any resources * associated with the iteration, frees the ref_iterator object, and * returns ITER_DONE. If you want to abort the iteration early, call - * ref_iterator_abort(), which also frees the ref_iterator object and + * ref_iterator_free(), which also frees the ref_iterator object and * any associated resources. If there was an internal error advancing * to the next entry, ref_iterator_advance() aborts the iteration, * frees the ref_iterator, and returns ITER_ERROR. @@ -293,7 +293,7 @@ enum do_for_each_ref_flags { * * while ((ok = ref_iterator_advance(iter)) == ITER_OK) { * if (want_to_stop_iteration()) { - * ok = ref_iterator_abort(iter); + * ok = ITER_DONE; * break; * } * @@ -307,6 +307,7 @@ enum do_for_each_ref_flags { * * if (ok != ITER_DONE) * handle_error(); + * ref_iterator_free(iter); */ struct ref_iterator { struct ref_iterator_vtable *vtable; @@ -333,12 +334,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator); int ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled); -/* - * End the iteration before it has been exhausted, freeing the - * reference iterator and any associated resources and returning - * ITER_DONE. If the abort itself failed, return ITER_ERROR. - */ -int ref_iterator_abort(struct ref_iterator *ref_iterator); +/* Free the reference iterator and any associated resources. */ +void ref_iterator_free(struct ref_iterator *ref_iterator); /* * An iterator over nothing (its first ref_iterator_advance() call @@ -438,13 +435,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0, void base_ref_iterator_init(struct ref_iterator *iter, struct ref_iterator_vtable *vtable); -/* - * Base class destructor for ref_iterators. Destroy the ref_iterator - * part of iter and shallow-free the object. This is meant to be - * called only by the destructors of derived classes. - */ -void base_ref_iterator_free(struct ref_iterator *iter); - /* Virtual function declarations for ref_iterators: */ /* @@ -463,15 +453,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator, /* * Implementations of this function should free any resources specific - * to the derived class, then call base_ref_iterator_free() to clean - * up and free the ref_iterator object. + * to the derived class. */ -typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator); +typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator); struct ref_iterator_vtable { ref_iterator_advance_fn *advance; ref_iterator_peel_fn *peel; - ref_iterator_abort_fn *abort; + ref_iterator_release_fn *release; }; /* diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 546861d64c..2d5f4afe6b 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) break; } - if (iter->err > 0) { - if (ref_iterator_abort(ref_iterator) != ITER_DONE) - return ITER_ERROR; + if (iter->err > 0) return ITER_DONE; - } - - if (iter->err < 0) { - ref_iterator_abort(ref_iterator); + if (iter->err < 0) return ITER_ERROR; - } - return ITER_OK; } @@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator, return -1; } -static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator) +static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator) { struct reftable_ref_iterator *iter = (struct reftable_ref_iterator *)ref_iterator; @@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator) free(iter->exclude_patterns[i]); free(iter->exclude_patterns); } - free(iter); - return ITER_DONE; } static struct ref_iterator_vtable reftable_ref_iterator_vtable = { .advance = reftable_ref_iterator_advance, .peel = reftable_ref_iterator_peel, - .abort = reftable_ref_iterator_abort + .release = reftable_ref_iterator_release, }; static int qsort_strcmp(const void *va, const void *vb) @@ -2020,17 +2011,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) break; } - if (iter->err > 0) { - if (ref_iterator_abort(ref_iterator) != ITER_DONE) - return ITER_ERROR; + if (iter->err > 0) return ITER_DONE; - } - - if (iter->err < 0) { - ref_iterator_abort(ref_iterator); + if (iter->err < 0) return ITER_ERROR; - } - return ITER_OK; } @@ -2041,21 +2025,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE return -1; } -static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator) +static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator) { struct reftable_reflog_iterator *iter = (struct reftable_reflog_iterator *)ref_iterator; reftable_log_record_release(&iter->log); reftable_iterator_destroy(&iter->iter); strbuf_release(&iter->last_name); - free(iter); - return ITER_DONE; } static struct ref_iterator_vtable reftable_reflog_iterator_vtable = { .advance = reftable_reflog_iterator_advance, .peel = reftable_reflog_iterator_peel, - .abort = reftable_reflog_iterator_abort + .release = reftable_reflog_iterator_release, }; static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs, diff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c index 6b297bd753..8d46e8ba40 100644 --- a/t/helper/test-dir-iterator.c +++ b/t/helper/test-dir-iterator.c @@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv) printf("(%s) [%s] %s\n", diter->relative_path, diter->basename, diter->path.buf); } + dir_iterator_free(diter); if (iter_status != ITER_DONE) { printf("dir_iterator_advance failure\n"); From 82c39c6055b5340f0e50acbe01a97e51d3907fec Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:16 +0100 Subject: [PATCH 37/43] refs/iterator: provide infrastructure to re-seek iterators Reftable iterators need to be scrapped after they have either been exhausted or aren't useful to the caller anymore, and it is explicitly not possible to reuse them for iterations. But enabling for reuse of iterators may allow us to tune them by reusing internal state of an iterator. The reftable iterators for example can already be reused internally, but we're not able to expose this to any users outside of the reftable backend. Introduce a new `.seek` function in the ref iterator vtable that allows callers to seek an iterator multiple times. It is expected to be functionally the same as calling `refs_ref_iterator_begin()` with a different (or the same) prefix. Note that it is not possible to adjust parameters other than the seeked prefix for now, so exclude patterns, trimmed prefixes and flags will remain unchanged. We do not have a usecase for changing these parameters right now, but if we ever find one we can adapt accordingly. Implement the callback for trivial cases. The other iterators will be implemented in subsequent commits. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/debug.c | 11 +++++++++++ refs/iterator.c | 24 ++++++++++++++++++++++++ refs/refs-internal.h | 24 ++++++++++++++++++++++++ 3 files changed, 59 insertions(+) diff --git a/refs/debug.c b/refs/debug.c index a9786da4ba..5390fa9c18 100644 --- a/refs/debug.c +++ b/refs/debug.c @@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator) return res; } +static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct debug_ref_iterator *diter = + (struct debug_ref_iterator *)ref_iterator; + int res = diter->iter->vtable->seek(diter->iter, prefix); + trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res); + return res; +} + static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator) static struct ref_iterator_vtable debug_ref_iterator_vtable = { .advance = debug_ref_iterator_advance, + .seek = debug_ref_iterator_seek, .peel = debug_ref_iterator_peel, .release = debug_ref_iterator_release, }; diff --git a/refs/iterator.c b/refs/iterator.c index d61474cba7..ea4db59481 100644 --- a/refs/iterator.c +++ b/refs/iterator.c @@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator) return ref_iterator->vtable->advance(ref_iterator); } +int ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + return ref_iterator->vtable->seek(ref_iterator, prefix); +} + int ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) return ITER_DONE; } +static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, + const char *prefix UNUSED) +{ + return 0; +} + static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED, struct object_id *peeled UNUSED) { @@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED) static struct ref_iterator_vtable empty_ref_iterator_vtable = { .advance = empty_ref_iterator_advance, + .seek = empty_ref_iterator_seek, .peel = empty_ref_iterator_peel, .release = empty_ref_iterator_release, }; @@ -373,6 +386,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) return ok; } +static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct prefix_ref_iterator *iter = + (struct prefix_ref_iterator *)ref_iterator; + free(iter->prefix); + iter->prefix = xstrdup_or_null(prefix); + return ref_iterator_seek(iter->iter0, prefix); +} + static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -392,6 +415,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator) static struct ref_iterator_vtable prefix_ref_iterator_vtable = { .advance = prefix_ref_iterator_advance, + .seek = prefix_ref_iterator_seek, .peel = prefix_ref_iterator_peel, .release = prefix_ref_iterator_release, }; diff --git a/refs/refs-internal.h b/refs/refs-internal.h index 7d3bab654b..e5862757a7 100644 --- a/refs/refs-internal.h +++ b/refs/refs-internal.h @@ -327,6 +327,22 @@ struct ref_iterator { */ int ref_iterator_advance(struct ref_iterator *ref_iterator); +/* + * Seek the iterator to the first reference with the given prefix. + * The prefix is matched as a literal string, without regard for path + * separators. If prefix is NULL or the empty string, seek the iterator to the + * first reference again. + * + * This function is expected to behave as if a new ref iterator with the same + * prefix had been created, but allows reuse of iterators and thus may allow + * the backend to optimize. Parameters other than the prefix that have been + * passed when creating the iterator will remain unchanged. + * + * Returns 0 on success, a negative error code otherwise. + */ +int ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix); + /* * If possible, peel the reference currently being viewed by the * iterator. Return 0 on success. @@ -445,6 +461,13 @@ void base_ref_iterator_init(struct ref_iterator *iter, */ typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator); +/* + * Seek the iterator to the first reference matching the given prefix. Should + * behave the same as if a new iterator was created with the same prefix. + */ +typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator, + const char *prefix); + /* * Peels the current ref, returning 0 for success or -1 for failure. */ @@ -459,6 +482,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator); struct ref_iterator_vtable { ref_iterator_advance_fn *advance; + ref_iterator_seek_fn *seek; ref_iterator_peel_fn *peel; ref_iterator_release_fn *release; }; From 9821d90f13c6442022bbbcb2d96f1b29aad76503 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:17 +0100 Subject: [PATCH 38/43] refs/iterator: implement seeking for merged iterators Implement seeking on merged iterators. The implementation is rather straight forward, with the only exception that we must not deallocate the underlying iterators once they have been exhausted. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/iterator.c | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/refs/iterator.c b/refs/iterator.c index ea4db59481..766d96e795 100644 --- a/refs/iterator.c +++ b/refs/iterator.c @@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator) struct merge_ref_iterator { struct ref_iterator base; - struct ref_iterator *iter0, *iter1; + struct ref_iterator *iter0, *iter0_owned; + struct ref_iterator *iter1, *iter1_owned; ref_iterator_select_fn *select; void *cb_data; @@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) if (!iter->current) { /* Initialize: advance both iterators to their first entries */ if ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) { - ref_iterator_free(iter->iter0); iter->iter0 = NULL; if (ok == ITER_ERROR) goto error; } if ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) { - ref_iterator_free(iter->iter1); iter->iter1 = NULL; if (ok == ITER_ERROR) goto error; @@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) * entry: */ if ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) { - ref_iterator_free(*iter->current); *iter->current = NULL; if (ok == ITER_ERROR) goto error; @@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) if (selection & ITER_SKIP_SECONDARY) { if ((ok = ref_iterator_advance(*secondary)) != ITER_OK) { - ref_iterator_free(*secondary); *secondary = NULL; if (ok == ITER_ERROR) goto error; @@ -226,6 +223,28 @@ error: return ITER_ERROR; } +static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct merge_ref_iterator *iter = + (struct merge_ref_iterator *)ref_iterator; + int ret; + + iter->current = NULL; + iter->iter0 = iter->iter0_owned; + iter->iter1 = iter->iter1_owned; + + ret = ref_iterator_seek(iter->iter0, prefix); + if (ret < 0) + return ret; + + ret = ref_iterator_seek(iter->iter1, prefix); + if (ret < 0) + return ret; + + return 0; +} + static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator) { struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator; - ref_iterator_free(iter->iter0); - ref_iterator_free(iter->iter1); + ref_iterator_free(iter->iter0_owned); + ref_iterator_free(iter->iter1_owned); } static struct ref_iterator_vtable merge_ref_iterator_vtable = { .advance = merge_ref_iterator_advance, + .seek = merge_ref_iterator_seek, .peel = merge_ref_iterator_peel, .release = merge_ref_iterator_release, }; @@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin( */ base_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable); - iter->iter0 = iter0; - iter->iter1 = iter1; + iter->iter0 = iter->iter0_owned = iter0; + iter->iter1 = iter->iter1_owned = iter1; iter->select = select; iter->cb_data = cb_data; iter->current = NULL; From 53de20c931faabdb6fa9a30d949266b2a7471497 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:18 +0100 Subject: [PATCH 39/43] refs/iterator: implement seeking for reftable iterators Implement seeking of reftable iterators. As the low-level reftable iterators already support seeking this change is straight-forward. Two notes though: - We do not support seeking on reflog iterators. It is unclear what seeking would even look like in this context, as you typically would want to seek to a specific entry in the reflog for a specific ref. There is currently no use case for this, but if one arises in the future, we can still implement seeking at that later point. - We start to check whether `reftable_stack_init_ref_iterator()` is successful. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 2d5f4afe6b..c8f86da731 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -547,7 +547,7 @@ struct reftable_ref_iterator { struct reftable_ref_record ref; struct object_id oid; - const char *prefix; + char *prefix; size_t prefix_len; char **exclude_patterns; size_t exclude_patterns_index; @@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) return ITER_OK; } +static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct reftable_ref_iterator *iter = + (struct reftable_ref_iterator *)ref_iterator; + + free(iter->prefix); + iter->prefix = xstrdup_or_null(prefix); + iter->prefix_len = prefix ? strlen(prefix) : 0; + iter->err = reftable_iterator_seek_ref(&iter->iter, prefix); + + return iter->err; +} + static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator) free(iter->exclude_patterns[i]); free(iter->exclude_patterns); } + free(iter->prefix); } static struct ref_iterator_vtable reftable_ref_iterator_vtable = { .advance = reftable_ref_iterator_advance, + .seek = reftable_ref_iterator_seek, .peel = reftable_ref_iterator_peel, .release = reftable_ref_iterator_release, }; @@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_ iter = xcalloc(1, sizeof(*iter)); base_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable); - iter->prefix = prefix; - iter->prefix_len = prefix ? strlen(prefix) : 0; iter->base.oid = &iter->oid; iter->flags = flags; iter->refs = refs; @@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_ if (ret) goto done; - reftable_stack_init_ref_iterator(stack, &iter->iter); - ret = reftable_iterator_seek_ref(&iter->iter, prefix); + ret = reftable_stack_init_ref_iterator(stack, &iter->iter); + if (ret) + goto done; + + ret = reftable_ref_iterator_seek(&iter->base, prefix); if (ret) goto done; @@ -2018,6 +2035,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) return ITER_OK; } +static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, + const char *prefix UNUSED) +{ + BUG("reftable reflog iterator cannot be seeked"); + return -1; +} + static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED, struct object_id *peeled UNUSED) { @@ -2036,6 +2060,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator) static struct ref_iterator_vtable reftable_reflog_iterator_vtable = { .advance = reftable_reflog_iterator_advance, + .seek = reftable_reflog_iterator_seek, .peel = reftable_reflog_iterator_peel, .release = reftable_reflog_iterator_release, }; From 84e656919cb7237f1b11a948974d0591d9d3434f Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:19 +0100 Subject: [PATCH 40/43] refs/iterator: implement seeking for ref-cache iterators Implement seeking of ref-cache iterators. This is done by splitting most of the logic to seek iterators out of `cache_ref_iterator_begin()` and putting it into `cache_ref_iterator_seek()` so that we can reuse the logic. Note that we cannot use the optimization anymore where we return an empty ref iterator when there aren't any references, as otherwise it wouldn't be possible to reseek the iterator to a different prefix that may exist. This shouldn't be much of a performance concern though as we now start to bail out early in case `advance()` sees that there are no more directories to be searched. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/ref-cache.c | 81 +++++++++++++++++++++++++++++++----------------- 1 file changed, 52 insertions(+), 29 deletions(-) diff --git a/refs/ref-cache.c b/refs/ref-cache.c index 6457e02c1e..c1f1bab1d5 100644 --- a/refs/ref-cache.c +++ b/refs/ref-cache.c @@ -362,9 +362,7 @@ struct cache_ref_iterator { struct ref_iterator base; /* - * The number of levels currently on the stack. This is always - * at least 1, because when it becomes zero the iteration is - * ended and this struct is freed. + * The number of levels currently on the stack. */ size_t levels_nr; @@ -376,7 +374,7 @@ struct cache_ref_iterator { * The prefix is matched textually, without regard for path * component boundaries. */ - const char *prefix; + char *prefix; /* * A stack of levels. levels[0] is the uppermost level that is @@ -389,6 +387,9 @@ struct cache_ref_iterator { struct cache_ref_iterator_level *levels; struct repository *repo; + struct ref_cache *cache; + + int prime_dir; }; static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) @@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) struct cache_ref_iterator *iter = (struct cache_ref_iterator *)ref_iterator; + if (!iter->levels_nr) + return ITER_DONE; + while (1) { struct cache_ref_iterator_level *level = &iter->levels[iter->levels_nr - 1]; @@ -444,6 +448,41 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) } } +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct cache_ref_iterator *iter = + (struct cache_ref_iterator *)ref_iterator; + struct cache_ref_iterator_level *level; + struct ref_dir *dir; + + dir = get_ref_dir(iter->cache->root); + if (prefix && *prefix) + dir = find_containing_dir(dir, prefix); + if (!dir) { + iter->levels_nr = 0; + return 0; + } + + if (iter->prime_dir) + prime_ref_dir(dir, prefix); + iter->levels_nr = 1; + level = &iter->levels[0]; + level->index = -1; + level->dir = dir; + + if (prefix && *prefix) { + free(iter->prefix); + iter->prefix = xstrdup(prefix); + level->prefix_state = PREFIX_WITHIN_DIR; + } else { + FREE_AND_NULL(iter->prefix); + level->prefix_state = PREFIX_CONTAINS_DIR; + } + + return 0; +} + static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -456,12 +495,13 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator) { struct cache_ref_iterator *iter = (struct cache_ref_iterator *)ref_iterator; - free((char *)iter->prefix); + free(iter->prefix); free(iter->levels); } static struct ref_iterator_vtable cache_ref_iterator_vtable = { .advance = cache_ref_iterator_advance, + .seek = cache_ref_iterator_seek, .peel = cache_ref_iterator_peel, .release = cache_ref_iterator_release, }; @@ -471,39 +511,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache, struct repository *repo, int prime_dir) { - struct ref_dir *dir; struct cache_ref_iterator *iter; struct ref_iterator *ref_iterator; - struct cache_ref_iterator_level *level; - - dir = get_ref_dir(cache->root); - if (prefix && *prefix) - dir = find_containing_dir(dir, prefix); - if (!dir) - /* There's nothing to iterate over. */ - return empty_ref_iterator_begin(); - - if (prime_dir) - prime_ref_dir(dir, prefix); CALLOC_ARRAY(iter, 1); ref_iterator = &iter->base; base_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable); ALLOC_GROW(iter->levels, 10, iter->levels_alloc); - iter->levels_nr = 1; - level = &iter->levels[0]; - level->index = -1; - level->dir = dir; - - if (prefix && *prefix) { - iter->prefix = xstrdup(prefix); - level->prefix_state = PREFIX_WITHIN_DIR; - } else { - level->prefix_state = PREFIX_CONTAINS_DIR; - } - iter->repo = repo; + iter->cache = cache; + iter->prime_dir = prime_dir; + + if (cache_ref_iterator_seek(&iter->base, prefix) < 0) { + ref_iterator_free(&iter->base); + return NULL; + } return ref_iterator; } From 22600c045298f4f3664f94f6cbbf337903c72e82 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:20 +0100 Subject: [PATCH 41/43] refs/iterator: implement seeking for packed-ref iterators Implement seeking of `packed-ref` iterators. The implementation is again straight forward, except that we cannot continue to use the prefix iterator as we would otherwise not be able to reseek the iterator anymore in case one first asks for an empty and then for a non-empty prefix. Instead, we open-code the logic to in `advance()`. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/packed-backend.c | 65 ++++++++++++++++++++++++++++--------------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 38a1956d1a..f4c82ba2c7 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -819,6 +819,8 @@ struct packed_ref_iterator { struct snapshot *snapshot; + char *prefix; + /* The current position in the snapshot's buffer: */ const char *pos; @@ -841,11 +843,9 @@ struct packed_ref_iterator { }; /* - * Move the iterator to the next record in the snapshot, without - * respect for whether the record is actually required by the current - * iteration. Adjust the fields in `iter` and return `ITER_OK` or - * `ITER_DONE`. This function does not free the iterator in the case - * of `ITER_DONE`. + * Move the iterator to the next record in the snapshot. Adjust the fields in + * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the + * iterator in the case of `ITER_DONE`. */ static int next_record(struct packed_ref_iterator *iter) { @@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) int ok; while ((ok = next_record(iter)) == ITER_OK) { + const char *refname = iter->base.refname; + const char *prefix = iter->prefix; + if (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY && !is_per_worktree_ref(iter->base.refname)) continue; @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) &iter->oid, iter->flags)) continue; + while (prefix && *prefix) { + if (*refname < *prefix) + BUG("packed-refs backend yielded reference preceding its prefix"); + else if (*refname > *prefix) + return ITER_DONE; + prefix++; + refname++; + } + return ITER_OK; } return ok; } +static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct packed_ref_iterator *iter = + (struct packed_ref_iterator *)ref_iterator; + const char *start; + + if (prefix && *prefix) + start = find_reference_location(iter->snapshot, prefix, 0); + else + start = iter->snapshot->start; + + free(iter->prefix); + iter->prefix = xstrdup_or_null(prefix); + iter->pos = start; + iter->eof = iter->snapshot->eof; + + return 0; +} + static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator) (struct packed_ref_iterator *)ref_iterator; strbuf_release(&iter->refname_buf); free(iter->jump); + free(iter->prefix); release_snapshot(iter->snapshot); } static struct ref_iterator_vtable packed_ref_iterator_vtable = { .advance = packed_ref_iterator_advance, + .seek = packed_ref_iterator_seek, .peel = packed_ref_iterator_peel, .release = packed_ref_iterator_release, }; @@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin( { struct packed_ref_store *refs; struct snapshot *snapshot; - const char *start; struct packed_ref_iterator *iter; struct ref_iterator *ref_iterator; unsigned int required_flags = REF_STORE_READ; @@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin( */ snapshot = get_snapshot(refs); - if (prefix && *prefix) - start = find_reference_location(snapshot, prefix, 0); - else - start = snapshot->start; - - if (start == snapshot->eof) - return empty_ref_iterator_begin(); - CALLOC_ARRAY(iter, 1); ref_iterator = &iter->base; base_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable); @@ -1130,19 +1155,15 @@ static struct ref_iterator *packed_ref_iterator_begin( iter->snapshot = snapshot; acquire_snapshot(snapshot); - - iter->pos = start; - iter->eof = snapshot->eof; strbuf_init(&iter->refname_buf, 0); - iter->base.oid = &iter->oid; - iter->repo = ref_store->repo; iter->flags = flags; - if (prefix && *prefix) - /* Stop iteration after we've gone *past* prefix: */ - ref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0); + if (packed_ref_iterator_seek(&iter->base, prefix) < 0) { + ref_iterator_free(&iter->base); + return NULL; + } return ref_iterator; } From a95da5c8aec733c5b2e051bdfaa451f11570e87a Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:21 +0100 Subject: [PATCH 42/43] refs/iterator: implement seeking for files iterators Implement seeking for "files" iterators. As we simply use a ref-cache iterator under the hood the implementation is straight-forward. Note that we do not implement seeking on reflog iterators, same as with the "reftable" backend. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs/files-backend.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/refs/files-backend.c b/refs/files-backend.c index e97a267ad6..5f921e85eb 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -918,6 +918,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) return ok; } +static int files_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *prefix) +{ + struct files_ref_iterator *iter = + (struct files_ref_iterator *)ref_iterator; + return ref_iterator_seek(iter->iter0, prefix); +} + static int files_ref_iterator_peel(struct ref_iterator *ref_iterator, struct object_id *peeled) { @@ -936,6 +944,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator) static struct ref_iterator_vtable files_ref_iterator_vtable = { .advance = files_ref_iterator_advance, + .seek = files_ref_iterator_seek, .peel = files_ref_iterator_peel, .release = files_ref_iterator_release, }; @@ -2294,6 +2303,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) return ok; } +static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, + const char *prefix UNUSED) +{ + BUG("ref_iterator_seek() called for reflog_iterator"); +} + static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED, struct object_id *peeled UNUSED) { @@ -2309,6 +2324,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator) static struct ref_iterator_vtable files_reflog_iterator_vtable = { .advance = files_reflog_iterator_advance, + .seek = files_reflog_iterator_seek, .peel = files_reflog_iterator_peel, .release = files_reflog_iterator_release, }; From 87d297f48367737444810f8c3e76ef88cb6aa4e3 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 12 Mar 2025 16:56:22 +0100 Subject: [PATCH 43/43] refs: reuse iterators when determining refname availability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When verifying whether refnames are available we have to verify whether any reference exists that is nested under the current reference. E.g. given a reference "refs/heads/foo", we must make sure that there is no other reference "refs/heads/foo/*". This check is performed using a ref iterator with the prefix set to the nested reference namespace. Until now it used to not be possible to reseek iterators, so we always had to reallocate the iterator for every single reference we're about to check. This keeps us from reusing state that the iterator may have and that may make it work more efficiently. Refactor the logic to reseek iterators. This leads to a sizeable speedup with the "reftable" backend: Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~) Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms] Range (min … max): 38.4 ms … 42.0 ms 62 runs Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms] Range (min … max): 29.8 ms … 34.3 ms 74 runs Summary update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~) The "files" backend doesn't really show a huge impact: Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~) Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms] Range (min … max): 384.6 ms … 404.5 ms 10 runs Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms] Range (min … max): 377.0 ms … 397.7 ms 10 runs Summary update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~) This is mostly because it is way slower to begin with because it has to create a separate file for each new reference, so the milliseconds we shave off by reseeking the iterator doesn't really translate into a significant relative improvement. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- refs.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/refs.c b/refs.c index eeb8fb1021..79d5a8b8d4 100644 --- a/refs.c +++ b/refs.c @@ -2564,8 +2564,13 @@ int refs_verify_refnames_available(struct ref_store *refs, if (!initial_transaction) { int ok; - iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0, - DO_FOR_EACH_INCLUDE_BROKEN); + if (!iter) { + iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0, + DO_FOR_EACH_INCLUDE_BROKEN); + } else if (ref_iterator_seek(iter, dirname.buf) < 0) { + goto cleanup; + } + while ((ok = ref_iterator_advance(iter)) == ITER_OK) { if (skip && string_list_has_string(skip, iter->refname)) @@ -2578,9 +2583,6 @@ int refs_verify_refnames_available(struct ref_store *refs, if (ok != ITER_DONE) BUG("error while iterating over references"); - - ref_iterator_free(iter); - iter = NULL; } extra_refname = find_descendant_ref(dirname.buf, extras, skip);