kernel/git - git - PowerEL Git System

Commit Graph

Author	SHA1	Message	Date
Ævar Arnfjörð Bjarmason	edb6a17c36	Makefile: NO_OPENSSL=1 should no longer imply BLK_SHA1=1 Use the collision detecting SHA-1 implementation by default even when NO_OPENSSL is set. Setting NO_OPENSSL=UnfortunatelyYes has implied BLK_SHA1=1 ever since the former was introduced in `dd53c7ab29` (Support for NO_OPENSSL, 2005-07-29). That implication should have been removed when the default SHA-1 implementation changed from OpenSSL to DC_SHA1 in `e6b07da278` (Makefile: make DC_SHA1 the default, 2017-03-17). Finish what that commit started by removing the BLK_SHA1 fallback setting so the default DC_SHA1 implementation will be used. Helped-by: Jonathan Nieder <jrnieder@gmail.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Johannes Schindelin	ed32b788c0	version --build-options: report commit, too, if possible In particular when local tags are used (or tags that are pushed to some fork) to build Git, it is very hard to figure out from which particular revision a particular Git executable was built. It gets worse when those tags are deleted, or even updated. Let's just report an exact, unabbreviated commit name in our build options. We need to be careful, though, to report when the current commit cannot be determined, e.g. when building from a tarball without any associated Git repository. This could be the case also when extracting Git's source code into an unrelated Git worktree. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Eric Sunshine	b22894049f	version --build-options: also report host CPU It can be helpful for bug reports to include information about the environment in which the bug occurs. "git version --build-options" can help to supplement this information. In addition to the size of 'long' already reported by --build-options, also report the host's CPU type. Example output: $ git version --build-options git version 2.9.3.windows.2.826.g06c0f2f cpu: x86_64 sizeof-long: 4 New Makefile variable HOST_CPU supports cross-compiling. Suggested-by: Adric Norris <landstander668@gmail.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Jonathan Tan	ddd3e31242	decorate: clean up and document API Improve the names of the identifiers in decorate.h, document them, and add an example of how to use these functions. The example is compiled and run as part of the test suite. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Jeff King	3f824e91c8	t/Makefile: introduce TEST_SHELL_PATH You may want to run the test suite with a different shell than you use to build Git. For instance, you may build with SHELL_PATH=/bin/sh (because it's faster, or it's what you expect to exist on systems where the build will be used) but want to run the test suite with bash (e.g., since that allows using "-x" reliably across the whole test suite). There's currently no good way to do this. You might think that doing two separate make invocations, like: make && make -C t SHELL_PATH=/bin/bash would work. And it _almost_ does. The second make will see our bash SHELL_PATH, and we'll use that to run the individual test scripts (or tell prove to use it to do so). So far so good. But this breaks down when "--tee" or "--verbose-log" is used. Those options cause the test script to actually re-exec itself using $SHELL_PATH. But wait, wouldn't our second make invocation have set SHELL_PATH correctly in the environment? Yes, but test-lib.sh sources GIT-BUILD-OPTIONS, which we built during the first "make". And that overrides the environment, giving us the original SHELL_PATH again. Let's introduce a new variable that lets you specify a specific shell to be run for the test scripts. Note that we have to touch both the main and t/ Makefiles, since we have to record it in GIT-BUILD-OPTIONS in one, and use it in the latter. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Thomas Gummerer	7c85a87c54	checkout: factor out functions to new lib file Factor the functions out, so they can be re-used from other places. In particular these functions will be re-used in builtin/worktree.c to make git worktree add dwim more. While there add some docs to the function. Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Jeff Hostetler	25ec7bcac0	list-objects: filter objects in traverse_commit_list Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted from the traversal. Update traverse_commit_list() to be a wrapper for the above with a null filter to minimize the number of callers that needed to be changed. Object filtering will be used in a future commit by rev-list and pack-objects for partial clone and fetch to omit unwanted objects from the result. traverse_bitmap_commit_list() does not work with filtering. If a packfile bitmap is present, it will not be used. It should be possible to extend such support in the future (at least to simple filters that do not require object pathnames), but that is beyond the scope of this patch series. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Reviewed-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Anders Kaseorg	474642b4a4	git-gui: sort entries in optimized tclIndex auto_mkindex expands wildcards in directory order, which depends on the underlying filesystem. To improve build reproducibility, sort the list of *.tcl files in the Makefile. The unoptimized loading case was previously fixed in gitgui-0.21.0~14 (git-gui: sort entries in tclIndex, 2015-01-26). Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Johannes Schindelin	39bb86b4e5	mingw: include the full version information in the resources This fixes https://github.com/git-for-windows/git/issues/723 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Brandon Williams	373d70efb2	protocol: introduce protocol extension mechanisms Create protocol.{c,h} and provide functions which future servers and clients can use to determine which protocol to use or is being used. Also introduce the 'GIT_PROTOCOL' environment variable which will be used to communicate a colon separated list of keys with optional values to a server. Unknown keys and values must be tolerated. This mechanism is used to communicate which version of the wire protocol a client would like to use with a server. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Ben Peart	14527b3002	fsmonitor: add a performance test Add a test utility (test-drop-caches) that flushes all changes to disk then drops file system cache on Windows, Linux, and OSX. Add a perf test (p7519-fsmonitor.sh) for fsmonitor. By default, the performance test will utilize the Watchman file system monitor if it is installed. If Watchman is not installed, it will use a dummy integration script that does not report any new or modified files. The dummy script has very little overhead which provides optimistic results. The performance test will also use the untracked cache feature if it is available as fsmonitor uses it to speed up scanning for untracked files. There are 4 environment variables that can be used to alter the default behavior of the performance test: GIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex GIT_PERF_7519_FSMONITOR: used to configure core.fsmonitor GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests The big win for using fsmonitor is the elimination of the need to scan the working directory looking for changed and untracked files. If the file information is all cached in RAM, the benefits are reduced. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Ben Peart	dd3551f491	fsmonitor: add a test tool to dump the index extension Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor index extension. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Ben Peart	883e248b8a	fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files. When the index is read from disk, the fsmonitor index extension is used to flag the last known potentially dirty index entries. The registered core.fsmonitor command is called with the time the index was last updated and returns the list of files changed since that time. This list is used to flag any additional dirty cache entries and untracked cache directories. We can then use this valid state to speed up preload_index(), ie_match_stat(), and refresh_cache_ent() as they do not need to lstat() files to detect potential changes for those entries marked CE_FSMONITOR_VALID. In addition, if the untracked cache is turned on valid_cached_dir() can skip checking directories for new or changed files as fsmonitor will invalidate the cache only for those directories that have been identified as having potential changes. To keep the CE_FSMONITOR_VALID state accurate during git operations; when git updates a cache entry to match the current state on disk, it will now set the CE_FSMONITOR_VALID bit. Inversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit is cleared and the corresponding untracked cache directory is marked invalid. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Jonathan Tan	9e6fabde82	oidmap: map with OID as key This is similar to using the hashmap in hashmap.c, but with an easier-to-use API. In particular, custom entry comparisons no longer need to be written, and lookups can be done without constructing a temporary entry structure. This is implemented as a thin wrapper over the hashmap API. In particular, this means that there is an additional 4-byte overhead due to the fact that the first 4 bytes of the hash is redundantly stored. For now, I'm taking the simpler approach, but if need be, we can reimplement oidmap without affecting the callers significantly. oidset has been updated to use oidmap. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Michael Haggerty	5b633610ec	packed_ref_cache: keep the `packed-refs` file mmapped if possible Keep a copy of the `packed-refs` file contents in memory for as long as a `packed_ref_cache` object is in use: * If the system allows it, keep the `packed-refs` file mmapped. * If not (either because the system doesn't support `mmap()` at all, or because a file that is currently mmapped cannot be replaced via `rename()`), then make a copy of the file's contents in heap-allocated space, and keep that around instead. We base the choice of behavior on a new build-time switch, `MMAP_PREVENTS_DELETE`. By default, this switch is set for Windows variants. After this commit, `MMAP_NONE` and `MMAP_TEMPORARY` are still handled identically. But the next commit will introduce a difference. This whole change is still pointless, because we only read the `packed-refs` file contents immediately after instantiating the `packed_ref_cache`. But that will soon change. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Jeff King	0e5bba53af	add UNLEAK annotation for reducing leak false positives It's a common pattern in git commands to allocate some memory that should last for the lifetime of the program and then not bother to free it, relying on the OS to throw it away. This keeps the code simple, and it's fast (we don't waste time traversing structures or calling free at the end of the program). But it also triggers warnings from memory-leak checkers like valgrind or LSAN. They know that the memory was still allocated at program exit, but they don't know _when_ the leaked memory stopped being useful. If it was early in the program, then it's probably a real and important leak. But if it was used right up until program exit, it's not an interesting leak and we'd like to suppress it so that we can see the real leaks. This patch introduces an UNLEAK() macro that lets us do so. To understand its design, let's first look at some of the alternatives. Unfortunately the suppression systems offered by leak-checking tools don't quite do what we want. A leak-checker basically knows two things: 1. Which blocks were allocated via malloc, and the callstack during the allocation. 2. Which blocks were left un-freed at the end of the program (and which are unreachable, but more on that later). Their suppressions work by mentioning the function or callstack of a particular allocation, and marking it as OK to leak. So imagine you have code like this: int cmd_foo(...) { /* this allocates some memory / char p = some_function(); printf("%s", p); return 0; } You can say "ignore allocations from some_function(), they're not leaks". But that's not right. That function may be called elsewhere, too, and we would potentially want to know about those leaks. So you can say "ignore the callstack when main calls some_function". That works, but your annotations are brittle. In this case it's only two functions, but you can imagine that the actual allocation is much deeper. If any of the intermediate code changes, you have to update the suppression. What we _really_ want to say is that "the value assigned to p at the end of the function is not a real leak". But leak-checkers can't understand that; they don't know about "p" in the first place. However, we can do something a little bit tricky if we make some assumptions about how leak-checkers work. They generally don't just report all un-freed blocks. That would report even globals which are still accessible when the leak-check is run. Instead they take some set of memory (like BSS) as a root and mark it as "reachable". Then they scan the reachable blocks for anything that looks like a pointer to a malloc'd block, and consider that block reachable. And then they scan those blocks, and so on, transitively marking anything reachable from a global as "not leaked" (or at least leaked in a different category). So we can mark the value of "p" as reachable by putting it into a variable with program lifetime. One way to do that is to just mark "p" as static. But that actually affects the run-time behavior if the function is called twice (you aren't likely to call main() twice, but some of our cmd_() functions are called from other commands). Instead, we can trick the leak-checker by putting the value into _any_ reachable bytes. This patch keeps a global linked-list of bytes copied from "unleaked" variables. That list is reachable even at program exit, which confers recursive reachability on whatever values we unleak. In other words, you can do: int cmd_foo(...) { char p = some_function(); printf("%s", p); UNLEAK(p); return 0; } to annotate "p" and suppress the leak report. But wait, couldn't we just say "free(p)"? In this toy example, yes. But UNLEAK()'s byte-copying strategy has several advantages over actually freeing the memory: 1. It's recursive across structures. In many cases our "p" is not just a pointer, but a complex struct whose fields may have been allocated by a sub-function. And in some cases (e.g., dir_struct) we don't even have a function which knows how to free all of the struct members. By marking the struct itself as reachable, that confers reachability on any pointers it contains (including those found in embedded structs, or reachable by walking heap blocks recursively. 2. It works on cases where we're not sure if the value is allocated or not. For example: char p = argc > 1 ? argv[1] : some_function(); It's safe to use UNLEAK(p) here, because it's not freeing any memory. In the case that we're pointing to argv here, the reachability checker will just ignore our bytes. 3. Likewise, it works even if the variable has _already_ been freed. We're just copying the pointer bytes. If the block has been freed, the leak-checker will skip over those bytes as uninteresting. 4. Because it's not actually freeing memory, you can UNLEAK() before we are finished accessing the variable. This is helpful in cases like this: char p = some_function(); return another_function(p); Writing this with free() requires: int ret; char p = some_function(); ret = another_function(p); free(p); return ret; But with unleak we can just write: char p = some_function(); UNLEAK(p); return another_function(p); This patch adds the UNLEAK() macro and enables it automatically when Git is compiled with SANITIZE=leak. In normal builds it's a noop, so we pay no runtime cost. It also adds some UNLEAK() annotations to show off how the feature works. On top of other recent leak fixes, these are enough to get t0000 and t0001 to pass when compiled with LSAN. Note the case in commit.c which actually converts a strbuf_release() into an UNLEAK. This code was already non-leaky, but the free didn't do anything useful, since we're exiting. Converting it to an annotation means that non-leak-checking builds pay no runtime cost. The cost is minimal enough that it's probably not worth going on a crusade to convert these kinds of frees to UNLEAKS. I did it here for consistency with the "sb" leak (though it would have been equally correct to go the other way, and turn them both into strbuf_release() calls). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jonathan Tan	4f39cd821d	pack: move pack name-related functions Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns. Create a new file, packfile.c, to hold all packfile-related functions currently in sha1_file.c. It has a corresponding header packfile.h. In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jonathan Nieder	b8f43b120b	vcs-svn: move remaining repo_tree functions to fast_export.h These used to be for manipulating the in-memory repo_tree structure, but nowadays they are convenience wrappers to handle a few git-vs-svn mismatches: 1. Git does not track empty directories but Subversion does. When looking up a path in git that Subversion thinks exists and finding nothing, we can safely assume that the path represents a directory. This is needed when a later Subversion revision modifies that directory. 2. Subversion allows deleting a file by copying. In Git fast-import we have to handle that more explicitly as a deletion. These are details of the tool's interaction with git fast-import. Move them to fast_export.c, where other such details are handled. This way the function names do not start with a repo_ prefix that would clash with the repository object introduced in v2.14.0-rc0~38^2~16 (repository: introduce the repository object, 2017-06-22) or an svn_ prefix that would clash with libsvn (in case someone wants to link this code with libsvn some day). Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Kevin Willford	3921a0b3c3	perf: add test for writing the index A performance test for writing the index to be able to determine if changes to allocating ondisk structure help. Signed-off-by: Kevin Willford <kewillf@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Takashi Iwai	3964cbbb5c	sha1dc: allow building with the external sha1dc library Some distros provide SHA1 collision-detect code as a shared library. It's the same code as we have in git tree (but may be with a different init default for hash), and git can link with it as well; at least, it may make maintenance easier, according to our security guys. This patch allows user to build git linking with the external sha1dc library instead of the built-in code. User needs to define DC_SHA1_EXTERNAL explicitly. As default without it, the built-in sha1dc code is used like before. Signed-off-by: Takashi Iwai <tiwai@suse.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Takashi Iwai	36f048c5e4	sha1dc: build git plumbing code more explicitly The plumbing code between sha1dc and git is defined in sha1dc_git.[ch], but these aren't compiled / included directly but only via the indirect inclusion from sha1dc code. This is slightly confusing when you try to trace the build flow. This patch brings the following changes for simplification: - Make sha1dc_git.c stand-alone and build from Makefile - sha1dc_git.h is the common header to include further sha1.h depending on the build condition - Move comments for plumbing codes from the header to definitions This is also meant as a preliminary work for further plumbing with external sha1dc shlib. Signed-off-by: Takashi Iwai <tiwai@suse.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Brandon Williams	2118805b92	Makefile: add style build rule Add the 'style' build rule which will run git-clang-format on the diff between HEAD and the current worktree. The result is a diff of suggested changes. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Johannes Schindelin	492595cfc7	git-gui (MinGW): make use of MSys2's msgfmt When Git for Windows was still based on MSys1, we had no gettext, ergo no msgfmt, either. Therefore, we introduced a small and simple Tcl script to perform the same task. However, with MSys2, we no longer need that because we have a proper msgfmt executable. Plus, the po2msg.sh script somehow manages to hang when run in parallel in Git for Windows' SDK (symptom: the Continuous Testing tasks timing out). Two reasons to use real msgfmt.exe instead. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Junio C Hamano	fc0fd5b23b	Makefile: help gettext tools to cope with our custom PRItime format We started using our own timestamp_t type and PRItime format specifier to go along with it, so that we can later change the underlying type and output format more easily, but this does not play well with gettext tools. Because gettext tools need to keep the *.po file portable across platforms, they have to special-case the format specifiers like PRIuMAX that are known types in inttypes.h, instead of letting CPP handle strings like "%" PRIuMAX " seconds ago" as an ordinary string concatenation. They fundamentally cannot do the same for our own custom type/format. Given that po/git.pot needs to be generated only once every release and by only one person, i.e. the l10n coordinator, let's update the Makefile rule to generate po/git.pot so that gettext tools are run on a munged set of sources in which all mentions of PRItime are replaced with PRIuMAX, which is what we happen to use right now. This way, developers do not have to care that PRItime does not play well with gettext, and translators do not have to care that we use our own PRItime. The credit for the idea to munge the source files goes to Dscho. Possible bugs are mine. Helped-by: Jiang Xin <worldhello.net@gmail.com> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
René Scharfe	425ca6710b	Makefile: allow combining UBSan with other sanitizers Multiple sanitizers can be specified as a comma-separated list. Set the flag NO_UNALIGNED_LOADS even if UndefinedBehaviorSanitizer is not the only sanitizer to build with. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	566cf0b3bd	Makefile: disable unaligned loads with UBSan The undefined behavior sanitizer complains about unaligned loads, even if they're OK for a particular platform in practice. It's possible that they _are_ a problem, of course, but since it's a known tradeoff the UBSan errors are just noise. Let's quiet it automatically by building with NO_UNALIGNED_LOADS when SANITIZE=undefined is in use. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	ddbc8a6d3e	Makefile: turn off -fomit-frame-pointer with sanitizers The ASan manual recommends disabling this optimization, as it can make the backtraces produced by the tool harder to follow (and since this is a test-debug build, we don't care about squeezing out every last drop of performance). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	56b5db30d0	Makefile: add helper for compiling with -fsanitize You can already build and test with ASan by doing: make CFLAGS=-fsanitize=address test but there are a few slight annoyances: 1. It's a little long to type. 2. It override your CFLAGS completely. You'd probably still want -O2, for instance. 3. It's a good idea to also turn off "recovery", which lets the program keep running after a problem is detected (with the intention of finding as many bugs as possible in a given run). Since Git's test suite should generally run without triggering any problems, it's better to abort immediately and fail the test when we do find an issue. With this patch, all of that happens automatically when you run: make SANITIZE=address test Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Junio C Hamano	cac87dc01d	sha1collisiondetection: automatically enable when submodule is populated If a user wants to experiment with the version of collision detecting sha1 from the submodule, the user needed to not just populate the submodule but also needed to turn the knob. A Makefile trick is easy enough to do so, so let's do this. When somebody with a copy of the submodule populated wants not to use it, that can be done by overriding it in config.mak or from the command line. Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	86cfd61e6b	sha1dc: optionally use sha1collisiondetection as a submodule Add an option to use the sha1collisiondetection library from the submodule in sha1collisiondetection/ instead of in the copy in the sha1dc/ directory. This allows us to try out the submodule in sha1collisiondetection without breaking the build for anyone who's not expecting them as we work out any kinks. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Brandon Williams	359efeffc1	repository: introduce the repository object Introduce the repository object 'struct repository' which can be used to hold all state pertaining to a git repository. Some of the benefits of object-ifying a repository are: 1. Make the code base more readable and easier to reason about. 2. Allow for working on multiple repositories, specifically submodules, within the same process. Currently the process for working on a submodule involves setting up an argv_array of options for a particular command and then launching a child process to execute the command in the context of the submodule. This is clunky and can require lots of little hacks in order to ensure correctness. Ideally it would be nice to simply pass a repository and an options struct to a command. 3. Eliminating reliance on global state will make it easier to enable the use of threading to improve performance. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Michael Haggerty	67be7c5a59	packed-backend: new module for handling packed references Now that the interface between `files_ref_store` and `packed_ref_store` is relatively narrow, move the latter into a new module, "refs/packed-backend.h" and "refs/packed-backend.c". It still doesn't quite implement the `ref_store` interface, but it will soon. This commit moves code around and adjusts its visibility, but doesn't change anything. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	3adf9fdecf	configure.ac: loosen FREAD_READS_DIRECTORIES test program We added an FREAD_READS_DIRECTORIES Makefile knob long ago in `cba22528f` (Add compat/fopen.c which returns NULL on attempt to open directory, 2008-02-08) to handle systems where reading from a directory returned garbage. This works by catching the problem at the fopen() stage and returning NULL. More recently, we found that there is a class of systems (including Linux) where fopen() succeeds but fread() fails. Since the solution is the same (having fopen return NULL), they use the same Makefile knob as of `e2d90fd1c` (config.mak.uname: set FREAD_READS_DIRECTORIES for Linux and FreeBSD, 2017-05-03). This works fine except for one thing: the autoconf test in configure.ac to set FREAD_READS_DIRECTORIES actually checks whether fread succeeds. Which means that on Linux systems, the knob isn't set (and we even override the config.mak.uname default). t1308 catches the failure. We can fix this by tweaking the autoconf test to cover both cases. In theory we might care about the distinction between the traditional "fread reads directories" case and the new "fopen opens directories". But since our solution catches the problem at the fopen stage either way, we don't actually need to know the difference. The "fopen" case is a superset. This does mean the FREAD_READS_DIRECTORIES name is slightly misleading. Probably FOPEN_OPENS_DIRECTORIES would be more accurate. But it would be disruptive to simply change the name (people's existing build configs would fail), and it's not worth the complexity of handling both. Let's just add a comment in the knob description. Reported-by: Øyvind A. Holm <sunny@sunbase.org> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	94da9193a6	grep: add support for PCRE v2 Add support for v2 of the PCRE API. This is a new major version of PCRE that came out in early 2015[1]. The regular expression syntax is the same, but while the API is similar, pretty much every function is either renamed or takes different arguments. Thus using it via entirely new functions makes sense, as opposed to trying to e.g. have one compile_pcre_pattern() that would call either PCRE v1 or v2 functions. Git can now be compiled with either USE_LIBPCRE1=YesPlease or USE_LIBPCRE2=YesPlease, with USE_LIBPCRE=YesPlease currently being a synonym for the former. Providing both is a compile-time error. With earlier patches to enable JIT for PCRE v1 the performance of the release versions of both libraries is almost exactly the same, with PCRE v2 being around 1% slower. However after I reported this to the pcre-dev mailing list[2] I got a lot of help with the API use from Zoltán Herczeg, he subsequently optimized some of the JIT functionality in v2 of the library. Running the p7820-grep-engines.sh performance test against the latest Subversion trunk of both, with both them and git compiled as -O3, and the test run against linux.git, gives the following results. Just the /perl/ tests shown: $ GIT_PERF_REPEAT_COUNT=30 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_COMMAND='grep -q LIBPCRE2 Makefile && make -j8 USE_LIBPCRE2=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre2/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre2/inst/lib \|\| make -j8 USE_LIBPCRE=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre/inst/lib' ./run HEAD~5 HEAD~ HEAD p7820-grep-engines.sh [...] Test HEAD~5 HEAD~ HEAD ----------------------------------------------------------------------------------------------------------------- 7820.3: perl grep 'how.to' 0.31(1.10+0.48) 0.21(0.35+0.56) -32.3% 0.21(0.34+0.55) -32.3% 7820.7: perl grep '^how to' 0.56(2.70+0.40) 0.24(0.64+0.52) -57.1% 0.20(0.28+0.60) -64.3% 7820.11: perl grep '[how] to' 0.56(2.66+0.38) 0.29(0.95+0.45) -48.2% 0.23(0.45+0.54) -58.9% 7820.15: perl grep '(e.t[^ ]\|v.ry) rare' 1.02(5.77+0.42) 0.31(1.02+0.54) -69.6% 0.23(0.50+0.54) -77.5% 7820.19: perl grep 'm(ú\|u)lt.b(æ\|y)te' 0.38(1.57+0.42) 0.27(0.85+0.46) -28.9% 0.21(0.33+0.57) -44.7% See commit ("perf: add a comparison test of grep regex engines", 2017-04-19) for details on the machine the above test run was executed on. Here HEAD~2 is git with PCRE v1 without JIT, HEAD~ is PCRE v1 with JIT, and HEAD is PCRE v2 (also with JIT). See previous commits of mine mentioning p7820-grep-engines.sh for more details on the test setup. For ease of readability, a different run just of HEAD~ (PCRE v1 with JIT v.s. PCRE v2), again with just the /perl/ tests shown: [...] Test HEAD~ HEAD ---------------------------------------------------------------------------------------- 7820.3: perl grep 'how.to' 0.21(0.42+0.52) 0.21(0.31+0.58) +0.0% 7820.7: perl grep '^how to' 0.25(0.65+0.50) 0.20(0.31+0.57) -20.0% 7820.11: perl grep '[how] to' 0.30(0.90+0.50) 0.23(0.46+0.53) -23.3% 7820.15: perl grep '(e.t[^ ]\|v.ry) rare' 0.30(1.19+0.38) 0.23(0.51+0.51) -23.3% 7820.19: perl grep 'm(ú\|u)lt.b(æ\|y)te' 0.27(0.84+0.48) 0.21(0.34+0.57) -22.2% I.e. the two are either neck-to-neck, but PCRE v2 usually pulls ahead, when it does it's around 20% faster. A brief note on thread safety: As noted in pcre2api(3) & pcre2jit(3) the compiled pattern can be shared between threads, but not some of the JIT context, however the grep threading support does all pattern & JIT compilation in separate threads, so this code doesn't need to concern itself with thread safety. See commit `63e7e9d8b6` ("git-grep: Learn PCRE", 2011-05-09) for the initial addition of PCRE v1. This change follows some of the same patterns it did (and which were discussed on list at the time), e.g. mocking up types with typedef instead of ifdef-ing them out when USE_LIBPCRE2 isn't defined. This adds some trivial memory use to the program, but makes the code look nicer. 1. https://lists.exim.org/lurker/message/20150105.162835.0666407a.en.html 2. https://lists.exim.org/lurker/thread/20170419.172322.833ee099.en.html Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	fb95e2e38d	grep: un-break building with PCRE >= 8.32 without --enable-jit Amend my change earlier in this series ("grep: add support for the PCRE v1 JIT API", 2017-04-11) to un-break the build on PCRE v1 versions later than 8.31 compiled without --enable-jit. As explained in that change and a later compatibility change in this series ("grep: un-break building with PCRE < 8.32", 2017-05-10) the pcre_jit_exec() function is a faster path to execute the JIT. Unfortunately there's no compatibility stub for that function compiled into the library if pcre_config(PCRE_CONFIG_JIT, &ret) would return 0, and no macro that can be used to check for it, so the only portable option to support builds without --enable-jit is via a new NO_LIBPCRE1_JIT=UnfortunatelyYes Makefile option[1]. Another option would be to make the JIT opt-in via USE_LIBPCRE1_JIT=YesPlease, after all it's not a default option of PCRE v1. I think it makes more sense to make it opt-out since even though it's not a default option, most packagers of PCRE seem to turn it on by default, with the notable exception of the MinGW package. Make the MinGW platform work by default by changing the build defaults to turn on NO_LIBPCRE1_JIT=UnfortunatelyYes. It is the only platform that turns on USE_LIBPCRE=YesPlease by default, see commit `df5218b4c3` ("config.mak.uname: support MSys2", 2016-01-13) for that change. 1. "How do I support pcre1 JIT on all versions?" (https://lists.exim.org/lurker/thread/20170601.103148.10253788.en.html) 2. https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-pcre/PKGBUILD (referenced from "Re: PCRE v2 compile error, was Re: What's cooking in git.git (May 2017, #01; Mon, 1)"; <alpine.DEB.2.20.1705021756530.3480@virtualbox>) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	68c7d2761d	test-lib: add a PTHREADS prerequisite Add a PTHREADS prerequisite which is false when git is compiled with NO_PTHREADS=YesPlease. There's lots of custom code that runs when threading isn't available, but before this prerequisite there was no way to test it. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	3485bea157	grep: change the internal PCRE macro names to be PCRE1 Change the internal USE_LIBPCRE define, & build options flag to use a naming convention ending in PCRE1, without changing the long-standing USE_LIBPCRE Makefile flag which enables this code. This is for preparation for libpcre2 support where having things like USE_LIBPCRE and USE_LIBPCRE2 in any more places than we absolutely need to for backwards compatibility with old Makefile arguments would be confusing. In some ways it would be better to change everything that now uses USE_LIBPCRE to use USE_LIBPCRE1, and to make specifying USE_LIBPCRE (or --with-pcre) an error. This would impose a one-time burden on packagers of git to s/USE_LIBPCRE/USE_LIBPCRE1/ in their build scripts. However I'd like to leave the door open to making USE_LIBPCRE=YesPlease eventually mean USE_LIBPCRE2=YesPlease, i.e. once PCRE v2 is ubiquitous enough that it makes sense to make it the default. This code and the USE_LIBPCRE Makefile argument was added in commit `63e7e9d8b6` ("git-grep: Learn PCRE", 2011-05-09). At the time there was no indication that the PCRE project would release an entirely new & incompatible API around 3 years later. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff Smith	f5dd754c36	blame: move origin-related methods to libgit Signed-off-by: Jeff Smith <whydoubt@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	a0103914c2	sha1dc: update from upstream Update sha1dc from the latest version by the upstream maintainer[1]. This version includes a commit of mine which allows for replacing the local modifications done to the upstream files in git.git with macro definitions to monkeypatch it in place. It also brings in a change[2] upstream made for the breakage 2.13.0 introduced on SPARC and other platforms that forbid unaligned access[3]. This means that the code customizations done since the initial import in commit `28dc98e343` ("sha1dc: add collision-detecting sha1 implementation", 2017-03-16) can be done purely via Makefile definitions and by including the content of our own sha1dc_git.[ch] in sha1dc/sha1.c via a macro. 1. `cc465543b3` 2. `33a694a9ee` 3. "Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default" (https://public-inbox.org/git/CACBZZX6nmKK8af0-UpjCKWV4R+hV-uk2xWXVA5U+_UQ3VXU03g@mail.gmail.com/) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	88b6197d0b	perf: add a GIT_PERF_MAKE_COMMAND for when *_MAKE_OPTS won't do Add a git GIT_PERF_MAKE_COMMAND variable to compliment the existing GIT_PERF_MAKE_OPTS facility. This allows specifying an arbitrary shell command to execute instead of 'make'. This is useful e.g. in cases where the name, semantics or defaults of a Makefile flag have changed over time. It can even be used to change the contents of the tree, useful for monkeypatching ancient versions of git to get them to build. This opens Pandora's box in some ways, it's now possible to "jailbreak" the perf environment and e.g. modify the source tree via this arbitrary instead of just issuing a custom "make" command, such a command has to be re-entrant in the sense that subsequent perf runs will re-use the possibly modified tree. It would be pointless to try to mitigate or work around that caveat in a tool purely aimed at Git developers, so this change makes no attempt to do so. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	072473e659	Makefile & configure: reword inaccurate comment about PCRE Reword an outdated & inaccurate comment which suggests that only git-grep can use PCRE. This comment was added back when PCRE support was initially added in commit `63e7e9d8b6` ("git-grep: Learn PCRE", 2011-05-09), and was true at the time. It hasn't been telling the full truth since git-log learned to use PCRE with --grep in commit `727b6fc3ed` ("log --grep: accept --basic-regexp and --perl-regexp", 2012-10-03), and more importantly is likely to get more inaccurate over time as more use is made of PCRE in other areas. Reword it to be more future-proof, and to more clearly explain that this enables user-initiated runtime behavior. Copy/pasting this so much in configure.ac is lame, these Makefile-like flags aren't even used by autoconf, just the corresponding --with[out]-* options. But copy/pasting the comments that make sense for the Makefile to configure.ac where they make less sense is the pattern everything else follows in that file. I'm not going to war against that as part of this change, just following the existing pattern. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ben Peart	99605d62e8	sub-process: move sub-process functions into separate files Move the sub-proces functions into sub-process.h/c. Add documentation for the new module in Documentation/technical/api-sub-process.txt Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Michael Haggerty	958f964691	refs: split `ref_cache` code into separate files The `ref_cache` code is currently too tightly coupled to `files-backend`, making the code harder to understand and making it awkward for new code to use `ref_cache` (as we indeed have planned). Start loosening that coupling by splitting `ref_cache` into a separate module. This commit moves code, adds declarations, and changes the visibility of some functions, but doesn't change any code. The modules are still too tightly coupled, but the situation will be improved in subsequent commits. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff Hostetler	a6db3fbb6e	read-cache: add strcmp_offset function Add strcmp_offset() function to also return the offset of the first change. Add unit test and helper to verify. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Nguyễn Thái Ngọc Duy	80f2a6097c	t/helper: add test-ref-store to test ref-store functions Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff Hostetler	e3482ccf27	test-online-cpus: helper to return cpu count Created helper executable to print the value of online_cpus() allowing multi-threaded tests to be skipped when appropriate. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Ævar Arnfjörð Bjarmason	c59c4939c2	perl: regenerate perl.mak if perl -V changes Change the perl/perl.mak build process so that the file is regenerated if the output of "perl -V" changes. Before this change updating e.g. /usr/bin/perl to a new major version would cause the next "make" command to fail, since perl.mak has hardcoded paths to perl library paths retrieved from its first run. Now the logic added in commit `ee9be06770` ("perl: detect new files in MakeMaker builds", 2012-07-27) is extended to regenerate perl/perl.mak if there's any change to "perl -V". This will in some cases redundantly trigger perl/perl.mak to be re-made, e.g. if @INC is modified in ways the build process doesn't care about through sitecustomize.pl, but the common case is that we just do the right thing and re-generate perl/perl.mak when needed. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff King	f5c2bc2b96	Makefile: detect errors in running spatch The "make coccicheck" target runs spatch against each source file. But it does so in a for loop, so "make" never sees the exit code of spatch. Worse, it redirects stderr to a log file, so the user has no indication of any failure. And then to top it all off, because we touched the patch file's mtime, make will refuse to repeat the command because it think the target is up-to-date. So for example: $ make coccicheck SPATCH=does-not-exist SPATCH contrib/coccinelle/free.cocci SPATCH contrib/coccinelle/qsort.cocci SPATCH contrib/coccinelle/xstrdup_or_null.cocci SPATCH contrib/coccinelle/swap.cocci SPATCH contrib/coccinelle/strbuf.cocci SPATCH contrib/coccinelle/object_id.cocci SPATCH contrib/coccinelle/array.cocci $ make coccicheck SPATCH=does-not-exist make: Nothing to be done for 'coccicheck'. With this patch, you get: $ make coccicheck SPATCH=does-not-exist SPATCH contrib/coccinelle/free.cocci /bin/sh: 4: does-not-exist: not found Makefile:2338: recipe for target 'contrib/coccinelle/free.cocci.patch' failed make: *** [contrib/coccinelle/free.cocci.patch] Error 1 It also dumps the log on failure, so any errors from spatch itself (like syntax errors in our .cocci files) will be seen by the user. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Jeff Hostetler	ea19489532	name-hash: add test-lazy-init-name-hash Add t/helper/test-lazy-init-name-hash.c test code to demonstrate performance times for lazy_init_name_hash() using the original single-threaded and the new multi-threaded code paths. Includes a --dump option to dump the created hashmaps to stdout. You can use this to run both code paths and confirm that they generate the same hashmaps. Includes a --analyze option to analyze performance of both code paths over a range of index sizes to help you find a lower bound for the LAZY_THREAD_COST in name-hash.c. For example, passing "-a 4000" will set "istate.cache_nr" to 4000 and then try the multi-threaded code -- probably giving 2 threads with 2000 entries each. It will then run both the single-threaded (1x4000) and the multi-threaded (2x2000) and compare the times. It will then repeat the test with 8000, 12000, and etc. so that you can see the cross over. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago
Junio C Hamano	e6b07da278	Makefile: make DC_SHA1 the default We used to use the SHA1 implementation from the OpenSSL library by default. As we are trying to be careful against collision attacks after the recent "shattered" announcement, switch the default to encourage people to use DC_SHA1 implementation instead. Those who want to use the implementation from OpenSSL can explicitly ask for it by OPENSSL_SHA1=YesPlease when running "make". Signed-off-by: Junio C Hamano <gitster@pobox.com>	8 years ago

1 2 3 4 5 ...

2211 Commits (ccc12e06764e551c86ff458b16db35958957645f)