kernel/git - git - PowerEL Git System

Commit Graph

Author	SHA1	Message	Date
SZEDER Gábor	7d4733c501	ci: fix GCC install in the Travis CI GCC OSX job A few days ago Travis CI updated their existing OSX images, including the Homebrew database in the xcode10.1 OSX image that we use. Since then installing dependencies in the 'osx-gcc' job fails when it tries to link gcc@8: + brew link gcc@8 Error: No such keg: /usr/local/Cellar/gcc@8 GCC8 is still installed but not linked to '/usr/local' in the updated image, as it was before this update, but now we have to link it by running 'brew link gcc'. So let's do that then, and fall back to linking gcc@8 if it doesn't, just to be sure. Our builds on Azure Pipelines are unaffected by this issue. The OSX image over there doesn't contain the gcc@8 package, so we have to 'brew install' it, which already takes care of linking it to '/usr/local'. After that the 'brew link gcc' command added by this patch fails, but the \|\|-chained fallback 'brew link gcc@8' command succeeds with an "already linked" warning. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Johannes Schindelin	0eb3671ed9	ci(osx): use new location of the `perforce` cask The Azure Pipelines builds are failing for macOS due to a change in the location of the perforce cask. The command outputs the following error: + brew install caskroom/cask/perforce Error: caskroom/cask was moved. Tap homebrew/cask-cask instead. So let's try to call `brew cask install perforce` first (which is what that error message suggests, in a most round-about way). Prior to `672f51cb` we used to install the 'perforce' package with 'brew install perforce' (note: no 'cask' in there). The justification for `672f51cb` was that the command 'brew install perforce' simply stopped working, after Homebrew folks decided that it's better to move the 'perforce' package to a "cask". Their justification for this move was that 'brew install perforce' "can fail due to a checksum mismatch ...", and casks can be installed without checksum verification. And indeed, both 'brew cask install perforce' and 'brew install caskroom/cask/perforce' printed something along the lines of: ==> No checksum defined for Cask perforce, skipping verification It is unclear why `672f51cb` used 'brew install caskroom/cask/perforce' instead of 'brew cask install perforce'. It appears (by running both commands on old Travis CI macOS images) that both commands worked all the same already back then. In any case, as the error message at the top of this commit message shows, 'brew install caskroom/cask/perforce' has stopped working recently, but 'brew cask install perforce' still does, so let's use that. CI servers are typically fresh virtual machines, but not always. To accommodate for that, let's try harder if `brew cask install perforce` fails, by specifically pulling the latest `master` of the `homebrew-cask` repository. This will still fail, of course, when `homebrew-cask` falls behind Perforce's release schedule. But once it is updated, we can now simply re-run the failed jobs and they will pick up that update. As for updating `homebrew-cask`: the beginnings of automating this in https://dev.azure.com/gitgitgadget/git/_build?definitionId=11&_a=summary will be finished once the next Perforce upgrade comes around. Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
Johannes Schindelin	411e4f4735	ci: run `hdr-check` as part of the `Static Analysis` job Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
SZEDER Gábor	c46ebc2496	travis-ci: do not skip successfully tested trees in debug mode Travis CI offers shell access to its virtual machine environment running the build jobs, called "debug mode" [1]. After restarting a build job in debug mode and logging in, the first thing I usually do is to install dependencies, i.e. run './ci/install-dependencies.sh'. This works just fine when I restarted a failed build job in debug mode. However, after restarting a successful build job in debug mode our CI scripts get all clever, and exit without doing anything useful, claiming that "This commit's tree has already been built and tested successfully" [2]. Our CI scripts are right, and we do want to skip building and testing already known good trees in "regular" CI builds. In debug mode, however, this is a nuisiance, because one has to delete the cache (or at least the 'good-trees' file in the cache) to proceed. Let's update our CI scripts, in particular the common 'ci/lib.sh', to not skip previously successfully built and tested trees in debug mode, so all those scripts will do what there were supposed to do even when a successful build job was restarted in debug mode. [1] https://docs.travis-ci.com/user/running-build-in-debug-mode/ [2] `9cc2c76f5e` (travis-ci: record and skip successfully built trees, 2017-12-31) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
brian m. carlson	f6461b82b9	Documentation: fix build with Asciidoctor 2 Our documentation toolchain has traditionally been built around DocBook 4.5. This version of DocBook is the last DTD-based version of DocBook. In 2009, DocBook 5 was introduced using namespaces and its syntax is expressed in RELAX NG, which is more expressive and allows a wider variety of syntax forms. Asciidoctor, one of the alternatives for building our documentation, moved support for DocBook 4.5 out of core in its recent 2.0 release and now only supports DocBook 5 in the main release. The DocBoook 4.5 converter is still available as a separate component, but this is not available in most distro packages. This would not be a problem but for the fact that we use xmlto, which is still stuck in the DocBook 4.5 era. xmlto performs DTD validation as part of the build process. This is not problematic for DocBook 4.5, which has a valid DTD, but it clearly cannot work for DocBook 5, since no DTD can adequately express its full syntax. In addition, even if xmlto did support RELAX NG validation, that wouldn't be sufficient because it uses the libxml2-based xmllint to do so, which has known problems with validating interleaves in RELAX NG. Fortunately, there's an easy way forward: ask Asciidoctor to use its DocBook 5 backend and tell xmlto to skip validation. Asciidoctor has supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported skipping validation for probably longer than that. We also need to teach xmlto how to use the namespaced DocBook XSLT stylesheets instead of the non-namespaced ones it usually uses. Normally these stylesheets are interchangeable, but the non-namespaced ones have a bug that causes them not to strip whitespace automatically from certain elements when namespaces are in use. This results in additional whitespace at the beginning of list elements, which is jarring and unsightly. We can do this by passing a custom stylesheet with the -x option that simply imports the namespaced stylesheets via a URL. Any system with support for XML catalogs will automatically look this URL up and reference a local copy instead without us having to know where this local copy is located. We know that anyone using xmlto will already have catalogs set up properly since the DocBook 4.5 DTD used during validation is also looked up via catalogs. All major Linux distributions distribute the necessary stylesheets and have built-in catalog support, and Homebrew does as well, albeit with a requirement to set an environment variable to enable catalog support. On the off chance that someone lacks support for catalogs, it is possible for xmlto (via xmllint) to download the stylesheets from the URLs in question, although this will likely perform poorly enough to attract attention. People still have the option of using the prebuilt documentation that we ship, so happily this should not be an impediment. Finally, we need to filter out some messages from other stylesheets that occur when invoking dblatex in the CI job. This tool strips namespaces much like the unnamespaced DocBook stylesheets and prints similar messages. If we permit these messages to be printed to standard error, our documentation CI job will fail because we check standard error for unexpected output. Due to dblatex's reliance on Python 2, we may need to revisit its use in the future, in which case this problem may go away, but this can be delayed until a future patch. The final message we filter is due to libxslt on modern Debian and Ubuntu. The patch which they use to implement reproducible ID generation also prints messages about the ID generation. While this doesn't affect our current CI images since they use Ubuntu 16.04 which lacks this patch, if we upgrade to Ubuntu 18.04 or a modern Debian, these messages will appear and, like the above messages, cause a CI failure. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
SZEDER Gábor	3960290675	ci: restore running httpd tests Once upon a time GIT_TEST_HTTPD was a tristate variable and we exported 'GIT_TEST_HTTPD=YesPlease' in our CI scripts to make sure that we run the httpd tests in the Linux Clang and GCC build jobs, or error out if they can't be run for any reason [1]. Then `3b072c577b` (tests: replace test_tristate with "git env--helper", 2019-06-21) came along, turned GIT_TEST_HTTPD into a bool, but forgot to update our CI scripts accordingly. So, since GIT_TEST_HTTPD is set explicitly, but its value is not one of the standardized true values, our CI jobs have been simply skipping the httpd tests in the last couple of weeks. Set 'GIT_TEST_HTTPD=true' to restore running httpd tests in our CI jobs. [1] `a1157b76eb` (travis-ci: set GIT_TEST_HTTPD in 'ci/lib-travisci.sh', 2017-12-12) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	5 years ago
SZEDER Gábor	fb9d7431cf	travis-ci: build with GCC 4.8 as well C99 'for' loop initial declaration, i.e. 'for (int i = 0; i < n; i++)', is not allowed in Git's codebase yet, to maintain compatibility with some older compilers. Our Travis CI builds used to catch 'for' loop initial declarations, because the GETTEXT_POISON job has always built Git with the default 'cc', which in Travis CI's previous default Linux image (based on Ubuntu 14.04 Trusty) is GCC 4.8, and that GCC version errors out on this construct (not only with DEVELOPER=1, but with our default CFLAGS as well). Alas, that's not the case anymore, becase after 14.04's EOL Travis CI's current default Linux image is based on Ubuntu 16.04 Xenial [1] and its default 'cc' is now GCC 5.4, which, just like all later GCC and Clang versions, simply accepts this construct, even if we don't explicitly specify '-std=c99'. Ideally we would adjust our CFLAGS used with DEVELOPER=1 to catch this undesired construct already when contributors build Git on their own machines. Unfortunately, however, there seems to be no compiler option that would catch only this particular construct without choking on many other things, e.g. while a later compiler with '-std=c90' and/or '-ansi' does catch this construct, it can't build Git because of several screenfulls of other errors. Add the 'linux-gcc-4.8' job to Travis CI, in order to build Git with GCC 4.8, and thus to timely catch any 'for' loop initial declarations. To catch those it's sufficient to only build Git with GCC 4.8, so don't run the test suite in this job, because 'make test' takes rather long [2], and it's already run five times in other jobs, so we wouldn't get our time's worth. [1] The Azure Pipelines builds have been using Ubuntu 16.04 images from the start, so I belive they never caught 'for' loop initial declarations. [2] On Travis CI 'make test' alone would take about 9 minutes in this new job (without running httpd, Subversion, and P4 tests). For comparison, starting the job and building Git with GCC 4.8 takes only about 2 minutes. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	37a2e35395	ci/lib.sh: update a comment about installed P4 and Git-LFS versions A comment in 'ci/lib.sh' claims that the "OS X build installs the latest available versions" of P4 and Git-LFS, but since `f2f47150` ("ci: don't update Homebrew", 2019-07-03) that's no longer the case, as it will install the versions which were recorded in the image's Homebrew database when the image was created. Update this comment accordingly. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	af8ed04778	ci: disable Homebrew's auto cleanup Lately Homebrew learned to automagically clean up information about outdated packages during other 'brew' commands, which might be useful for the avarage user, but is a waste of time in CI build jobs, because the next build jobs will start from the exact same image containing the same outdated packages anyway. Export HOMEBREW_NO_INSTALL_CLEANUP=1 to disable this auto cleanup feature, shaving off about 20-30s from the time needed to install dependencies in our macOS build jobs on Travis CI. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	f2f4715033	ci: don't update Homebrew Lately our GCC macOS build job on Travis CI has been erroring out while installing dependencies with: +brew link gcc@8 Error: No such keg: /usr/local/Cellar/gcc@8 The command "ci/install-dependencies.sh" failed and exited with 1 during . Now, while gcc@8 is still pre-installed (but not linked) and would be perfectly usable in the Travis CI macOS image we use [1], it's at version 8.2. However, when installing dependencies we first explicitly run 'brew update', which spends over two minutes to update itself and information about the available packages, and it learns about GCC 8.3. After that point gcc@8 exclusively refers to v8.3, and, unfortunately, 'brew' is just too dumb to be able to do anything with the still installed 8.2 package, and the subsequent 'brew link gcc@8' fails. (Even 'brew uninstall gcc@8' fails with the same error!) Don't run 'brew update' to keep the already installed GCC 8.2 'brew link'-able. Note that in addition we have to 'export HOMEBREW_NO_AUTO_UPDATE=1' first, because 'brew' is so very helpful that it would implicitly run update for us on the next 'brew install <pkg>' otherwise. Disabling 'brew update' has additional benefits: - It shaves off 2-3mins from the ~4mins currently spent on installing dependencies, and the macOS build jobs have always been prone to exceeding the time limit on Travis CI. - Our builds won't suddenly break because of the occasional Homebrew breakages [2]. The drawback is that we'll be stuck with slightly older versions of the packages that we install via Homebrew (Git-LFS 2.5.2 and Perforce 2018.1; they are currently at 2.7.2 and 2019.1, respectively). We might want to reconsider this decision as time goes on and/or switch to a more recent macOS image as they become available. [1] `2000ac9fbf` (travis-ci: switch to Xcode 10.1 macOS image, 2019-01-17) [2] See e.g. `a1ccaedd62` (travis-ci: make the OSX build jobs' 'brew update' more quiet, 2019-02-02) or https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#+u Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Ævar Arnfjörð Bjarmason	1ff750b128	tests: make GIT_TEST_GETTEXT_POISON a boolean Change the GIT_TEST_GETTEXT_POISON variable from being "non-empty?" to being a more standard boolean variable. Since it needed to be checked in both C code and shellscript (via test -n) it was one of the remaining shellscript-like variables. Now that we have "env--helper" we can change that. There's a couple of tricky edge cases that arise because we're using git_env_bool() early, and the config-reading "env--helper". If GIT_TEST_GETTEXT_POISON is set to an invalid value die_bad_number() will die, but to do so it would usually call gettext(). Let's detect the special case of GIT_TEST_GETTEXT_POISON and always emit that message in the C locale, lest we infinitely loop. As seen in the updated tests in t0017-env-helper.sh there's also a caveat related to "env--helper" needing to read the config for trace2 purposes. Since the C_LOCALE_OUTPUT prerequisite is lazy and relies on "env--helper" we could get invalid results if we failed to read the config (e.g. because we'd loop on includes) when combined with e.g. "test_i18ngrep" wanting to check with "env--helper" if GIT_TEST_GETTEXT_POISON was true or not. I'm crossing my fingers and hoping that a test similar to the one I removed in the earlier "config tests: simplify include cycle test" change in this series won't happen again, and testing for this explicitly in "env--helper"'s own tests. This change breaks existing uses of e.g. GIT_TEST_GETTEXT_POISON=YesPlease, which we've documented in po/README and other places. As noted in [1] we might want to consider also accepting "YesPlease" in "env--helper" as a special-case. But as the lack of uproar over `6cdccfce1e` ("i18n: make GETTEXT_POISON a runtime option", 2018-11-08) demonstrates the audience for this option is a really narrow set of git developers, who shouldn't have much trouble modifying their test scripts, so I think it's better to deal with that minor headache now and make all the relevant GIT_TEST_* variables boolean in the same way than carry the "YesPlease" special-case forward. 1. https://public-inbox.org/git/xmqqtvckm3h8.fsf@gitster-ct.c.googlers.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	db864306cf	ci: install 'libsvn-perl' instead of 'git-svn' Since `e7e9f5e7a1` (travis-ci: enable Git SVN tests t91xx on Linux, 2016-05-19) some of our Travis CI build jobs install the 'git-svn' package, because it was a convenient way to install its dependencies, which are necessary to run our 'git-svn' tests (we don't actually need the 'git-svn' package itself). However, from those dependencies, namely the 'libsvn-perl', 'libyaml-perl', and 'libterm-readkey-perl' packages, only 'libsvn-perl' is necessary to run those tests, the others arent, not even to fulfill some prereqs. So update 'ci/install-dependencies.sh' to install only 'libsvn-perl' instead of 'git-svn' and its additional dependencies. Note that this change has more important implications than merely not installing three unnecessary packages, as it keeps our builds working with Travis CI's Xenial images. In our '.travis.yml' we never explicitly specified which Linux image we want to use to run our Linux build jobs, and so far they have been run on the default Ubuntu 14.04 Trusty image. However, 14.04 just reached its EOL, and Travis CI has already began the transition to use 16.04 Xenial as the default Linux build environment [1]. Alas, our Linux Clang and GCC build jobs can't simply 'apt-get install git-svn' in the current Xenial images [2], like they did in the Trusty images, and, consequently, fail. Installing only 'libsvn-perl' avoids this issue, while the 'git svn' tests are still run as they should. [1] https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment [2] 'apt-get install git-svn' in the Xenial image fails with: The following packages have unmet dependencies: git-svn : Depends: git (< 1:2.7.4-.) E: Unable to correct problems, you have held broken packages. The reason is that both the Trusty and Xenial images contain the 'git' package installed from 'ppa:git-core/ppa', so it's considerably newer than the 'git' package in the corresponding standard Ubuntu package repositories. The difference is that the Trusty image still contains these third-party apt repositories, so the 'git-svn' package was installed from the same PPA, and its version matched the version of the already installed 'git' package. In the Xenial image, however, these third-party apt-repositories are removed (to reduce the risk of unrelated interference and faster 'apt-get update') [3], and the version of the 'git-svn' package coming from the standard Ubuntu package repositories doesn't match the much more recent version of the 'git' package installed from the PPA, resulting in this dependecy error. Adding back the 'ppa:git-core/ppa' package repository would solve this dependency issue as well, but since the troublesome package happens to be unnecessary, not installing it in the first place is better. [3] https://docs.travis-ci.com/user/reference/xenial/#third-party-apt-repositories-removed Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	37fc8cb15f	ci: fix AsciiDoc/Asciidoctor stderr check in the documentation build job In 'ci/test-documentation.sh' we save the standard error of 'make doc', and, in an attempt to make sure that neither AsciiDoc nor Asciidoctor printed any warnings, we check the emptiness of the resulting file with '! test -s stderr.log'. This check has never actually worked, because in our 'ci/' build scripts we rely on 'set -e' aborting the build job when a command exits with error, and, unfortunately, the combination of the two doesn't work as intended. According to POSIX [1]: "The -e setting shall be ignored when executing [...] a pipeline beginning with the ! reserved word" [2] Watch and learn: $ echo unexpected >file $ ( set -e; ! test -s file ; echo "should not reach this" ) ; echo $? should not reach this 0 This is why we haven't noticed the warnings from Asciidoctor that were fixed in the first patches of this patch series, though some of them were already there in the build of v2.18.0-rc0 [3]. Check the emptiness of that file with 'test ! -s' instead, which works properly with 'set -e': $ ( set -e; test ! -s file ; echo "should not reach this" ) ; echo $? 1 Furthermore, dump the contents of that file to the log for our convenience, so if it were to unexpectedly end up being non-empty, then we wouldn't have to scroll through all that long build log looking for warnings, but could see them right away near the end of the log. Note that we are only really interested in the standard error of AsciiDoc and Asciidoctor, but by saving the stderr of 'make doc' we also save any error output from the make rules. Currently there is only one such line: we build the docs with Asciidoctor right after a 'make clean', meaning that 'make USE_ASCIIDOCTOR=1 doc' always starts with running 'GIT-VERSION-GEN', which in turn prints the version to stderr. A 'sed' command was supposed to remove this version line to prevent it from triggering that (previously defunct) emptiness check, but, unfortunately, this command doesn't work as intended, either, because it leaves the file to be checked intact, but that defunct emptiness check hid this issue, too... Furthermore, in the near future there will be an other line on stderr, because commit `9a71722b4d` (Doc: auto-detect changed build flags, 2019-03-17) in the currently cooking branch 'ma/doc-diff-doc-vs-doctor-comparison' will print " new asciidoc flags" at the beginning of both 'make doc' invokations. Extend that 'sed' command to remove this line, too, wrap it in a helper function so the output of both 'make doc' is filtered the same way, and change its invokation to actually write the logfile to be checked. [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#set [2] POSIX doesn't discuss the meaning of '! cmd' in case of simple commands, but it defines that "A pipeline is a sequence of one or more commands separated by the control operator '\|'", so apparently a simple command is considered as pipeline as well. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02 [3] https://travis-ci.org/git/git/jobs/385932007#L1463 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	615a6c37e1	ci: stick with Asciidoctor v1.5.8 for now The recent release of Asciidoctor v2.0.0 broke our documentation build job on Travis CI, where we 'gem install asciidoctor', which always brings us the latest and (supposedly) greatest. Alas, we are not ready for that just yet, because it removed support for DocBook 4.5, and we have been requiring that particular DocBook version to build 'user-manual.xml' with Asciidoctor, resulting in: ASCIIDOC user-manual.xml asciidoctor: FAILED: missing converter for backend 'docbook45'. Processing aborted. Use --trace for backtrace make[1]: *** [user-manual.xml] Error 1 Unfortunately, we can't simply switch to DocBook 5 right away, as doing so leads to validation errors from 'xmlto', and working around those leads to yet another errors... [1] So let's stick with Asciidoctor v1.5.8 (latest stable release before v2.0.0) in our documentation build job on Travis CI for now, until we figure out how to deal with the fallout from Asciidoctor v2.0.0. [1] https://public-inbox.org/git/20190324162131.GL4047@pobox.com/ Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	f34a1bd96c	ci: install Asciidoctor in 'ci/install-dependencies.sh' When our '.travis.yml' was split into several 'ci/*' scripts [1], the installation of the 'asciidoctor' gem somehow ended up in 'ci/test-documentation.sh'. Install it in 'ci/install-dependencies.sh', where we install other dependencies of the Documentation build job as well (asciidoc, xmlto). [1] `657343a602` (travis-ci: move Travis CI code into dedicated scripts, 2017-09-10) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	50b206371d	travis: remove the hack to build the Windows job on Azure Pipelines Since Travis did not support Windows (and now only supports very limited Windows jobs, too limited for our use, the test suite would time out all the time), we added a hack where a Travis job would trigger an Azure Pipeline (which back then was still called VSTS Build), wait for it to finish (or time out), and download the log (if available). Needless to say that it was a horrible hack, necessitated by a bad situation. Nowadays, however, we have Azure Pipelines support, and do not need that hack anymore. So let's retire it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Junio C Hamano	a8c51f77d1	ci: clear and mark MAKEFLAGS exported just once Clearing it once upfront, and turning all the assignment into appending, would future-proof the code even more, to prevent mistakes the previous one fixed from happening again. Also, mark the variable exported just once at the beginning. There is no point in marking it exported repeatedly. Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	406f93ae48	ci: make sure we build Git parallel Commit `2c8921db2b` (travis-ci: build with the right compiler, 2019-01-17) started to use MAKEFLAGS to specify which compiler to use to build Git. A bit later, and in a different topic branch commit `eaa62291ff` (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) started to use MAKEFLAGS as well. Unfortunately, there is a semantic conflict between these two commits: both of them set MAKEFLAGS, and since the line adding CC from `2c8921db2b` comes later in 'ci/lib.sh', it overwrites the number of parallel jobs added in `eaa62291ff`. Consequently, since both commits have been merged all our CI jobs have been building Git, building its documentation, and applying semantic patches sequentially, making all build jobs a bit slower. Running the test suite is unaffected, because the number of test jobs comes from GIT_PROVE_OPTS. Append to MAKEFLAGS when setting the compiler to use, to ensure that the number of parallel jobs to use is preserved. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	a1ccaedd62	travis-ci: make the OSX build jobs' 'brew update' more quiet Before installing the necessary dependencies, our OSX build jobs run 'brew update --quiet'. This is problematic for two reasons: - This '--quiet' flag apparently broke overnight, resulting in errored builds: +brew update --quiet ==> Downloading https://homebrew.bintray.com/bottles-portable-ruby/portable-ruby-2.3.7.mavericks.bottle.tar.gz ######################################################################## 100.0% ==> Pouring portable-ruby-2.3.7.mavericks.bottle.tar.gz Usage: brew update_report [--preinstall] The Ruby implementation of brew update. Never called manually. --preinstall Run in 'auto-update' mode (faster, less output). -f, --force Override warnings and enable potentially unsafe operations. -d, --debug Display any debugging information. -v, --verbose Make some output more verbose. -h, --help Show this message. Error: invalid option: --quiet The command "ci/install-dependencies.sh" failed and exited with 1 during . I belive that this breakage will be noticed and fixed soon-ish, so we could probably just wait a bit for this issue to solve itself, but: - 'brew update --quiet' wasn't really quiet in the first place, as it listed over about 2000 lines worth of available packages that we absolutely don't care about, see e.g. one of the latest 'master' builds: https://travis-ci.org/git/git/jobs/486134962#L113 So drop this '--quiet' option and redirect 'brew update's standard output to /dev/null to make it really quiet, thereby making the OSX builds work again despite the above mentioned breakage. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	b819f1d2ce	ci: parallelize testing on Windows The fact that Git's test suite is implemented in Unix shell script that is as portable as we can muster, combined with the fact that Unix shell scripting is foreign to Windows (and therefore has to be emulated), results in pretty abysmal speed of the test suite on that platform, for pretty much no other reason than that language choice. For comparison: while the Linux build & test is typically done within about 8 minutes, the Windows build & test typically lasts about 80 minutes in Azure Pipelines. To help with that, let's use the Azure Pipeline feature where you can parallelize jobs, make jobs depend on each other, and pass artifacts between them. The tests are distributed using the following heuristic: listing all test scripts ordered by size in descending order (as a cheap way to estimate the overall run time), every Nth script is run (where N is the total number of parallel jobs), starting at the index corresponding to the parallel job. This slicing is performed by a new function that is added to the `test-tool`. To optimize the overall runtime of the entire Pipeline, we need to move the Windows jobs to the beginning (otherwise there would be a very decent chance for the Pipeline to be run only the Windows build, while all the parallel Windows test jobs wait for this single one). We use Azure Pipelines Artifacts for both the minimal Git for Windows SDK as well as the built executables, as deduplication and caching close to the agents makes that really fast. For comparison: while downloading and unpacking the minimal Git for Windows SDK via PowerShell takes only one minute (down from anywhere between 2.5 to 7 when using a shallow clone), uploading it as Pipeline Artifact takes less than 30s and downloading and unpacking less than 20s (sometimes even as little as only twelve seconds). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	a87e427e35	ci: speed up Windows phase As Unix shell scripting comes at a hefty price on Windows, we have to see where we can save some time to run the test suite. Let's skip the chain linting and the bin-wrappers/ redirection on Windows; this seems to shave of anywhere between 10-30% from the overall runtime. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	27be78173d	Add a build definition for Azure DevOps This commit adds an azure-pipelines.yml file which is Azure DevOps' equivalent to Travis CI's .travis.yml. The main idea is to replicate the Travis configuration as faithfully as possible, to make it easy to compare the Azure Pipeline builds to the Travis ones (spoiler: some parts, especially the macOS jobs, are way faster in Azure Pileines). Meaning: the number and the order of the jobs added in this commit faithfully replicates what we have in .travis.yml. Note: Our .travis.yml configuration has a Windows part that is not replicated in the Azure Pipelines definition. The reason is easy to see: As Travis cannot support our Windws needs (even with the preliminary Windows support that was recently added to Travis after waiting for years for that feature, our test suite would simply hit Travis' timeout every single time). To make things a bit easier to understand, we refrain from using the `matrix` feature here because (while it is powerful) it can be a bit confusing to users who are not familiar with CI setups. Therefore, we use a separate phase even for similar configurations (such as GCC vs Clang on Linux, GCC vs Clang on macOS). Also, we make use of the shiny new feature we just introduced where the test suite can output JUnit-style .xml files. This information is made available in a nice UI that allows the viewer to filter by phase and/or test number, and to see trends such as: number of (failing) tests, time spent running the test suite, etc. (While this seemingly contradicts the intention to replicate the Travis configuration as faithfully as possible, it is just too nice to show off that capability here already.) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	6141a2edc9	ci/lib.sh: add support for Azure Pipelines This patch introduces a conditional arm that defines some environment variables and a function that displays the URL given the job id (to identify previous runs for known-good trees). Because Azure Pipeline's macOS agents already have git-lfs and gettext installed, we can leave `BREW_INSTALL_PACKAGES` empty (unlike in Travis' case). Note: this patch does not introduce an Azure Pipelines definition yet; That is left for the next patch. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	4b060a4d97	ci: use a junction on Windows instead of a symlink Symbolic links are still not quite as easy to use on Windows as on Linux (for example, on versions older than Windows 10, only administrators can create symlinks, and on Windows 10 you still need to be in developer mode for regular users to have permission), but NTFS junctions can give us a way out. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	eaa62291ff	ci: inherit --jobs via MAKEFLAGS in run-build-and-tests Let's not decide in the generic ci/ part how many jobs to run in parallel; different CI configurations would favor a different number of parallel jobs, and it is easy enough to hand that information down via the `MAKEFLAGS` variable. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	b011fabd6e	ci/lib.sh: encapsulate Travis-specific things The upcoming patches will allow building git.git via Azure Pipelines (i.e. Azure DevOps' Continuous Integration), where variable names and URLs look a bit different than in Travis CI. Also, the configurations of the available agents are different. For example, Travis' and Azure Pipelines' macOS agents are set up differently, so that on Travis, we have to install the git-lfs and gettext Homebrew packages, and on Azure Pipelines we do not need to. Likewise, Azure Pipelines' Ubuntu agents already have asciidoctor installed. Finally, on Azure Pipelines the natural way is not to base64-encode tar files of the trash directories of failed tests, but to publish build artifacts instead. Therefore, that code to log those base64-encoded tar files is guarded to be Travis-specific. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	c2160f2d19	ci: rename the library of common functions The name is hard-coded to reflect that we use Travis CI for continuous testing. In the next commits, we will extend this to be able use Azure DevOps, too. So let's adjust the name to make it more generic. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Johannes Schindelin	4096a98d79	travis: fix skipping tagged releases When building a PR, TRAVIS_BRANCH refers to the target branch. Therefore, if a PR targets `master`, and `master` happened to be tagged, we skipped the build by mistake. Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the source branch) when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also known as "push builds"). Let's give it a new variable name, too: CI_BRANCH (as it is different from TRAVIS_BRANCH). This also prepares for the upcoming patches which will make our ci/* code a bit more independent from Travis and open it to other CI systems (in particular to Azure Pipelines). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	2c8921db2b	travis-ci: build with the right compiler Our 'Makefile' hardcodes the compiler to build Git as 'CC = cc'. This CC variable can be overridden from the command line, i.e. 'make CC=gcc-X.Y' will build with that particular GCC version, but not from the environment, i.e. 'CC=gcc-X.Y make' will still build with whatever 'cc' happens to be on the platform. Our build jobs on Travis CI are badly affected by this. In the build matrix we have dedicated build jobs to build Git with GCC and Clang both on Linux and macOS from the very beginning (`522354d70f` (Add Travis CI support, 2015-11-27)). Alas, this never really worked as supposed to, because Travis CI specifies the compiler for those build jobs as 'export CC=gcc' and 'export CC=clang' (which works fine for projects built with './configure && make'). Consequently, our 'linux-clang' build job has always used GCC, because that's where 'cc' points at in Travis CI's Linux images, while the 'osx-gcc' build job has always used Clang. Furthermore, `37fa4b3c78` (travis-ci: run gcc-8 on linux-gcc jobs, 2018-05-19) added an 'export CC=gcc-8' in an attempt to build with a more modern compiler, but to no avail. Set MAKEFLAGS with CC based on the $CC environment variable, so 'make' will run the "right" compiler. The Xcode 10.1 macOS image on Travis CI already contains the gcc@8 package from Homebrew, but we have to 'brew link' it first to be able to use it. So with this patch our build jobs will build Git with the following compiler versions: linux-clang: clang version 5.0.0 (tags/RELEASE_500/final) linux-gcc: gcc-8 (Ubuntu 8.1.0-5ubuntu1~14.04) 8.1.0 osx-clang: Apple LLVM version 10.0.0 (clang-1000.11.45.5) osx-gcc: gcc-8 (Homebrew GCC 8.2.0) 8.2.0 GETTEXT_POISON: gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	bbf24adb87	travis-ci: don't be '--quiet' when running the tests All Travis CI build jobs run the test suite with 'make --quiet test'. On one hand, being quiet doesn't save us from much clutter in the output: $ make test \|wc -l 861 $ make --quiet test \|wc -l 848 It only spares 13 lines, mostly the output of entering the 't/' directory and the pre- and post-cleanup commands, which is negligible compared to the ~700 lines printed while building Git and the ~850 lines of 'prove' output. On the other hand, it's asking for trouble. In our CI build scripts we build Git and run the test suite in two separate 'make' invocations. In a prelimiary version of one of the later patches in this series, to explicitly specify which compiler to use, I changed them to basically run: make CC=$CC make --quiet test naively thinking that it should Just Work... but then that 'make --quiet test' got all clever on me, noticed the changed build flags, and then proceeded to rebuild everything with the default 'cc'. And because of that '--quiet' option, it did so, well, quietly, only saying "* new build flags", and it was by mere luck that I happened to notice that something is amiss. Let's just drop that '--quiet' option when running the test suite in all build scripts. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Ævar Arnfjörð Bjarmason	6cdccfce1e	i18n: make GETTEXT_POISON a runtime option Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON test parameter to only be a GIT_TEST_GETTEXT_POISON=<non-empty?> runtime parameter, to be consistent with other parameters documented in "Running tests with special setups" in t/README. When I added GETTEXT_POISON in `bb946bba76` ("i18n: add GETTEXT_POISON to simulate unfriendly translator", 2011-02-22) I was concerned with ensuring that the _() function would get constant folded if NO_GETTEXT was defined, and likewise that GETTEXT_POISON would be compiled out unless it was defined. But as the benchmark in my [1] shows doing a one-off runtime getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was originally added the GIT_TEST_* env variables have become the common idiom for turning on special test setups. So change GETTEXT_POISON to work the same way. Now the GETTEXT_POISON=YesPlease compile-time option is gone, and running the tests with GIT_TEST_GETTEXT_POISON=[YesPlease\|] can be toggled on/off without recompiling. This allows for conditionally amending tests to test with/without poison, similar to what `859fdc0c3c` ("commit-graph: define GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do some of that, now we e.g. always run the t0205-gettext-poison.sh test. I did enough there to remove the GETTEXT_POISON prerequisite, but its inverse C_LOCALE_OUTPUT is still around, and surely some tests using it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=. Notes on the implementation: * We still compile a dedicated GETTEXT_POISON build in Travis CI. Perhaps this should be revisited and integrated into the "linux-gcc" build, see `ae59a4e44f` ("travis: run tests with GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then again maybe not, see [2]. * We now skip a test in t0000-basic.sh under GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This test relies on C locale output, but due to an edge case in how the previous implementation of GETTEXT_POISON worked (reading it from GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does, and needs to be skipped. * The getenv() function is not reentrant, so out of paranoia about code of the form: printf(_("%s"), getenv("some-env")); call use_gettext_poison() in our early setup in git_setup_gettext() so we populate the "poison_requested" variable in a codepath that's won't suffer from that race condition. * We error out in the Makefile if you're still saying GETTEXT_POISON=YesPlease to prompt users to change their invocation. * We should not print out poisoned messages during the test initialization itself to keep it more readable, so the test library hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during setup. See [3]. See also [4] for more on the motivation behind this patch, and the history of the GETTEXT_POISON facility. 1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/ 2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/ 3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/ 4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	0f0c51181d	travis-ci: install packages in 'ci/install-dependencies.sh' Ever since we started using Travis CI, we specified the list of packages to install in '.travis.yml' via the APT addon. While running our builds on Travis CI's container-based infrastructure we didn't have another choice, because that environment didn't support 'sudo', and thus we didn't have permission to install packages ourselves. With the switch to the VM-based infrastructure in the previous patch we do get a working 'sudo', so we can install packages by running 'sudo apt-get -y install ...' as well. Let's make use of this and install necessary packages in 'ci/install-dependencies.sh', so all the dependencies (i.e. both packages and "non-packages" (P4 and Git-LFS)) are handled in the same file. Install gcc-8 only in the 'linux-gcc' build job; so far it has been unnecessarily installed in the 'linux-clang' build job as well. Print the versions of P4 and Git-LFS conditionally, i.e. only when they have been installed; with this change even the static analysis and documentation build jobs start using 'ci/install-dependencies.sh' to install packages, and neither of these two build jobs depend on and thus install those. This change will presumably be beneficial for the upcoming Azure Pipelines integration [1]: preliminary versions of that patch series run a couple of 'apt-get' commands to install the necessary packages before running 'ci/install-dependencies.sh', but with this patch it will be sufficient to run only 'ci/install-dependencies.sh'. [1] https://public-inbox.org/git/1a22efe849d6da79f2c639c62a1483361a130238.1539598316.git.gitgitgadget@gmail.com/ Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
Derrick Stolee	97164c9fe9	ci: add optional test variables The commit-graph and multi-pack-index features introduce optional data structures that are not required for normal Git operations. It is important to run the normal test suite without them enabled, but it is helpful to also run the test suite using them. Our continuous integration scripts include a second test stage that runs with optional GIT_TEST_* variables enabled. Add the following two variables to that stage: GIT_TEST_COMMIT_GRAPH GIT_TEST_MULTI_PACK_INDEX This will slow down the operation, as we build a commit-graph file after every 'git commit' operation and build a multi-pack-index during every 'git repack' operation. However, it is important that future changes are compatible with these features. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
SZEDER Gábor	aea8879a6a	travis-ci: include the trash directories of failed tests in the trace log The trash directory of a failed test might contain invaluable information about the cause of the failure, but we have no access to the trash directories of Travis CI build jobs. The only feedback we get from there is the build job's trace log, so... Modify 'ci/print-test-failures.sh' to create a tar.gz archive of the trash directory of each failed test, encode that archive with base64, and print the resulting block of ASCII text, so it gets embedded in the trace log. Furthermore, run tests with '--immediate' to faithfully preserve the failed state. Extracting the trash directories from the trace log turned out to be a bit of a hassle, partly because of the size of these logs (usually resulting in several hundreds or even thousands of lines of base64-encoded text), and partly because these logs have CRLF, CRCRLF and occasionally even CRCRCRLF line endings, which cause 'base64 -d' from coreutils to complain about "invalid input". For convenience add a small script 'ci/util/extract-trash-dirs.sh', which will extract and unpack all base64-encoded trash directories embedded in the log fed to its standard input, and include an example command to be copy-pasted into a terminal to do it all at the end of the failure report. A few of our tests create sizeable trash directories, so limit the size of each included base64-encoded block, let's say, to 1MB. And just in case something fundamental gets broken and a lot of tests fail at once, don't include trash directories when the combined size of the included base64-encoded blocks would exceed 1MB. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	0860a7641b	travis-ci: fail if Coccinelle static analysis found something to transform Coccinelle's and in turn 'make coccicheck's exit code only indicates that Coccinelle managed to finish its analysis without any errors (e.g. no unknown --options, no missing files, no syntax errors in the semantic patches, etc.), but it doesn't indicate whether it found any undesired code patterns to transform or not. To find out the latter, one has to look closer at 'make coccicheck's standard output and look for lines like: SPATCH result: contrib/coccinelle/<something>.cocci.patch And this only indicates that there is something to transform, but to see what the suggested transformations are one has to actually look into those '.cocci.patch' files. This makes the automated static analysis build job on Travis CI not particularly useful, because it neither draws our attention to Coccinelle's findings, nor shows the actual findings. Consequently, new topics introducing undesired code patterns graduated to master on several occasions without anyone noticing. The only way to draw attention in such an automated setting is to fail the build job. Therefore, modify the 'ci/run-static-analysis.sh' build script to check all the resulting '.cocci.patch' files, and fail the build job if any of them turns out to be not empty. Include those files' contents, i.e. Coccinelle's suggested transformations, in the build job's trace log, so we'll know why it failed. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	4ab8d1af33	travis-ci: run Coccinelle static analysis with two parallel jobs Currently the static analysis build job runs Coccinelle using a single 'make' job. Using two parallel jobs cuts down the build job's run time from around 10-12mins to 6-7mins, sometimes even under 6mins (there is quite large variation between build job runtimes). More than two parallel jobs don't seem to bring further runtime benefits. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Nguyễn Thái Ngọc Duy	9ac3f0e5b3	pack-objects: fix performance issues on packing large deltas Let's start with some background about oe_delta_size() and oe_set_delta_size(). If you already know, skip the next paragraph. These two are added in `0aca34e826` (pack-objects: shrink delta_size field in struct object_entry - 2018-04-14) to help reduce 'struct object_entry' size. The delta size field in this struct is reduced to only contain max 1MB. So if any new delta is produced and larger than 1MB, it's dropped because we can't really save such a large size anywhere. Fallback is provided in case existing packfiles already have large deltas, then we can retrieve it from the pack. While this should help small machines repacking large repos without large deltas (i.e. less memory pressure), dropping large deltas during the delta selection process could end up with worse pack files. And if existing packfiles already have >1MB delta and pack-objects is instructed to not reuse deltas, all of them will be dropped on the floor, and the resulting pack would be definitely bigger. There is also a regression in terms of CPU/IO if we have large on-disk deltas because fallback code needs to parse the pack every time the delta size is needed and just access to the mmap'd pack data is enough for extra page faults when memory is under pressure. Both of these issues were reported on the mailing list. Here's some numbers for comparison. Version Pack (MB) MaxRSS(kB) Time (s) ------- --------- ---------- -------- 2.17.0 5498 43513628 2494.85 2.18.0 10531 40449596 4168.94 This patch provides a better fallback that is - cheaper in terms of cpu and io because we won't have to read existing pack files as much - better in terms of pack size because the pack heuristics is back to 2.17.0 time, we do not drop large deltas at all If we encounter any delta (on-disk or created during try_delta phase) that is larger than the 1MB limit, we stop using delta_size_ field for this because it can't contain such size anyway. A new array of delta size is dynamically allocated and can hold all the deltas that 2.17.0 can. This array only contains delta sizes that delta_size_ can't contain. With this, we do not have to drop deltas in try_delta() anymore. Of course the downside is we use slightly more memory, even compared to 2.17.0. But since this is considered an uncommon case, a bit more memory consumption should not be a problem. Delta size limit is also raised from 1MB to 16MB to better cover common case and avoid that extra memory consumption (99.999% deltas in this reported repo are under 12MB; Jeff noted binary artifacts topped out at about 3MB in some other private repos). Other fields are shuffled around to keep this struct packed tight. We don't use more memory in common case even with this limit update. A note about thread synchronization. Since this code can be run in parallel during delta searching phase, we need a mutex. The realloc part in packlist_alloc() is not protected because it only happens during the object counting phase, which is always single-threaded. Access to e->delta_size_ (and by extension pack->delta_size[e - pack->objects]) is unprotected as before, the thread scheduler in pack-objects must make sure "e" is never updated by two different threads. The area under the new lock is as small as possible, avoiding locking at all in common case, since lock contention with high thread count could be expensive (most blobs are small enough that delta compute time is short and we end up taking the lock very often). The previous attempt to always hold a lock in oe_delta_size() and oe_set_delta_size() increases execution time by 33% when repacking linux.git with with 40 threads. Reported-by: Elijah Newren <newren@gmail.com> Helped-by: Elijah Newren <newren@gmail.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Nguyễn Thái Ngọc Duy	37fa4b3c78	travis-ci: run gcc-8 on linux-gcc jobs Switch from gcc-4.8 to gcc-8. Newer compilers come with more warning checks (usually in -Wextra). Since -Wextra is enabled in developer mode (which is also enabled in travis), this lets travis report more warnings before other people do it. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Nguyễn Thái Ngọc Duy	f6a5576d52	ci: exercise the whole test suite with uncommon code in pack-objects Some recent optimizations have been added to pack-objects to reduce memory usage and some code paths are split into two: one for common use cases and one for rare ones. Make sure the rare cases are tested with Travis since it requires manual test configuration that is unlikely to be done by developers. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Nguyễn Thái Ngọc Duy	4c2db93807	read-cache.c: make $GIT_TEST_SPLIT_INDEX boolean While at there, document about this special mode when running the test suite. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	aedffe9525	travis-ci: run tests with '-x' tracing Now that the test suite runs successfully with '-x' tracing even with /bin/sh, enable it on Travis CI in order to - get more information about test failures, and - catch constructs breaking '-x' with /bin/sh sneaking into our test suite. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	6b995760dc	travis-ci: don't fail if user already exists on 32 bit Linux build job The 32 bit Linux build job runs in a Docker container, which lends itself to running and debugging locally, too. Especially during debugging one usually doesn't want to start with a fresh container every time, to save time spent on installing a bunch of dependencies. However, that doesn't work quite smootly, because the script running in the container always creates a new user, which then must be removed every time before subsequent executions, or the build script fails. Make this process more convenient and don't try to create that user if it already exists and has the right user ID in the container, so developers don't have to bother with running a 'userdel' each time before they run the build script. The build job on Travis CI always starts with a fresh Docker container, so this change doesn't make a difference there. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	533033024a	travis-ci: don't run the test suite as root in the 32 bit Linux build Travis CI runs the 32 bit Linux build job in a Docker container, where all commands are executed as root by default. Therefore, ever since we added this build job in `88dedd5e7` (Travis: also test on 32-bit Linux, 2017-03-05), we have a bit of code to create a user in the container matching the ID of the host user and then to run the test suite as this user. Matching the host user ID is important, because otherwise the host user would have no access to any files written by processes running in the container, notably the logs of failed tests couldn't be included in the build job's trace log. Alas, this piece of code never worked, because it sets the variable holding the user name ($CI_USER) in a subshell, meaning it doesn't have any effect by the time we get to the point to actually use the variable to switch users with 'su'. So all this time we were running the test suite as root. Reorganize that piece of code in 'ci/run-linux32-build.sh' a bit to avoid that problematic subshell and to ensure that we switch to the right user. Furthermore, make the script's optional host user ID option mandatory, so running the build accidentally as root will become harder when debugging locally. If someone really wants to run the test suite as root, whatever the reasons might be, it'll still be possible to do so by explicitly passing '0' as host user ID. Finally, one last catch: since commit `7e72cfcee` (travis-ci: save prove state for the 32 bit Linux build, 2017-12-27) the 'prove' test harness has been writing its state to the Travis CI cache directory from within the Docker container while running as root. After this patch 'prove' will run as a regular user, so in future build jobs it won't be able overwrite a previously written, still root-owned state file, resulting in build job failures. To resolve this we should manually delete caches containing such root-owned files, but that would be a hassle. Instead, work this around by changing the owner of the whole contents of the cache directory to the host user ID. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	b2cbaa091c	travis-ci: don't repeat the path of the cache directory Some of our 'ci/*' scripts repeat the name or full path of the Travis CI cache directory, and the following patches will add new places using that path. Use a variable to refer to the path of the cache directory instead, so it's hard-coded only in a single place. Pay extra attention to the 32 bit Linux build: it runs in a Docker container, so pass the path of the cache directory from the host to the container in an environment variable. Note that an environment variable passed this way is exported inside the container, therefore its value is directly available in the 'su' snippet even though that snippet is single quoted. Furthermore, use the variable in the container only if it's been assigned a non-empty value, to prevent errors when someone is running or debugging the Docker build locally, because in that case the variable won't be set as there won't be any Travis CI cache. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	04d47e969a	travis-ci: use 'set -e' in the 32 bit Linux build job The script 'ci/run-linux32-build.sh' running inside the Docker container of the 32 bit Linux build job uses an && chain to break the build if one of the commands fails. This is problematic for two reasons: - The && chain is broken, because there is this in the middle: test -z $HOST_UID \|\| (CI_USER="ci" && useradd -u $HOST_UID $CI_USER) && Luckily it is broken in a way that it didn't lead to false successes. If installing dependencies fails, then the rest of the first && chain is skipped and execution resumes after the \|\| operator. At that point $HOST_UID is still unset, causing 'useradd' to error out with "invalid user ID 'ci'", which in turn causes the second && chain to abort the script and thus break the build. - All other 'ci/*' scripts use 'set -e' to break the build if one of the commands fails. This inconsistency among these scripts is asking for trouble: I forgot about the && chain more than once while working on this patch series. Enable 'set -e' for the whole script and for the commands executed under 'su' as well. While touching every line in the 'su' command block anyway, change their indentation to use a tab instead of spaces. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	f63b12392a	travis-ci: use 'set -x' for the commands under 'su' in the 32 bit Linux build Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
Thomas Gummerer	ae59a4e44f	travis: run tests with GIT_TEST_SPLIT_INDEX Split index mode only has a few dedicated tests, but as the index is involved in nearly every git operation, this doesn't quite cover all the ways repositories with split index can break. To use split index mode throughout the test suite a GIT_TEST_SPLIT_INDEX environment variable can be set, which makes git split the index at random and thus excercises the functionality much more thoroughly. As this is not turned on by default, it is not executed nearly as often as the test suite is run, so occationally breakages slip through. Try to counteract that by running the test suite with GIT_TEST_SPLIT_INDEX mode turned on on travis. To avoid using too many cycles on travis only run split index mode in the linux-gcc target only. The Linux build was chosen over the Mac OS builds because it tends to be much faster to complete. The linux gcc build was chosen over the linux clang build because the linux clang build is the fastest build, so it can serve as an early indicator if something is broken and we want to avoid spending the extra cycles of running the test suite twice for that. Helped-by: Lars Schneider <larsxschneider@gmail.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	3c93b82920	travis-ci: build Git during the 'script' phase Ever since we started building and testing Git on Travis CI (`522354d70` (Add Travis CI support, 2015-11-27)), we build Git in the 'before_script' phase and run the test suite in the 'script' phase (except in the later introduced 32 bit Linux and Windows build jobs, where we build in the 'script' phase'). Contrarily, the Travis CI practice is to build and test in the 'script' phase; indeed Travis CI's default build command for the 'script' phase of C/C++ projects is: ./configure && make && make test The reason why Travis CI does it this way and why it's a better approach than ours lies in how unsuccessful build jobs are categorized. After something went wrong in a build job, its state can be: - 'failed', if a command in the 'script' phase returned an error. This is indicated by a red 'X' on the Travis CI web interface. - 'errored', if a command in the 'before_install', 'install', or 'before_script' phase returned an error, or the build job exceeded the time limit. This is shown as a red '!' on the web interface. This makes it easier, both for humans looking at the Travis CI web interface and for automated tools querying the Travis CI API, to decide when an unsuccessful build is our responsibility requiring human attention, i.e. when a build job 'failed' because of a compiler error or a test failure, and when it's caused by something beyond our control and might be fixed by restarting the build job, e.g. when a build job 'errored' because a dependency couldn't be installed due to a temporary network error or because the OSX build job exceeded its time limit. The drawback of building Git in the 'before_script' phase is that one has to check the trace log of all 'errored' build jobs, too, to see what caused the error, as it might have been caused by a compiler error. This requires additional clicks and page loads on the web interface and additional complexity and API requests in automated tools. Therefore, move building Git from the 'before_script' phase to the 'script' phase, updating the script's name accordingly as well. 'ci/run-builds.sh' now becomes basically empty, remove it. Several of our build job configurations override our default 'before_script' to do nothing; with this change our default 'before_script' won't do anything, either, so remove those overriding directives as well. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	b92cb86ea1	travis-ci: check that all build artifacts are .gitignore-d Every once in a while our explicit .gitignore files get out of sync when our build process learns to create new artifacts, like test helper executables, but the .gitignore files are not updated accordingly. Use Travis CI to help catch such issues earlier: check that there are no untracked files at the end of any build jobs building Git (i.e. the 64 bit Clang and GCC Linux and OSX build jobs, plus the GETTEXT_POISON and 32 bit Linux build jobs) or its documentation, and fail the build job if there are any present. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago
SZEDER Gábor	88e00b7033	travis-ci: don't store P4 and Git LFS in the working tree The Clang and GCC 64 bit Linux build jobs download and store the P4 and Git LFS executables under the current directory, which is the working tree that we are about to build and test. This means that Git commands like 'status' or 'ls-files' would list these files as untracked. The next commit is about to make sure that there are no untracked files present after the build, and the downloaded executables in the working tree are interfering with those upcoming checks. Therefore, let's download P4 and Git LFS in the home directory, outside of the working tree. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	7 years ago

1 2

96 Commits (763b47bafab6f781dc52461985b3300d447d1332)