kernel/git - git - PowerEL Git System

Author	SHA1	Message	Date
Ævar Arnfjörð Bjarmason	6fae3aaf22	spatchcache: add a ccache-alike for "spatch" Add a rather trivial "spatchcache", with this running e.g.: make cocciclean make contrib/coccinelle/free.cocci.patch \ SPATCH=contrib/coccicheck/spatchcache \ SPATCH_FLAGS=--very-quiet Is cut down from ~20s to ~5s on my system. Much of that is either fixable shell overhead, or the around 40 files we "CANTCACHE" (see the implementation). This uses "redis" as a cache by default, but it's configurable. See the embedded documentation. This is not like ccache in that we won't cache failed spatch invocations, or those where spatch suggests changes for us. Those cases are so rare that I didn't think it was worth the bother, by far the most common case is that it has no suggested changes. We'll also refuse to cache any "spatch" invocation that has output on stderr, which means that "--very-quiet" must be added to "SPATCH_FLAGS". Because we narrow the cache to that we don't need to save away stdout, stderr & the exit code. We simply cache the cases where we had no suggested changes. Another benchmark is to compare this with the previous SPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running: make clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0 Would take 33s, but with the preceding changes running without this "spatchcache" is slightly slower, or around 35s: make clean; time make contrib/coccinelle/array.cocci.patch Now doing the same with SPATCH=contrib/coccinelle/spatchcache will take around 6s, but we'll need to compile the *.o files first to take full advantage of it (which can be fast with "ccache"): make clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache 1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Taylor Blau <me@ttaylorr.com>	2 years ago
Ævar Arnfjörð Bjarmason	d0e624aed7	cocci: run against a generated ALL.cocci The preceding commits to make the "coccicheck" target incremental made it slower in some cases. As an optimization let's not have the many=many mapping of <.cocci>=<.[ch]>, but instead concat the <.cocci> into an ALL.cocci, and then run one-to-many ALL.cocci=<.[ch]>. A "make coccicheck" is now around 2x as fast as it was on "master", and around 1.5x as fast as the preceding change to make the run incremental: $ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' -r 3 Benchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' in 'origin/master Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s] Range (min … max): 4.241 s … 4.268 s 3 runs Benchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' in 'HEAD~ Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s] Range (min … max): 5.281 s … 5.436 s 3 runs Benchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' in 'HEAD Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s] Range (min … max): 2.667 s … 2.792 s 3 runs Summary 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' in 'HEAD' ran 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' in 'origin/master' 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o.c builtin/h.c))"' in 'HEAD~' This can be turned off with SPATCH_CONCAT_COCCI, but as the beneficiaries of "SPATCH_CONCAT_COCCI=" would mainly be those developing the .cocci rules themselves, let's leave this optimization on by default. For more information see my "Optimizing .cocci rules by concat'ing them" (<220901.8635dbjfko.gmgdl@evledraar.gmail.com>) on the cocci@inria.fr mailing list. This potentially changes the results of our .cocci rules, but as noted in that discussion it should be safe for our use. We don't name rules, or if we do their names don't conflict across our .cocci files. To the extent that we'd have any inter-dependencies between rules this doesn't make that worse, as we'd have them now if we ran "make coccicheck", applied the results, and would then have (due to hypothetical interdependencies) suggested changes on the subsequent "make coccicheck". Our "coccicheck-test" target makes use of the ALL.cocci when running tests, e.g. when testing unused.{c,out} we test it against ALL.cocci, not unused.cocci. We thus assert (to the extent that we have test coverage) that this concatenation doesn't change the expected results of running these rules. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Taylor Blau <me@ttaylorr.com>	2 years ago
Ævar Arnfjörð Bjarmason	316e3886e3	cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES Improve the incremental rebuilding support of "coccicheck" by piggy-backing on the computed dependency information of the corresponding .o file, rather than rebuilding all <RULE>/<FILE> pairs if either their corresponding file changes, or if any header changes. This in effect uses the same method that the "sparse" target was made to use in `c234e8a0ec` (Makefile: make the "sparse" target non-.PHONY, 2021-09-23), except that the dependency on the .o file isn't a hard one, we check with $(wildcard) if the .o file exists, and if so we'll depend on it. This means that the common case of: make make coccicheck Will benefit from incremental rebuilding, now changing e.g. a header will only re-run "spatch" on those those .c files that make use of it: By depending on the .o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See `c234e8a0ec` (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the .sp files. E.g.: make contrib/coccinelle/free.cocci.patch make -W column.h contrib/coccinelle/free.cocci.patch Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the .o files. But around 2 seconds if I did and we have those ".o" files. Notes about the approach of piggy-backing on .o for dependencies: It is a trade-off since we'll pay the extra cost of running the C compiler, but we're probably doing that anyway. The compiler is much faster than "spatch", so even though we need to re-compile the .o to create the dependency info for the .c for "spatch" it's faster (especially if using "ccache"). * There are use-cases where some would like to have .o files around, but to have the "make coccicheck" ignore them. See: https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/ For those users a: make make coccicheck SPATCH_USE_O_DEPENDENCIES= Will avoid considering the .o files. * If that .o file doesn't exist we'll depend on an intermediate file of ours which in turn depends on $(FOUND_H_SOURCES). This covers both an initial build, or where "coccicheck" is run without running "all" beforehand, and because we run "coccicheck" on e.g. files in compat/ that we don't know how to build unless the requisite flag was provided to the Makefile. Most of the runtime of "incremental" runs is now spent on various compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and therefore conflate whether we can compile an object and generate dependency information for it with whether we'd like to link it into our binary. Before this change the distinction didn't matter, but now one way to make this even faster on incremental builds would be to peel those concerns apart so that we can see that e.g. compat/mmap.c doesn't depend on column.h. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Taylor Blau <me@ttaylorr.com>	2 years ago
SZEDER Gábor	dd5d052c39	coccicheck: introduce 'pending' semantic patches Teach `make coccicheck` to avoid patches named "*.pending.cocci" and handle them separately in a new `make coccicheck-pending` instead. This means that we can separate "critical" patches from "FYI" patches. The former target can continue causing Travis to fail its static analysis job, while the latter can let us keep an eye on ongoing (pending) transitions without them causing too much fallout. Document the intended use-cases around these two targets. As the process around the pending patches is not yet fully explored, leave that out. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Based-on-work-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>	6 years ago
brian m. carlson	db1d80b8fa	contrib/coccinelle: add basic Coccinelle transforms Coccinelle (http://coccinelle.lip6.fr/) is a program which performs mechanical transformations on C programs using semantic patches. These semantic patches can be used to implement automatic refactoring and maintenance tasks. Add a set of basic semantic patches to convert common patterns related to the struct object_id transformation, as well as a README. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>	9 years ago