kernel
/
git
mirror of https://git.kernel.org/pub/scm/git/git.git
1
0
Fork 0
Code Releases Activity
Browse Source

Merge branch 'ib/test-selectively-run' 

Allow specifying only certain individual test pieces to be run
using a range notation (e.g. "t1234-test.sh --run='1-4 6 8 9-'").

* ib/test-selectively-run:
  t0000-*.sh: fix the GIT_SKIP_TESTS sub-tests
  test-lib: '--run' to run only specific tests
  test-lib: tests skipped by GIT_SKIP_TESTS say so
  test-lib: document short options in t/README
maint
Junio C Hamano 11 years ago
parent
c6d3abbf99 7e28c16fdb
commit
7e1a5381b0
3 changed files with 617 additions and 12 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 85
      t/README
  2. 425
      t/t0000-basic.sh
  3. 119
      t/test-lib.sh

85
t/README
Unescape View File

@ -71,6 +71,7 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate @@ -71,6 +71,7 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate
(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
appropriately before running "make".

-v::
--verbose::
This makes the test more verbose. Specifically, the
command being run and their output if any are also
@ -81,6 +82,7 @@ appropriately before running "make". @@ -81,6 +82,7 @@ appropriately before running "make".
numbers matching <pattern>. The number matched against is
simply the running count of the test within the file.

-d::
--debug::
This may help the person who is developing a new test.
It causes the command defined with test_debug to run.
@ -89,6 +91,7 @@ appropriately before running "make". @@ -89,6 +91,7 @@ appropriately before running "make".
failed tests so that you can inspect its contents after
the test finished.

-i::
--immediate::
This causes the test to immediately exit upon the first
failed test. Cleanup commands requested with
@ -96,10 +99,17 @@ appropriately before running "make". @@ -96,10 +99,17 @@ appropriately before running "make".
in order to keep the state for inspection by the tester
to diagnose the bug.

-l::
--long-tests::
This causes additional long-running tests to be run (where
available), for more exhaustive testing.

-r::
--run=<test-selector>::
Run only the subset of tests indicated by
<test-selector>. See section "Skipping Tests" below for
<test-selector> syntax.

--valgrind=<tool>::
Execute all Git binaries under valgrind tool <tool> and exit
with status 126 on errors (just like regular tests, this will
@ -187,10 +197,77 @@ and either can match the "t[0-9]{4}" part to skip the whole @@ -187,10 +197,77 @@ and either can match the "t[0-9]{4}" part to skip the whole
test, or t[0-9]{4} followed by ".$number" to say which
particular test to skip.

Note that some tests in the existing test suite rely on previous
test item, so you cannot arbitrarily disable one and expect the
remainder of test to check what the test originally was intended
to check.
For an individual test suite --run could be used to specify that
only some tests should be run or that some tests should be
excluded from a run.

The argument for --run is a list of individual test numbers or
ranges with an optional negation prefix that define what tests in
a test suite to include in the run. A range is two numbers
separated with a dash and matches a range of tests with both ends
been included. You may omit the first or the second number to
mean "from the first test" or "up to the very last test"
respectively.

Optional prefix of '!' means that the test or a range of tests
should be excluded from the run.

If --run starts with an unprefixed number or range the initial
set of tests to run is empty. If the first item starts with '!'
all the tests are added to the initial set. After initial set is
determined every test number or range is added or excluded from
the set one by one, from left to right.

Individual numbers or ranges could be separated either by a space
or a comma.

For example, to run only tests up to a specific test (21), one
could do this:

$ sh ./t9200-git-cvsexport-commit.sh --run='1-21'

or this:

$ sh ./t9200-git-cvsexport-commit.sh --run='-21'

Common case is to run several setup tests (1, 2, 3) and then a
specific test (21) that relies on that setup:

$ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'

or:

$ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21

or:

$ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'

As noted above, the test set is built going though items left to
right, so this:

$ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'

will run tests 1, 2, and 4. Items that comes later have higher
precendence. It means that this:

$ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'

would just run tests from 1 to 4, including 3.

You may use negation with ranges. The following will run all
test in the test suite except from 7 up to 11:

$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'

Some tests in a test suite rely on the previous tests performing
certain actions, specifically some tests are designated as
"setup" test, so you cannot _arbitrarily_ disable one test and
expect the rest to function correctly.

--run is mostly useful when you want to focus on a specific test
and know what setup is needed for it. Or when you want to run
everything up to a certain test.


Naming Tests

425
t/t0000-basic.sh
Unescape View File

@ -42,9 +42,9 @@ test_expect_success 'success is reported like this' ' @@ -42,9 +42,9 @@ test_expect_success 'success is reported like this' '
:
'

run_sub_test_lib_test () {
name="$1" descr="$2" # stdin is the body of the test code
shift 2
_run_sub_test_lib_test_common () {
neg="$1" name="$2" descr="$3" # stdin is the body of the test code
shift 3
mkdir "$name" &&
(
# Pretend we're not running under a test harness, whether we
@ -70,10 +70,23 @@ run_sub_test_lib_test () { @@ -70,10 +70,23 @@ run_sub_test_lib_test () {
export TEST_DIRECTORY &&
TEST_OUTPUT_DIRECTORY=$(pwd) &&
export TEST_OUTPUT_DIRECTORY &&
./"$name.sh" "$@" >out 2>err
if test -z "$neg"
then
./"$name.sh" "$@" >out 2>err
else
! ./"$name.sh" "$@" >out 2>err
fi
)
}

run_sub_test_lib_test () {
_run_sub_test_lib_test_common '' "$@"
}

run_sub_test_lib_test_err () {
_run_sub_test_lib_test_common '!' "$@"
}

check_sub_test_lib_test () {
name="$1" # stdin is the expected output from the test
(
@ -84,6 +97,18 @@ check_sub_test_lib_test () { @@ -84,6 +97,18 @@ check_sub_test_lib_test () {
)
}

check_sub_test_lib_test_err () {
name="$1" # stdin is the expected output output from the test
# expected error output is in descriptior 3
(
cd "$name" &&
sed -e 's/^> //' -e 's/Z$//' >expect.out &&
test_cmp expect.out out &&
sed -e 's/^> //' -e 's/Z$//' <&3 >expect.err &&
test_cmp expect.err err
)
}

test_expect_success 'pretend we have a fully passing test suite' "
run_sub_test_lib_test full-pass '3 passing tests' <<-\\EOF &&
for i in 1 2 3
@ -270,6 +295,398 @@ test_expect_success 'test --verbose-only' ' @@ -270,6 +295,398 @@ test_expect_success 'test --verbose-only' '
EOF
'

test_expect_success 'GIT_SKIP_TESTS' "
(
GIT_SKIP_TESTS='git.2' && export GIT_SKIP_TESTS &&
run_sub_test_lib_test git-skip-tests-basic \
'GIT_SKIP_TESTS' <<-\\EOF &&
for i in 1 2 3
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test git-skip-tests-basic <<-\\EOF
> ok 1 - passing test #1
> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)
> ok 3 - passing test #3
> # passed all 3 test(s)
> 1..3
EOF
)
"

test_expect_success 'GIT_SKIP_TESTS several tests' "
(
GIT_SKIP_TESTS='git.2 git.5' && export GIT_SKIP_TESTS &&
run_sub_test_lib_test git-skip-tests-several \
'GIT_SKIP_TESTS several tests' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test git-skip-tests-several <<-\\EOF
> ok 1 - passing test #1
> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)
> ok 3 - passing test #3
> ok 4 - passing test #4
> ok 5 # skip passing test #5 (GIT_SKIP_TESTS)
> ok 6 - passing test #6
> # passed all 6 test(s)
> 1..6
EOF
)
"

test_expect_success 'GIT_SKIP_TESTS sh pattern' "
(
GIT_SKIP_TESTS='git.[2-5]' && export GIT_SKIP_TESTS &&
run_sub_test_lib_test git-skip-tests-sh-pattern \
'GIT_SKIP_TESTS sh pattern' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test git-skip-tests-sh-pattern <<-\\EOF
> ok 1 - passing test #1
> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)
> ok 3 # skip passing test #3 (GIT_SKIP_TESTS)
> ok 4 # skip passing test #4 (GIT_SKIP_TESTS)
> ok 5 # skip passing test #5 (GIT_SKIP_TESTS)
> ok 6 - passing test #6
> # passed all 6 test(s)
> 1..6
EOF
)
"

test_expect_success '--run basic' "
run_sub_test_lib_test run-basic \
'--run basic' --run='1 3 5' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-basic <<-\\EOF
> ok 1 - passing test #1
> ok 2 # skip passing test #2 (--run)
> ok 3 - passing test #3
> ok 4 # skip passing test #4 (--run)
> ok 5 - passing test #5
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run with a range' "
run_sub_test_lib_test run-range \
'--run with a range' --run='1-3' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-range <<-\\EOF
> ok 1 - passing test #1
> ok 2 - passing test #2
> ok 3 - passing test #3
> ok 4 # skip passing test #4 (--run)
> ok 5 # skip passing test #5 (--run)
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run with two ranges' "
run_sub_test_lib_test run-two-ranges \
'--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-two-ranges <<-\\EOF
> ok 1 - passing test #1
> ok 2 - passing test #2
> ok 3 # skip passing test #3 (--run)
> ok 4 # skip passing test #4 (--run)
> ok 5 - passing test #5
> ok 6 - passing test #6
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run with a left open range' "
run_sub_test_lib_test run-left-open-range \
'--run with a left open range' --run='-3' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-left-open-range <<-\\EOF
> ok 1 - passing test #1
> ok 2 - passing test #2
> ok 3 - passing test #3
> ok 4 # skip passing test #4 (--run)
> ok 5 # skip passing test #5 (--run)
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run with a right open range' "
run_sub_test_lib_test run-right-open-range \
'--run with a right open range' --run='4-' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-right-open-range <<-\\EOF
> ok 1 # skip passing test #1 (--run)
> ok 2 # skip passing test #2 (--run)
> ok 3 # skip passing test #3 (--run)
> ok 4 - passing test #4
> ok 5 - passing test #5
> ok 6 - passing test #6
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run with basic negation' "
run_sub_test_lib_test run-basic-neg \
'--run with basic negation' --run='"'!3'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-basic-neg <<-\\EOF
> ok 1 - passing test #1
> ok 2 - passing test #2
> ok 3 # skip passing test #3 (--run)
> ok 4 - passing test #4
> ok 5 - passing test #5
> ok 6 - passing test #6
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run with two negations' "
run_sub_test_lib_test run-two-neg \
'--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-two-neg <<-\\EOF
> ok 1 - passing test #1
> ok 2 - passing test #2
> ok 3 # skip passing test #3 (--run)
> ok 4 - passing test #4
> ok 5 - passing test #5
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run a range and negation' "
run_sub_test_lib_test run-range-and-neg \
'--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-range-and-neg <<-\\EOF
> ok 1 - passing test #1
> ok 2 # skip passing test #2 (--run)
> ok 3 - passing test #3
> ok 4 - passing test #4
> ok 5 # skip passing test #5 (--run)
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run range negation' "
run_sub_test_lib_test run-range-neg \
'--run range negation' --run='"'!1-3'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-range-neg <<-\\EOF
> ok 1 # skip passing test #1 (--run)
> ok 2 # skip passing test #2 (--run)
> ok 3 # skip passing test #3 (--run)
> ok 4 - passing test #4
> ok 5 - passing test #5
> ok 6 - passing test #6
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run include, exclude and include' "
run_sub_test_lib_test run-inc-neg-inc \
'--run include, exclude and include' \
--run='"'1-5 !1-3 2'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-inc-neg-inc <<-\\EOF
> ok 1 # skip passing test #1 (--run)
> ok 2 - passing test #2
> ok 3 # skip passing test #3 (--run)
> ok 4 - passing test #4
> ok 5 - passing test #5
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run include, exclude and include, comma separated' "
run_sub_test_lib_test run-inc-neg-inc-comma \
'--run include, exclude and include, comma separated' \
--run=1-5,\!1-3,2 <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-inc-neg-inc-comma <<-\\EOF
> ok 1 # skip passing test #1 (--run)
> ok 2 - passing test #2
> ok 3 # skip passing test #3 (--run)
> ok 4 - passing test #4
> ok 5 - passing test #5
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run exclude and include' "
run_sub_test_lib_test run-neg-inc \
'--run exclude and include' \
--run='"'!3- 5'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-neg-inc <<-\\EOF
> ok 1 - passing test #1
> ok 2 - passing test #2
> ok 3 # skip passing test #3 (--run)
> ok 4 # skip passing test #4 (--run)
> ok 5 - passing test #5
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run empty selectors' "
run_sub_test_lib_test run-empty-sel \
'--run empty selectors' \
--run='1,,3,,,5' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-empty-sel <<-\\EOF
> ok 1 - passing test #1
> ok 2 # skip passing test #2 (--run)
> ok 3 - passing test #3
> ok 4 # skip passing test #4 (--run)
> ok 5 - passing test #5
> ok 6 # skip passing test #6 (--run)
> # passed all 6 test(s)
> 1..6
EOF
"

test_expect_success '--run invalid range start' "
run_sub_test_lib_test_err run-inv-range-start \
'--run invalid range start' \
--run='a-5' <<-\\EOF &&
test_expect_success \"passing test #1\" 'true'
test_done
EOF
check_sub_test_lib_test_err run-inv-range-start \
<<-\\EOF_OUT 3<<-\\EOF_ERR
> FATAL: Unexpected exit with code 1
EOF_OUT
> error: --run: invalid non-numeric in range start: 'a-5'
EOF_ERR
"

test_expect_success '--run invalid range end' "
run_sub_test_lib_test_err run-inv-range-end \
'--run invalid range end' \
--run='1-z' <<-\\EOF &&
test_expect_success \"passing test #1\" 'true'
test_done
EOF
check_sub_test_lib_test_err run-inv-range-end \
<<-\\EOF_OUT 3<<-\\EOF_ERR
> FATAL: Unexpected exit with code 1
EOF_OUT
> error: --run: invalid non-numeric in range end: '1-z'
EOF_ERR
"

test_expect_success '--run invalid selector' "
run_sub_test_lib_test_err run-inv-selector \
'--run invalid selector' \
--run='1?' <<-\\EOF &&
test_expect_success \"passing test #1\" 'true'
test_done
EOF
check_sub_test_lib_test_err run-inv-selector \
<<-\\EOF_OUT 3<<-\\EOF_ERR
> FATAL: Unexpected exit with code 1
EOF_OUT
> error: --run: invalid non-numeric in test selector: '1?'
EOF_ERR
"


test_set_prereq HAVEIT
haveit=no
test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '

119
t/test-lib.sh
Unescape View File

@ -192,6 +192,14 @@ do @@ -192,6 +192,14 @@ do
immediate=t; shift ;;
-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)
GIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;
-r)
shift; test "$#" -ne 0 || {
echo 'error: -r requires an argument' >&2;
exit 1;
}
run_list=$1; shift ;;
--run=*)
run_list=$(expr "z$1" : 'z[^=]*=\(.*\)'); shift ;;
-h|--h|--he|--hel|--help)
help=t; shift ;;
-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)
@ -367,6 +375,99 @@ match_pattern_list () { @@ -367,6 +375,99 @@ match_pattern_list () {
return 1
}

match_test_selector_list () {
title="$1"
shift
arg="$1"
shift
test -z "$1" && return 0

# Both commas and whitespace are accepted as separators.
OLDIFS=$IFS
IFS=' ,'
set -- $1
IFS=$OLDIFS

# If the first selector is negative we include by default.
include=
case "$1" in
!*) include=t ;;
esac

for selector
do
orig_selector=$selector

positive=t
case "$selector" in
!*)
positive=
selector=${selector##?}
;;
esac

test -z "$selector" && continue

case "$selector" in
*-*)
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $title: invalid non-numeric in range" \
"start: '$orig_selector'" >&2
exit 1
fi
if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $title: invalid non-numeric in range" \
"end: '$orig_selector'" >&2
exit 1
fi
;;
*)
if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $title: invalid non-numeric in test" \
"selector: '$orig_selector'" >&2
exit 1
fi
esac

# Short cut for "obvious" cases
test -z "$include" && test -z "$positive" && continue
test -n "$include" && test -n "$positive" && continue

case "$selector" in
-*)
if test $arg -le ${selector#-}
then
include=$positive
fi
;;
*-)
if test $arg -ge ${selector%-}
then
include=$positive
fi
;;
*-*)
if test ${selector%%-*} -le $arg \
&& test $arg -le ${selector#*-}
then
include=$positive
fi
;;
*)
if test $arg -eq $selector
then
include=$positive
fi
;;
esac
done

test -n "$include"
}

maybe_teardown_verbose () {
test -z "$verbose_only" && return
exec 4>/dev/null 3>/dev/null
@ -453,25 +554,35 @@ test_finish_ () { @@ -453,25 +554,35 @@ test_finish_ () {

test_skip () {
to_skip=
skipped_reason=
if match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS
then
to_skip=t
skipped_reason="GIT_SKIP_TESTS"
fi
if test -z "$to_skip" && test -n "$test_prereq" &&
! test_have_prereq "$test_prereq"
then
to_skip=t
fi
case "$to_skip" in
t)

of_prereq=
if test "$missing_prereq" != "$test_prereq"
then
of_prereq=" of $test_prereq"
fi
skipped_reason="missing $missing_prereq${of_prereq}"
fi
if test -z "$to_skip" && test -n "$run_list" &&
! match_test_selector_list '--run' $test_count "$run_list"
then
to_skip=t
skipped_reason="--run"
fi

case "$to_skip" in
t)
say_color skip >&3 "skipping test: $@"
say_color skip "ok $test_count # skip $1 (missing $missing_prereq${of_prereq})"
say_color skip "ok $test_count # skip $1 ($skipped_reason)"
: true
;;
*)

Write Preview
Loading…
Cancel
Save