From 681ef777349ced847d7cc5e511a3af413844d069 Mon Sep 17 00:00:00 2001 From: Derick Rethans Date: Wed, 23 Apr 2025 17:40:56 +0100 Subject: [PATCH] Convert https://qa.php.net/running-tests.php --- docs/source/index.rst | 1 + docs/source/miscellaneous/running-tests.rst | 160 ++++++++++++++++++++ docs/source/miscellaneous/writing-tests.rst | 8 +- 3 files changed, 165 insertions(+), 4 deletions(-) create mode 100644 docs/source/miscellaneous/running-tests.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 5d81398edef..21e2526f47f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -21,6 +21,7 @@ miscellaneous/stubs miscellaneous/writing-tests + miscellaneous/running-tests Welcome to the php-src documentation! diff --git a/docs/source/miscellaneous/running-tests.rst b/docs/source/miscellaneous/running-tests.rst new file mode 100644 index 00000000000..c85e9f4c24d --- /dev/null +++ b/docs/source/miscellaneous/running-tests.rst @@ -0,0 +1,160 @@ +############### + Running Tests +############### + +The easiest way to test your PHP build is to run make test from the command line after successfully +compiling. This will run the all tests for all enabled functionalities and extensions located in +tests folders under the source root directory using the PHP CLI binary. + +``make test`` executes the ``run-tests.php`` script under the source root (parallel builds will not +work). Therefore you can execute the script as follows: + +.. code:: shell + + TEST_PHP_EXECUTABLE=sapi/cli/php \ + sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB] + +****************************************** + Which php executable does make test use? +****************************************** + +If you are running the ``run-tests.php`` script from the command line (as above) you must set the +``TEST_PHP_EXECUTABLE`` environment variable to explicitly select the PHP executable that is to be +tested, that is, used to run the test scripts. + +If you run the tests using make test, the PHP CLI and CGI executables are automatically set for you. +``make test`` executes ``run-tests.php`` script with the CLI binary. Some test scripts such as +session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all +tests. + +**Note:** The PHP binary executing ``run-tests.php`` and the PHP binary used for executing test +scripts may differ. If you use different PHP binary for executing ``run-tests.php`` script, you may +get errors. + +************************ + Which php.ini is used? +************************ + +``make test`` uses the same ``php.ini`` file as it would once installed. The tests have been written +to be independent of that ``php.ini`` file, so if you find a test that is affected by a setting, +please report this, so we can address the issue. + +********************************** + Which test scripts are executed? +********************************** + +The ``run-tests.php`` (``make test``), without any arguments executes all test scripts by extracting +all directories named tests from the source root and any subdirectories below. If there are files, +which have a phpt extension, ``run-tests.php`` looks at the sections in these files, determines +whether it should run it, by evaluating the ``SKIPIF`` section. If the test is eligible for +execution, the ``FILE`` section is extracted into a ``.php`` file (with the same name besides the +extension) and gets executed. When an argument is given or ``TESTS`` environment variable is set, +the GLOB is expanded by the shell and any file with extension ``*.phpt`` is regarded as a test file. + +Tester can easily execute tests selectively with as follows: + +.. code:: shell + + ./sapi/cli/php run-tests.php ext/mbstring/* + ./sapi/cli/php run-tests.php ext/mbstring/020.phpt + +************** + Test results +************** + +Test results are printed to standard output. If there is a failed test, the ``run-tests.php`` script +saves the result, the expected result and the code executed to the test script directory. For +example, if ``ext/myext/tests/myext.phpt`` fails to pass, the following files are created: + +- ``ext/myext/tests/myext.php`` - actual test file executed +- ``ext/myext/tests/myext.log`` - log of test execution (L) +- ``ext/myext/tests/myext.exp`` - expected output (E) +- ``ext/myext/tests/myext.out`` - output from test script (O) +- ``ext/myext/tests/myext.diff`` - diff of .out and .exp (D) + +Failed tests are always bugs. Either the test is bugged or not considering factors applying to the +tester's environment, or there is a bug in PHP. If this is a known bug, we strive to provide bug +numbers, in either the test name or the file name. You can check the status of such a bug, by going +to: ``https://bugs.php.net/12345`` where 12345 is the bug number. For clarity and automated +processing, bug numbers are prefixed by a hash sign '#' in test names and/or test cases are named +``bug12345.phpt``. + +**Note:** The files generated by tests can be selected by setting the environment variable +``TEST_PHP_LOG_FORMAT``. For each file you want to be generated use the character in brackets as +shown above (default is LEOD). The php file will be generated always. + +**Note**: You can set environment variable ``TEST_PHP_DETAILED`` to enable detailed test +information. + +******************* + Automated testing +******************* + +If you like to keep up to speed, with latest developments and quality assurance, setting the +environment variable ``NO_INTERACTION`` to 1, will not prompt the tester for any user input. + +Normally, the exit status of make test is zero, regardless of the results of independent tests. Set +the environment variable ``REPORT_EXIT_STATUS`` to ``1``, and make test will set the exit status +("$?") to non-zero, when an individual test has failed. + +Example script to be run by cron: + +.. code:: shell + + ========== qa-test.sh ============= + #!/bin/sh + + CO_DIR=$HOME/cvs/php7 + MYMAIL=qa-test@domain.com + TMPDIR=/var/tmp + TODAY=`date +"%Y%m%d"` + + # Make sure compilation environment is correct + CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre' + export MAKE=gmake + export CC=gcc + + # Set test environment + export NO_INTERACTION=1 + export REPORT_EXIT_STATUS=1 + + cd $CO_DIR + cvs update . >>$TMPDIR/phpqatest.$TODAY + ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE + $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1 + if test $? -gt 0 + then + cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL + fi + ========== end of qa-test.sh ============= + +**Note:** The exit status of ``run-tests.php`` will be ``1`` when ``REPORT_EXIT_STATUS`` is set. The +result of make test may be higher than that. At present, gmake 3.79.1 returns 2, so it is advised to +test for non-zero, rather then a specific value. + +When ``make test`` finished running tests, and if there are any failed tests, the script asks to +send the logs to the PHP QA mailing list. Please answer ``y`` to this question so that we can +efficiently process the results, entering your e-mail address (which will not be transmitted in +plain text to any list) enables us to ask you some more information if a test failed. Note that this +script also uploads php -i output so your hostname may be transmitted. + +Specific tests can also be executed, like running tests for a certain extension. To do this you can +do like so (for example the standard library): + +.. code:: shell + + make test TESTS=ext/standard. + +Where ``TESTS=`` points to a directory containing .phpt files or a single .phpt file like: + +.. code:: shell + + make test TESTS=tests/basic/001.phpt. + +You can also pass options directly to the underlying script that runs the test suite +(``run-tests.phpt``) using ``TESTS=``, for example to check for memory leaks using Valgrind, the +``-m`` option can be passed along: ``make test TESTS="-m Zend/"``. For a full list of options that +can be passed along, then run ``make test TESTS=-h``. + +*Windows users:* On Windows the ``make`` command is called ``nmake`` instead of ``make``. This means +that on Windows you will have to run ``nmake test``, to run the test suite. diff --git a/docs/source/miscellaneous/writing-tests.rst b/docs/source/miscellaneous/writing-tests.rst index 8b5445cf419..fd09d80f127 100644 --- a/docs/source/miscellaneous/writing-tests.rst +++ b/docs/source/miscellaneous/writing-tests.rst @@ -1,6 +1,6 @@ -############ - Test Files -############ +############### + Writing Tests +############### ****************** phpt Test Basics @@ -501,7 +501,7 @@ configurations, preventing your test from failing due inconsistencies in the err Alternatively you can use ``--EXPECTF--`` and check for the message by replacing the path of the source of the message with ``%s`` and the line number with ``%d``. The end of a message in a test file ``example.phpt`` then looks like ``in %sexample.php on line %d``. We explicitly dropped the -last path devider as that is a system dependent character ``/`` or ``\``. +last path divider as that is a system dependent character ``/`` or ``\``. Last bit ========