mirror of
https://github.com/php/php-src.git
synced 2025-08-16 05:58:45 +02:00
160 lines
7 KiB
ReStructuredText
160 lines
7 KiB
ReStructuredText
###############
|
|
Running Tests
|
|
###############
|
|
|
|
The easiest way to test your PHP build is to run make test from the command line after successfully
|
|
compiling. This will run the all tests for all enabled functionalities and extensions located in
|
|
tests folders under the source root directory using the PHP CLI binary.
|
|
|
|
``make test`` executes the ``run-tests.php`` script under the source root (parallel builds will not
|
|
work). Therefore you can execute the script as follows:
|
|
|
|
.. code:: shell
|
|
|
|
TEST_PHP_EXECUTABLE=sapi/cli/php \
|
|
sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
|
|
|
|
******************************************
|
|
Which php executable does make test use?
|
|
******************************************
|
|
|
|
If you are running the ``run-tests.php`` script from the command line (as above) you must set the
|
|
``TEST_PHP_EXECUTABLE`` environment variable to explicitly select the PHP executable that is to be
|
|
tested, that is, used to run the test scripts.
|
|
|
|
If you run the tests using make test, the PHP CLI and CGI executables are automatically set for you.
|
|
``make test`` executes ``run-tests.php`` script with the CLI binary. Some test scripts such as
|
|
session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all
|
|
tests.
|
|
|
|
**Note:** The PHP binary executing ``run-tests.php`` and the PHP binary used for executing test
|
|
scripts may differ. If you use different PHP binary for executing ``run-tests.php`` script, you may
|
|
get errors.
|
|
|
|
************************
|
|
Which php.ini is used?
|
|
************************
|
|
|
|
``make test`` uses the same ``php.ini`` file as it would once installed. The tests have been written
|
|
to be independent of that ``php.ini`` file, so if you find a test that is affected by a setting,
|
|
please report this, so we can address the issue.
|
|
|
|
**********************************
|
|
Which test scripts are executed?
|
|
**********************************
|
|
|
|
The ``run-tests.php`` (``make test``), without any arguments executes all test scripts by extracting
|
|
all directories named tests from the source root and any subdirectories below. If there are files,
|
|
which have a phpt extension, ``run-tests.php`` looks at the sections in these files, determines
|
|
whether it should run it, by evaluating the ``SKIPIF`` section. If the test is eligible for
|
|
execution, the ``FILE`` section is extracted into a ``.php`` file (with the same name besides the
|
|
extension) and gets executed. When an argument is given or ``TESTS`` environment variable is set,
|
|
the GLOB is expanded by the shell and any file with extension ``*.phpt`` is regarded as a test file.
|
|
|
|
Tester can easily execute tests selectively with as follows:
|
|
|
|
.. code:: shell
|
|
|
|
./sapi/cli/php run-tests.php ext/mbstring/*
|
|
./sapi/cli/php run-tests.php ext/mbstring/020.phpt
|
|
|
|
**************
|
|
Test results
|
|
**************
|
|
|
|
Test results are printed to standard output. If there is a failed test, the ``run-tests.php`` script
|
|
saves the result, the expected result and the code executed to the test script directory. For
|
|
example, if ``ext/myext/tests/myext.phpt`` fails to pass, the following files are created:
|
|
|
|
- ``ext/myext/tests/myext.php`` - actual test file executed
|
|
- ``ext/myext/tests/myext.log`` - log of test execution (L)
|
|
- ``ext/myext/tests/myext.exp`` - expected output (E)
|
|
- ``ext/myext/tests/myext.out`` - output from test script (O)
|
|
- ``ext/myext/tests/myext.diff`` - diff of .out and .exp (D)
|
|
|
|
Failed tests are always bugs. Either the test is bugged or not considering factors applying to the
|
|
tester's environment, or there is a bug in PHP. If this is a known bug, we strive to provide bug
|
|
numbers, in either the test name or the file name. You can check the status of such a bug, by going
|
|
to: ``https://bugs.php.net/12345`` where 12345 is the bug number. For clarity and automated
|
|
processing, bug numbers are prefixed by a hash sign '#' in test names and/or test cases are named
|
|
``bug12345.phpt``.
|
|
|
|
**Note:** The files generated by tests can be selected by setting the environment variable
|
|
``TEST_PHP_LOG_FORMAT``. For each file you want to be generated use the character in brackets as
|
|
shown above (default is LEOD). The php file will be generated always.
|
|
|
|
**Note**: You can set environment variable ``TEST_PHP_DETAILED`` to enable detailed test
|
|
information.
|
|
|
|
*******************
|
|
Automated testing
|
|
*******************
|
|
|
|
If you like to keep up to speed, with latest developments and quality assurance, setting the
|
|
environment variable ``NO_INTERACTION`` to 1, will not prompt the tester for any user input.
|
|
|
|
Normally, the exit status of make test is zero, regardless of the results of independent tests. Set
|
|
the environment variable ``REPORT_EXIT_STATUS`` to ``1``, and make test will set the exit status
|
|
("$?") to non-zero, when an individual test has failed.
|
|
|
|
Example script to be run by cron:
|
|
|
|
.. code:: shell
|
|
|
|
========== qa-test.sh =============
|
|
#!/bin/sh
|
|
|
|
CO_DIR=$HOME/cvs/php7
|
|
MYMAIL=qa-test@domain.com
|
|
TMPDIR=/var/tmp
|
|
TODAY=`date +"%Y%m%d"`
|
|
|
|
# Make sure compilation environment is correct
|
|
CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
|
|
export MAKE=gmake
|
|
export CC=gcc
|
|
|
|
# Set test environment
|
|
export NO_INTERACTION=1
|
|
export REPORT_EXIT_STATUS=1
|
|
|
|
cd $CO_DIR
|
|
cvs update . >>$TMPDIR/phpqatest.$TODAY
|
|
./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
|
|
$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
|
|
if test $? -gt 0
|
|
then
|
|
cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
|
|
fi
|
|
========== end of qa-test.sh =============
|
|
|
|
**Note:** The exit status of ``run-tests.php`` will be ``1`` when ``REPORT_EXIT_STATUS`` is set. The
|
|
result of make test may be higher than that. At present, gmake 3.79.1 returns 2, so it is advised to
|
|
test for non-zero, rather then a specific value.
|
|
|
|
When ``make test`` finished running tests, and if there are any failed tests, the script asks to
|
|
send the logs to the PHP QA mailing list. Please answer ``y`` to this question so that we can
|
|
efficiently process the results, entering your e-mail address (which will not be transmitted in
|
|
plain text to any list) enables us to ask you some more information if a test failed. Note that this
|
|
script also uploads php -i output so your hostname may be transmitted.
|
|
|
|
Specific tests can also be executed, like running tests for a certain extension. To do this you can
|
|
do like so (for example the standard library):
|
|
|
|
.. code:: shell
|
|
|
|
make test TESTS=ext/standard.
|
|
|
|
Where ``TESTS=`` points to a directory containing .phpt files or a single .phpt file like:
|
|
|
|
.. code:: shell
|
|
|
|
make test TESTS=tests/basic/001.phpt.
|
|
|
|
You can also pass options directly to the underlying script that runs the test suite
|
|
(``run-tests.phpt``) using ``TESTS=``, for example to check for memory leaks using Valgrind, the
|
|
``-m`` option can be passed along: ``make test TESTS="-m Zend/"``. For a full list of options that
|
|
can be passed along, then run ``make test TESTS=-h``.
|
|
|
|
*Windows users:* On Windows the ``make`` command is called ``nmake`` instead of ``make``. This means
|
|
that on Windows you will have to run ``nmake test``, to run the test suite.
|