###############
 Running Tests
###############

The easiest way to test your PHP build is to run make test from the command line after successfully
compiling. This will run the all tests for all enabled functionalities and extensions located in
tests folders under the source root directory using the PHP CLI binary.

``make test`` executes the ``run-tests.php`` script under the source root (parallel builds will not
work). Therefore you can execute the script as follows:

.. code:: shell

   sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]

******************************************
 Which php executable does make test use?
******************************************

If you are running the ``run-tests.php`` script from the command line (as above) you can set the
``TEST_PHP_EXECUTABLE`` environment variable to explicitly select the PHP executable that is to be
tested, that is, used to run the test scripts, otherwise it will use the PHP CLI binary that you
have compiled (``sapi/cli/php``).

If you run the tests using make test, the PHP CLI and CGI executables are automatically set for you.
``make test`` executes ``run-tests.php`` script with the CLI binary. Some test scripts such as
session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all
tests.

**Note:** The PHP binary executing ``run-tests.php`` and the PHP binary used for executing test
scripts may differ. If you use different PHP binary for executing ``run-tests.php`` script, you may
get errors.

************************
 Which php.ini is used?
************************

``make test`` uses the same ``php.ini`` file as it would once installed. The tests have been written
to be independent of that ``php.ini`` file, so if you find a test that is affected by a setting,
please report this, so we can address the issue.

**********************************
 Which test scripts are executed?
**********************************

The ``run-tests.php`` (``make test``), without any arguments executes all test scripts by extracting
all directories named tests from the source root and any subdirectories below. If there are files,
which have a phpt extension, ``run-tests.php`` looks at the sections in these files, determines
whether it should run it, by evaluating the ``SKIPIF`` section. If the test is eligible for
execution, the ``FILE`` section is extracted into a ``.php`` file (with the same name besides the
extension) and gets executed. When an argument is given or ``TESTS`` environment variable is set,
the GLOB is expanded by the shell and any file with extension ``*.phpt`` is regarded as a test file.

Tester can easily execute tests selectively with as follows:

.. code:: shell

   ./sapi/cli/php run-tests.php ext/mbstring/*
   ./sapi/cli/php run-tests.php ext/mbstring/020.phpt

*********************
 Test Runner Options
*********************

The ``run-tests.php`` test runner has many options. You can see these options by using the ``-h``
option with ``run-tests.php``.

You can set options by specifying them on the command line when you run ``php run-tests.php`` or if
you use ``make test`` through the ``TEST_PHP_ARGS`` environment variable:

.. code:: shell

   php run-tests.php -j24
   # or
   TEST_PHP_ARGS="-j24" make test

Running Tests in Parallel
=========================

The test runner can run tests in parallel, by using the ``-j`` option:

.. code:: shell

   php run-tests.php -j24 ext/date/*.phpt

**************
 Test results
**************

Test results are printed to standard output. If there is a failed test, the ``run-tests.php`` script
saves the result, the expected result and the code executed to the test script directory. For
example, if ``ext/myext/tests/myext.phpt`` fails to pass, the following files are created:

-  ``ext/myext/tests/myext.php`` - actual test file executed
-  ``ext/myext/tests/myext.log`` - log of test execution (L)
-  ``ext/myext/tests/myext.exp`` - expected output (E)
-  ``ext/myext/tests/myext.out`` - output from test script (O)
-  ``ext/myext/tests/myext.diff`` - diff of .out and .exp (D)

Failed tests are always bugs. Either the test is bugged or not considering factors applying to the
tester's environment, or there is a bug in PHP. If this is a known bug, we strive to provide bug
numbers, in either the test name or the file name. You can check the status of such a bug, by going
to: ``https://bugs.php.net/12345`` where 12345 is the bug number. For clarity and automated
processing, bug numbers are prefixed by a hash sign '#' in test names and/or test cases are named
``bug12345.phpt``.

**Note:** The files generated by tests can be selected by setting the environment variable
``TEST_PHP_LOG_FORMAT``. For each file you want to be generated use the character in brackets as
shown above (default is LEOD). The php file will be generated always.

**Note**: You can set environment variable ``TEST_PHP_DETAILED`` to enable detailed test
information.

*******************
 Automated testing
*******************

If you like to keep up to speed, with latest developments and quality assurance, setting the
environment variable ``NO_INTERACTION`` to 1, will not prompt the tester for any user input.

Normally, the exit status of make test is zero, regardless of the results of independent tests. Set
the environment variable ``REPORT_EXIT_STATUS`` to ``1``, and make test will set the exit status
("$?") to non-zero, when an individual test has failed.

Example script to be run by cron:

.. code:: shell

   ========== qa-test.sh =============
   #!/bin/sh

   CO_DIR=$HOME/cvs/php7
   MYMAIL=qa-test@domain.com
   TMPDIR=/var/tmp
   TODAY=`date +"%Y%m%d"`

   # Make sure compilation environment is correct
   CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
   export MAKE=gmake
   export CC=gcc

   # Set test environment
   export NO_INTERACTION=1
   export REPORT_EXIT_STATUS=1

   cd $CO_DIR
   cvs update . >>$TMPDIR/phpqatest.$TODAY
   ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
   $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
   if test $? -gt 0
   then
           cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
   fi
   ========== end of qa-test.sh =============

**Note:** The exit status of ``run-tests.php`` will be ``1`` when ``REPORT_EXIT_STATUS`` is set. The
result of make test may be higher than that. At present, gmake 3.79.1 returns 2, so it is advised to
test for non-zero, rather then a specific value.

When ``make test`` finished running tests, and if there are any failed tests, the script asks to
send the logs to the PHP QA mailing list. Please answer ``y`` to this question so that we can
efficiently process the results, entering your e-mail address (which will not be transmitted in
plain text to any list) enables us to ask you some more information if a test failed. Note that this
script also uploads php -i output so your hostname may be transmitted.

Specific tests can also be executed, like running tests for a certain extension. To do this you can
do like so (for example the standard library):

.. code:: shell

   make test TESTS=ext/standard.

Where ``TESTS=`` points to a directory containing .phpt files or a single .phpt file like:

.. code:: shell

   make test TESTS=tests/basic/001.phpt.

You can also pass options directly to the underlying script that runs the test suite
(``run-tests.phpt``) using ``TESTS=``, for example to check for memory leaks using Valgrind, the
``-m`` option can be passed along: ``make test TESTS="-m Zend/"``. For a full list of options that
can be passed along, then run ``make test TESTS=-h``.

*Windows users:* On Windows the ``make`` command is called ``nmake`` instead of ``make``. This means
that on Windows you will have to run ``nmake test``, to run the test suite.