Convert https://qa.php.net/running-tests.php

docs/source/index.rst

@@ -21,6 +21,7 @@
 
    miscellaneous/stubs
    miscellaneous/writing-tests
+   miscellaneous/running-tests
 
 Welcome to the php-src documentation!
 

docs/source/miscellaneous/running-tests.rst

@@ -0,0 +1,160 @@
+###############
+ Running Tests
+###############
+
+The easiest way to test your PHP build is to run make test from the command line after successfully
+compiling. This will run the all tests for all enabled functionalities and extensions located in
+tests folders under the source root directory using the PHP CLI binary.
+
+``make test`` executes the ``run-tests.php`` script under the source root (parallel builds will not
+work). Therefore you can execute the script as follows:
+
+.. code:: shell
+
+   TEST_PHP_EXECUTABLE=sapi/cli/php \
+   sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
+
+******************************************
+ Which php executable does make test use?
+******************************************
+
+If you are running the ``run-tests.php`` script from the command line (as above) you must set the
+``TEST_PHP_EXECUTABLE`` environment variable to explicitly select the PHP executable that is to be
+tested, that is, used to run the test scripts.
+
+If you run the tests using make test, the PHP CLI and CGI executables are automatically set for you.
+``make test`` executes ``run-tests.php`` script with the CLI binary. Some test scripts such as
+session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all
+tests.
+
+**Note:** The PHP binary executing ``run-tests.php`` and the PHP binary used for executing test
+scripts may differ. If you use different PHP binary for executing ``run-tests.php`` script, you may
+get errors.
+
+************************
+ Which php.ini is used?
+************************
+
+``make test`` uses the same ``php.ini`` file as it would once installed. The tests have been written
+to be independent of that ``php.ini`` file, so if you find a test that is affected by a setting,
+please report this, so we can address the issue.
+
+**********************************
+ Which test scripts are executed?
+**********************************
+
+The ``run-tests.php`` (``make test``), without any arguments executes all test scripts by extracting
+all directories named tests from the source root and any subdirectories below. If there are files,
+which have a phpt extension, ``run-tests.php`` looks at the sections in these files, determines
+whether it should run it, by evaluating the ``SKIPIF`` section. If the test is eligible for
+execution, the ``FILE`` section is extracted into a ``.php`` file (with the same name besides the
+extension) and gets executed. When an argument is given or ``TESTS`` environment variable is set,
+the GLOB is expanded by the shell and any file with extension ``*.phpt`` is regarded as a test file.
+
+Tester can easily execute tests selectively with as follows:
+
+.. code:: shell
+
+   ./sapi/cli/php run-tests.php ext/mbstring/*
+   ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
+
+**************
+ Test results
+**************
+
+Test results are printed to standard output. If there is a failed test, the ``run-tests.php`` script
+saves the result, the expected result and the code executed to the test script directory. For
+example, if ``ext/myext/tests/myext.phpt`` fails to pass, the following files are created:
+
+-  ``ext/myext/tests/myext.php`` - actual test file executed
+-  ``ext/myext/tests/myext.log`` - log of test execution (L)
+-  ``ext/myext/tests/myext.exp`` - expected output (E)
+-  ``ext/myext/tests/myext.out`` - output from test script (O)
+-  ``ext/myext/tests/myext.diff`` - diff of .out and .exp (D)
+
+Failed tests are always bugs. Either the test is bugged or not considering factors applying to the
+tester's environment, or there is a bug in PHP. If this is a known bug, we strive to provide bug
+numbers, in either the test name or the file name. You can check the status of such a bug, by going
+to: ``https://bugs.php.net/12345`` where 12345 is the bug number. For clarity and automated
+processing, bug numbers are prefixed by a hash sign '#' in test names and/or test cases are named
+``bug12345.phpt``.
+
+**Note:** The files generated by tests can be selected by setting the environment variable
+``TEST_PHP_LOG_FORMAT``. For each file you want to be generated use the character in brackets as
+shown above (default is LEOD). The php file will be generated always.
+
+**Note**: You can set environment variable ``TEST_PHP_DETAILED`` to enable detailed test
+information.
+
+*******************
+ Automated testing
+*******************
+
+If you like to keep up to speed, with latest developments and quality assurance, setting the
+environment variable ``NO_INTERACTION`` to 1, will not prompt the tester for any user input.
+
+Normally, the exit status of make test is zero, regardless of the results of independent tests. Set
+the environment variable ``REPORT_EXIT_STATUS`` to ``1``, and make test will set the exit status
+("$?") to non-zero, when an individual test has failed.
+
+Example script to be run by cron:
+
+.. code:: shell
+
+   ========== qa-test.sh =============
+   #!/bin/sh
+
+   CO_DIR=$HOME/cvs/php7
+   MYMAIL=qa-test@domain.com
+   TMPDIR=/var/tmp
+   TODAY=`date +"%Y%m%d"`
+
+   # Make sure compilation environment is correct
+   CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
+   export MAKE=gmake
+   export CC=gcc
+
+   # Set test environment
+   export NO_INTERACTION=1
+   export REPORT_EXIT_STATUS=1
+
+   cd $CO_DIR
+   cvs update . >>$TMPDIR/phpqatest.$TODAY
+   ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
+   $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
+   if test $? -gt 0
+   then
+           cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
+   fi
+   ========== end of qa-test.sh =============
+
+**Note:** The exit status of ``run-tests.php`` will be ``1`` when ``REPORT_EXIT_STATUS`` is set. The
+result of make test may be higher than that. At present, gmake 3.79.1 returns 2, so it is advised to
+test for non-zero, rather then a specific value.
+
+When ``make test`` finished running tests, and if there are any failed tests, the script asks to
+send the logs to the PHP QA mailing list. Please answer ``y`` to this question so that we can
+efficiently process the results, entering your e-mail address (which will not be transmitted in
+plain text to any list) enables us to ask you some more information if a test failed. Note that this
+script also uploads php -i output so your hostname may be transmitted.
+
+Specific tests can also be executed, like running tests for a certain extension. To do this you can
+do like so (for example the standard library):
+
+.. code:: shell
+
+   make test TESTS=ext/standard.
+
+Where ``TESTS=`` points to a directory containing .phpt files or a single .phpt file like:
+
+.. code:: shell
+
+   make test TESTS=tests/basic/001.phpt.
+
+You can also pass options directly to the underlying script that runs the test suite
+(``run-tests.phpt``) using ``TESTS=``, for example to check for memory leaks using Valgrind, the
+``-m`` option can be passed along: ``make test TESTS="-m Zend/"``. For a full list of options that
+can be passed along, then run ``make test TESTS=-h``.
+
+*Windows users:* On Windows the ``make`` command is called ``nmake`` instead of ``make``. This means
+that on Windows you will have to run ``nmake test``, to run the test suite.

docs/source/miscellaneous/writing-tests.rst

@@ -1,6 +1,6 @@
-############
- Test Files
-############
+###############
+ Writing Tests
+###############
 
 ******************
  phpt Test Basics
@@ -501,7 +501,7 @@ configurations, preventing your test from failing due inconsistencies in the err
 Alternatively you can use ``--EXPECTF--`` and check for the message by replacing the path of the
 source of the message with ``%s`` and the line number with ``%d``. The end of a message in a test
 file ``example.phpt`` then looks like ``in %sexample.php on line %d``. We explicitly dropped the
-last path devider as that is a system dependent character ``/`` or ``\``.
+last path divider as that is a system dependent character ``/`` or ``\``.
 
 Last bit
 ========