ec348a12d1
Starting many years ago, libmbfl included a 'mblen_table' for selected
text encodings. This table allows looking up the byte length of a
(possibly multi-byte) character from the value of the first byte.
libmbfl uses these tables to optimize certain operations; if a
text-processing operation can be performed using an mblen_table,
it may not be necessary to decode the text to codepoints. Since
libmbfl's decoding filters are generally slow, this improves
performance.
Since mbstring is (or was) based on libmbfl, it has always used
these mblen_tables to implement some functions. This design has a
significant downside. Let me explain:
While some mbstring functions are implemented by converting input text
to codepoints and operating on the codepoints, others operate directly
on the original input bytes (using an mblen_table to identify character
boundaries). Both of these implementation styles, if correctly coded,
yield equivalent results on valid strings. However, on strings which
contain encoding errors, the results are often different.
When decoding byte strings to codepoints using some text encoding,
mbstring uses the non-existent codepoint 0xFFFFFFFF to represent a
byte sequence which cannot be decoded. Then, when mbstring indexes into
the resulting sequence of codepoints, the index of any particular
character depends on the number of such 'error markers' which were
produced during the decoding process. In contrast, when an mblen_table
is used to split a byte sequence into characters, there is no question
of counting encoding errors; rather, table lookups into the mblen_table
are used to repeatedly 'bite off' some number of bytes (which are
treated as one 'character'). In the presence of encoding errors, these
two methods of mapping between byte indices and character indices are
inherently different and will rarely agree.
(For completeness, it must be said that some internal mbstring code
which operates only on UTF-8 text uses a third method for mapping
between byte indices and character indices, that is: counting
non-continuation UTF-8 bytes, which are all bytes whose binary
representation is NOT like 0b10xxxxxx. This method happens to agree with
the method which involves decoding the input text to codepoints and then
counting the codepoints.)
I have been aware of this issue for years, but only recently became
aware that in the case of mb_strstr, mb_strpos, and mb_substr,
this issue can cause seriously unintuitive behavior (and even security
vulnerabilities). This was reported by Stefan Schiller.
Stefan Schiller shared the following example for mb_strstr:
var_dump(mb_strstr("\xf0start", "start", false, "UTF-8"));
// string(2) "rt"
Similarly, when mb_strpos and mb_substr are used to identify and
extract a substring from a string with encoding errors, Stefan Schiller
pointed out that the extracted portion may be completely different than
desired. This is because (for UTF-8 strings) mb_strpos works by counting
non-continuation bytes, but mb_substr uses an mblen_table.
Since some mbstring functions *cannot* be implemented using an
mblen_table, as long as mblen_tables are used, similar inconsistencies
cannot be totally avoided. But the mblen_tables are critical to
mbstring's performance. Or are they? Benchmarking mb_substr on various
UTF-8, SJIS, and EUC-JP strings revealed something interesting.
On all SJIS and EUC-JP test cases, mb_substr was slightly faster when
the mblen_table based code was deleted. For some UTF-8 test cases, the
mblen_table-based code was a tiny bit faster, while for others the
fallback code was a touch faster; in no case was the difference
significant.
Therefore, the simple fix is to delete the mblen_table-based
implementation of mb_substr.
Aside from making the function behave consistently with other mbstring
functions on invalid strings, there is ONE case where behavior is now
different on valid strings: that is, on SJIS-Mac (MacJapanese) strings
which contain any of the following code units:
0x85AB-0x85AD, 0x85BF, 0x85C0, 0x85C1, 0x8645, 0x864B, 0x865D, 0x869E,
0x86CE, 0x86D3-0x86D5, 0x86D6, 0x8971, 0x8792, 0x879D, 0x87FB, 0x87FC,
0xEB41, 0xEB42, 0xEB50, 0xEB5B, 0xEB5D, 0xEB60-0xEB6E, and all from
0xEB81 and above.
All of these SJIS-Mac code units share the (very unusual) property that
they do not correspond to any one Unicode codepoint. When converting
from SJIS-Mac to Unicode, these must be converted to 2, 3, 4, or 5
codepoints each.
The previous, mblen_table-based implementation of mb_substr would treat
all of these SJIS-Mac byte sequences as 'one character'. Now, they are
treated as multiple characters (one for each of the Unicode codepoints
which they decode to). The new behavior is more consistent with other
mbstring functions.
I don't know if SJIS-Mac users will like this change or not (probably
most will never notice), but the BC break is justified by the very
real security impact of the previous, inconsistent behavior.
Finally, I should comment on whether similar changes are needed
elsewhere. The remaining functions which use an mblen_table are:
mb_str_split, mb_strcut, and various search functions (such as
mb_strpos). The search functions are only affected now when they
receive a positive 'offset' parameter specifying where to start
searching from.
The search functions should definitely be fixed so they do not use
an mblen_table to implement the 'offset' parameter. I am not convinced
that there is any good reason to change mb_str_split and mb_strcut.
|
||
|---|---|---|
| .circleci | ||
| .github | ||
| appveyor | ||
| benchmark | ||
| build | ||
| docs | ||
| ext | ||
| main | ||
| pear | ||
| sapi | ||
| scripts | ||
| tests | ||
| travis | ||
| TSRM | ||
| win32 | ||
| Zend | ||
| .cirrus.yml | ||
| .editorconfig | ||
| .gdbinit | ||
| .gitattributes | ||
| .gitignore | ||
| .travis.yml | ||
| buildconf | ||
| buildconf.bat | ||
| CODEOWNERS | ||
| CODING_STANDARDS.md | ||
| configure.ac | ||
| CONTRIBUTING.md | ||
| EXTENSIONS | ||
| LICENSE | ||
| NEWS | ||
| php.ini-development | ||
| php.ini-production | ||
| README.md | ||
| README.REDIST.BINS | ||
| run-tests.php | ||
| SECURITY.md | ||
| UPGRADING | ||
| UPGRADING.INTERNALS | ||
The PHP Interpreter
PHP is a popular general-purpose scripting language that is especially suited to web development. Fast, flexible and pragmatic, PHP powers everything from your blog to the most popular websites in the world. PHP is distributed under the PHP License v3.01.
Documentation
The PHP manual is available at php.net/docs.
Installation
Prebuilt packages and binaries
Prebuilt packages and binaries can be used to get up and running fast with PHP.
For Windows, the PHP binaries can be obtained from
windows.php.net. After extracting the archive the
*.exe files are ready to use.
For other systems, see the installation chapter.
Building PHP source code
For Windows, see Build your own PHP on Windows.
For a minimal PHP build from Git, you will need autoconf, bison, and re2c. For a default build, you will additionally need libxml2 and libsqlite3.
On Ubuntu, you can install these using:
sudo apt install -y pkg-config build-essential autoconf bison re2c \
libxml2-dev libsqlite3-dev
On Fedora, you can install these using:
sudo dnf install re2c bison autoconf make libtool ccache libxml2-devel sqlite-devel
Generate configure:
./buildconf
Configure your build. --enable-debug is recommended for development, see
./configure --help for a full list of options.
# For development
./configure --enable-debug
# For production
./configure
Build PHP. To speed up the build, specify the maximum number of jobs using -j:
make -j4
The number of jobs should usually match the number of available cores, which
can be determined using nproc.
Testing PHP source code
PHP ships with an extensive test suite, the command make test is used after
successful compilation of the sources to run this test suite.
It is possible to run tests using multiple cores by setting -jN in
TEST_PHP_ARGS:
make TEST_PHP_ARGS=-j4 test
Shall run make test with a maximum of 4 concurrent jobs: Generally the maximum
number of jobs should not exceed the number of cores available.
The qa.php.net site provides more detailed info about testing and quality assurance.
Installing PHP built from source
After a successful build (and test), PHP may be installed with:
make install
Depending on your permissions and prefix, make install may need super user
permissions.
PHP extensions
Extensions provide additional functionality on top of PHP. PHP consists of many essential bundled extensions. Additional extensions can be found in the PHP Extension Community Library - PECL.
Contributing
The PHP source code is located in the Git repository at github.com/php/php-src. Contributions are most welcome by forking the repository and sending a pull request.
Discussions are done on GitHub, but depending on the topic can also be relayed to the official PHP developer mailing list internals@lists.php.net.
New features require an RFC and must be accepted by the developers. See Request for comments - RFC and Voting on PHP features for more information on the process.
Bug fixes don't require an RFC. If the bug has a GitHub issue, reference it in
the commit message using GH-NNNNNN. Use #NNNNNN for tickets in the old
bugs.php.net bug tracker.
Fix GH-7815: php_uname doesn't recognise latest Windows versions
Fix #55371: get_magic_quotes_gpc() throws deprecation warning
See Git workflow for details on how pull requests are merged.
Guidelines for contributors
See further documents in the repository for more information on how to contribute:
Credits
For the list of people who've put work into PHP, please see the PHP credits page.