Tokenizer 関数
If you run token_get_all() on a string which starts with #!... (eg, #!/usr/local/bin/php) this will be lost.
If you rewrite the file using token_get_all() and writing back all tokens afterwards, the #!... will be gone.
This causes commandline executable scripts (interpreted by PHP) to be unexecutable.
token_get_all
(PHP 4 >= 4.2.0, PHP 5)
token_get_all — 指定したソースを PHP トークンに分割する
説明
array token_get_all
( string $source
)
token_get_all() は指定した文字列 source をパースし、Zend engine の 字句解析スキャナにより PHP 言語のトークンに分割します。
パーサトークンの一覧を得るには、パーサトークンの一覧 を参照するか、 あるいは token_name() でトークン値を文字列表現に 変換します。
パラメータ
- source
-
パースする PHP ソース。
返り値
トークン ID の配列を返します。配列の各要素には、1 文字単位の文字列 (例: ;、.、 >、! など...)、 またはトークンのインデックスを 0 番目の要素、トークンの文字列表現を 1 番目の要素、 行番号を 2 番目の要素とする配列が含まれます。
例
例1 token_get_all() の例
<?php
$tokens = token_get_all('<?php echo; ?>'); /* => array(
array(T_OPEN_TAG, '<?php'),
array(T_ECHO, 'echo'),
';',
array(T_CLOSE_TAG, '?>') ); */
/* 以下の例では、文字列が T_COMMENT (PHP <5 では T_ML_COMMENT)
ではなく T_INLINE_HTML とパースされていることに注意しましょう。
これは、指定した "code" の中に開始タグ/終了タグが含まれていないからです。
通常のファイルで、コメントを <?php ?> タグの外部に書いた場合にも同じようになります。 */
$tokens = token_get_all('/* comment */'); // => array(array(T_INLINE_HTML, '/* comment */'));
?>
変更履歴
| バージョン | 説明 |
|---|---|
| 5.2.2 | 2 番目の要素として行番号を返すようになりました。 |
add a note
User Contributed Notestoken_get_all
strrev xc.noxeh@ellij
22-Dec-2008 03:45
22-Dec-2008 03:45
kevin at metalaxe dot com
27-Apr-2008 12:58
27-Apr-2008 12:58
Rogier, thanks for that fix. This bug still exists in php 5.2.5. I did notice though that it is possible for a notice to pop up from your code. Changing this line:
$temp[] = $tokens[0][2];
To read this:
$temp[] = isset($tokens[0][2])?$tokens[0][2]:'unknown';
fixes this notice.
rogier
10-Jan-2008 09:01
10-Jan-2008 09:01
Complementary note to code below:
Note that only the FIRST 2 (or 3, if needed) array elements will be updated.
Since I only encountered incorrect results on the FIRST occurence of T_OPEN_TAG, I wrote this quick fix.
Any other following T_OPEN_TAG are, on my testing system (Apache 2.0.52, PHP 5.0.3), parsed correctly.
So, This function assumes only a possibly incorrect first T_OPEN_TAG.
Also, this function assumes the very first element (and ONLY the first element) of the token array to be the possibly incorrect token.
This effectively translates to the first character of the tokenized source to be the start of a php script opening tag '<', followed by either 'php' OR '%' (ASP_style)
rogier at dsone dot nl
10-Jan-2008 06:37
10-Jan-2008 06:37
On several PHP versions (pre 5.1), if token_get_all is used, the result will NOT always return the correct result.
This bug will only show (as far as I know) when PHP is loaded as a module. In the CLI the bug seems non-existent.
Related here are bugs 29761 and 34782.
To work around this, here's a fixing function:
<?php
//fixes related bugs: 29761, 34782 => token_get_all returns <?php NOT as T_OPEN_TAG
function token_fix( &$tokens ) {
if (!is_array($tokens) || (count($tokens)<2)) {
return;
}
//return of no fixing needed
if (is_array($tokens[0]) && (($tokens[0][0]==T_OPEN_TAG) || ($tokens[0][0]==T_OPEN_TAG_WITH_ECHO)) ) {
return;
}
//continue
$p1 = (is_array($tokens[0])?$tokens[0][1]:$tokens[0]);
$p2 = (is_array($tokens[1])?$tokens[1][1]:$tokens[1]);
$p3 = '';
if (($p1.$p2 == '<?') || ($p1.$p2 == '<%')) {
$type = ($p2=='?')?T_OPEN_TAG:T_OPEN_TAG_WITH_ECHO;
$del = 2;
//update token type for 3rd part?
if (count($tokens)>2) {
$p3 = is_array($tokens[2])?$tokens[2][1]:$tokens[2];
$del = (($p3=='php') || ($p3=='='))?3:2;
$type = ($p3=='=')?T_OPEN_TAG_WITH_ECHO:$type;
}
//rebuild erroneous token
$temp = array($type, $p1.$p2.$p3);
if (version_compare(phpversion(), '5.2.2', '<' )===false) {
$temp[] = $token[0][2];
}
//rebuild
$tokens[1] = '';
if ($del==3) $tokens[2]='';
$tokens[0] = $temp;
}
return;
}
?>
nicolas dot grekas+php at gmail dot com
03-Dec-2007 11:10
03-Dec-2007 11:10
Well, there is a way to parse for errors. See
http://www.php.net/manual/function.php-check-syntax.php#77318
smp_info at yahoo dot com
30-Nov-2007 04:50
30-Nov-2007 04:50
As far as I am aware, there is no way to tell if the source code passed is free of parse errors.
You might come across such a situation when you're using PHP to analyze PHP source code.
In a case like this.. You'll get a warning similar to (but varying) Warning: Unexpected character in input: ''' (ASCII=39) state=1
If it doesn't matter to you that the source is free of parse errors, use @token_get_all($source) to suppress the error.
phpcomments at majiclab dot com
01-Aug-2005 08:08
01-Aug-2005 08:08
Regarding bertrand at toggg dot com's comment: there is another case of the { } curly braces being used in PHP, but the token_get_all() function treats it just like a code block: string index. Example:
<?php
$text = "Hello";
if ($text{ 0 } == 'H') {
echo "This example uses { for both a PHP block and a string index.";
}
?>
Just in case some people were wondering. Since PHP treats them as the same token, it makes some things a little more interesting for parsing. You can't just assume that { ... } is a code block, it could just be a number referring to an index of a string.
bertrand at toggg dot com
08-Mar-2005 08:41
08-Mar-2005 08:41
If you want to retrieve the PHP blocks then you will count up the opening curly braces '{' and down the closing ones '}' (counter zero means block finished)
CAUTION: the opening curly braces token can take 3 values:
1) '{' for all PHP code blocks,
2) T_CURLY_OPEN for "protected" variables within strings as "{$var}"
3) T_DOLLAR_OPEN_CURLY_BRACES for extended format "${var}"
On the other hand, closing token is allways '}' !
So counting up must take place on the 3 tokens:
'{' , T_CURLY_OPEN and T_DOLLAR_OPEN_CURLY_BRACES
Have fun with PHP tokenizer !
bishop
08-Dec-2004 08:58
08-Dec-2004 08:58
You may want to know the line and column number at which a token begins (or ends). Since this tokenizer interface doesn't provide that information, you have to track it manually, like below:
<?php
function update_line_and_column_positions($c, &$line, &$col)
{
// update line count
$numNewLines = substr_count($c, "\n");
if (1 <= $numNewLines) {
// have new lines, add them in
$line += $numNewLines;
$col = 1;
// skip to right past the last new line, as it won't affect the column position
$c = substr($c, strrpos($c, "\n") + 1);
if ($c === false) {
$c = '';
}
}
// update column count
$col += strlen($c);
}
?>
Now use it, something like:
<?php
$line = 1;
$col = 1;
foreach ($tokens as $token) {
if (is_array($token)) {
list ($token, $text) = $token;
} else if (is_string($token)) {
$text = $token;
}
update_line_and_column_positions($text, $line, $col);
}
?>
Note this assumes that your desired coordinate system is 1-based (eg (1,1) is the upper left). Zero-based is left as an exercise for the reader.
Leon Atkinson
07-Dec-2002 01:17
07-Dec-2002 01:17
This function parses PHP code. Here's an example of it's use.
<?
$code = '<?$a = 3;?>';
foreach(token_get_all($code) as $c)
{
if(is_array($c))
{
print(token_name($c[0]) . ": '" . htmlentities($c[1]) . "'\n");
}
else
{
print("$c\n");
}
}
?>