JS Tokenizer generates errors when processing some decimals

[photodude]

In trying to figure out why testing with my custom standard is failing to complete, I ran a quick test against the Pear standard in PHPCS just to make sure things are running right. I ended up getting the following errors at about 74%

PHP Notice:  Undefined index: bracket_closer in /CodeSniffer/Tokenizers/JS.php on line 1062

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. PHP_CodeSniffer_File->_parse() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:471
PHP   9. PHP_CodeSniffer_File::tokenizeString() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:689
PHP  10. PHP_CodeSniffer_Tokenizers_JS->processAdditional() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:1447

This error

PHP Notice:  Undefined index: content in /CodeSniffer/Fixer.php on line 139

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. PHP_CodeSniffer_Fixer->startFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:472

This error

PHP Notice:  Undefined index: bracket_closer in CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 951

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. Generic_Sniffs_WhiteSpace_ScopeIndentSniff->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:567

this error

PHP Notice:  Undefined index: line in CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 952

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. Generic_Sniffs_WhiteSpace_ScopeIndentSniff->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:567

this error

PHP Notice:  Undefined offset: 2845 in CodeSniffer/Standards/Generic/Sniffs/WhiteSpace/ScopeIndentSniff.php on line 217

PHP Stack trace:
PHP   1. {main}() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /home/travis/.phpenv/versions/5.6.5/bin/phpcs:25
PHP   3. PHP_CodeSniffer_CLI->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:104
PHP   4. PHP_CodeSniffer->processFiles() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/CLI.php:886
PHP   5. PHP_CodeSniffer->processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:619
PHP   6. PHP_CodeSniffer->_processFile() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1714
PHP   7. PHP_CodeSniffer_File->start() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer.php:1836
PHP   8. Generic_Sniffs_WhiteSpace_ScopeIndentSniff->process() /home/travis/.phpenv/versions/5.6.5/pear/PHP/CodeSniffer/File.php:567

and lots more errors
see Travis-ci RAW log for details and all the errors
Also Failing with similar errors using the PSR2 standard
And with the PSR1 standard
And with the PHPCS standard

• a file that you analyze: Testing against this Repo on Travis-CI
• standard definition you're using: PEAR (for this test only)
• used PHP_CodeSniffer version: Current stable 2.5.1
• used PHP version: 5.6.5

[gsherwood]

This is the file that is causing the errors: media/editors/codemirror/addon/search/search.js

[gsherwood]

Smallest bit of code to replicate:

function() {
  if (condition)
    foo = .4
}

[photodude]

I made a tweak to the file you mentioned to bypass this bug for the moment. (just added a 0 in front of the decimal as a temporary work around)

After bypassing the bug, Looks like there is another file in this repo throwing an error in the JS Tokenizer
/media/editors/codemirror/mode/soy/soy.js

[gsherwood]

After bypassing the bug, Looks like there is another file in this repo throwing an error in the JS Tokenizer
/media/editors/codemirror/mode/soy/soy.js

Yep, I'm on that now. It's a regex tokenizing issue.

[gsherwood]

Both those issues are now fixed. Are you able to give it a test with your custom standard?

[photodude]

It took some work, and a lot of trial and error, but I finally figured out getting the PHPCS master branch and the custom standard in a state that I can run them on Travis-CI.

Looks good, for the fix on the second file

I'll have to run another test with my temporary workaround for the decimal bug removed. But it's probably not an issue anymore.

[gsherwood]

Thanks for testing. I'll close this issue, but please post again if it's not fixed for you.

[photodude]

Will do, Thanks for addressing this issue. (now I just need to fix the issue with the FileCommentSniff in our custom standard, maybe someday before phpcs 3 we'll get the our custom standard out of a beta state)