Skip to content

Instantly share code, notes, and snippets.

@szydell
Created February 22, 2019 18:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save szydell/4982253537f404317efb0952c7afc32e to your computer and use it in GitHub Desktop.
Save szydell/4982253537f404317efb0952c7afc32e to your computer and use it in GitHub Desktop.
simdjson test ok
[szydell@laPtak simdjson]$ make SANITIZE=1 -j12 test
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o jsoncheck src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp src/stage2_build_tape.cpp tests/jsoncheck.cpp -I.
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o numberparsingcheck tests/numberparsingcheck.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp -I. -DJSON_TEST_NUMBERS
g++ -std=c++17 -march=native -Wall -Wextra -Wshadow -Iinclude -Ibenchmark/linux -g3 -O0 -fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined -o stringparsingcheck tests/stringparsingcheck.cpp src/jsonioutil.cpp src/jsonparser.cpp src/stage1_find_marks.cpp -I. -DJSON_TEST_STRINGS
./numberparsingcheck
Usage: ./numberparsingcheck <directorywithjsonfiles>
We are going to assume you mean to use the 'jsonchecker' and 'jsonexamples' directories.
File fail11.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File fail13.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail14.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail29.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail30.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail31.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail37.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File fail38.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File fail39_EXCLUDE.json is valid --- integers: 2 floats: 0 invalid: 0 total numbers: 2
include/simdjson/numberparsing.h:393:9: runtime error: signed integer overflow: 9223372036854775800 + 8 cannot be represented in type 'long int'
Warning: foundInvalidNumber 9223372036854775808
whereas strtod parses it to 9223372036854775808.000000, while parsing jsonchecker/fail40_s64boverflow.json
File fail40_s64boverflow.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1844674407370955161 * 10 cannot be represented in type 'long int'
Warning: foundInvalidNumber 18446744073709551616
whereas strtod parses it to 18446744073709551616.000000, while parsing jsonchecker/fail41_toolarge.json
File fail41_toolarge.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail43.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber -2.]
��������������������������� whereas strtod parses it to -2.000000, while parsing jsonchecker/fail44.json
File fail44.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 0.e1]
�������������������������� whereas strtod parses it to 0.000000, while parsing jsonchecker/fail45.json
File fail45.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 2.e+3]
������������������������� whereas strtod parses it to 2000.000000, while parsing jsonchecker/fail46.json
File fail46.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 2.e-3]
������������������������� whereas strtod parses it to 0.002000, while parsing jsonchecker/fail47.json
File fail47.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 2.e3]
�������������������������� whereas strtod parses it to 2000.000000, while parsing jsonchecker/fail48.json
File fail48.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber -.123]
������������������������� whereas strtod parses it to -0.123000, while parsing jsonchecker/fail49.json
File fail49.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
Warning: foundInvalidNumber 1.]
���������������������������� whereas strtod parses it to 1.000000, while parsing jsonchecker/fail50.json
File fail50.json is not valid --- integers: 0 floats: 0 invalid: 1 total numbers: 1
File fail55.json is not valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass01.json is valid --- integers: 22 floats: 10 invalid: 0 total numbers: 32
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
File pass04.json is valid --- integers: 0 floats: 2 invalid: 0 total numbers: 2
include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
File pass05.json is valid --- integers: 0 floats: 1 invalid: 0 total numbers: 1
File pass08.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass11.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass12.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass13.json is valid --- integers: 1 floats: 0 invalid: 0 total numbers: 1
File pass14.json is valid --- integers: 2 floats: 0 invalid: 0 total numbers: 2
All ok.
File apache_builds.json is valid --- integers: 2 floats: 0 invalid: 0 total numbers: 2
File canada.json is valid --- integers: 46 floats: 111080 invalid: 0 total numbers: 111126
File citm_catalog.json is valid --- integers: 14392 floats: 0 invalid: 0 total numbers: 14392
File github_events.json is valid --- integers: 149 floats: 0 invalid: 0 total numbers: 149
File instruments.json is valid --- integers: 4935 floats: 0 invalid: 0 total numbers: 4935
File marine_ik.json is valid --- integers: 130225 floats: 114950 invalid: 0 total numbers: 245175
File mesh.json is valid --- integers: 40613 floats: 32400 invalid: 0 total numbers: 73013
File mesh.pretty.json is valid --- integers: 40613 floats: 32400 invalid: 0 total numbers: 73013
File numbers.json is valid --- integers: 0 floats: 10001 invalid: 0 total numbers: 10001
File random.json is valid --- integers: 5002 floats: 0 invalid: 0 total numbers: 5002
File twitter.json is valid --- integers: 2108 floats: 1 invalid: 0 total numbers: 2109
File twitterescaped.json is valid --- integers: 2108 floats: 1 invalid: 0 total numbers: 2109
All ok.
./stringparsingcheck
Usage: ./stringparsingcheck <directorywithjsonfiles>
We are going to assume you mean to use the 'jsonchecker' and 'jsonexamples' directories.
File fail01_EXCLUDE.json is valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 58.0
File fail02.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 14.0
File fail04.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 11.0
File fail05.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 18.0
File fail07.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 21.0
File fail08.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 11.0
File fail09.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 11.0
File fail10.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 23.0
File fail11.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 18.0
File fail12.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 18.0
File fail13.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 34.0
File fail14.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 21.0
File fail15.json is not valid --- bad strings: 1
File fail17.json is not valid --- bad strings: 1
File fail18_EXCLUDE.json is valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 8.0
File fail19.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 13.0
File fail20.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 12.0
File fail21.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 22.0
File fail22.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 22.0
File fail23.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 9.0
File fail25.json is not valid --- bad strings: 1
File fail26.json is not valid --- bad strings: 1
File fail27.json is not valid --- bad strings: 1
File fail28.json is not valid --- bad strings: 1
File fail32.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 30.0
File fail33.json is not valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 8.0
File fail36.json is not valid --- bad strings: 1
File fail39_EXCLUDE.json is valid --- bad strings: 0 good strings: 3 empty strings: 0 average string length: 15.3
include/simdjson/numberparsing.h:393:9: runtime error: signed integer overflow: 9223372036854775800 + 8 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1844674407370955161 * 10 cannot be represented in type 'long int'
File fail56.json is not valid --- bad strings: 0 good strings: 1 empty strings: 1 average string length: 0.0
File pass01.json is valid --- bad strings: 0 good strings: 54 empty strings: 1 average string length: 10.7
File pass02.json is valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 12.0
File pass03.json is valid --- bad strings: 0 good strings: 5 empty strings: 0 average string length: 19.4
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
File pass10.json is valid --- bad strings: 0 good strings: 1 empty strings: 0 average string length: 3.0
File pass14.json is valid --- bad strings: 0 good strings: 2 empty strings: 0 average string length: 22.0
94 strings checked.
All ok.
File apache_builds.json is valid --- bad strings: 0 good strings: 5289 empty strings: 1 average string length: 14.6
File canada.json is valid --- bad strings: 0 good strings: 12 empty strings: 0 average string length: 7.5
File citm_catalog.json is valid --- bad strings: 0 good strings: 26604 empty strings: 0 average string length: 8.3
File github_events.json is valid --- bad strings: 0 good strings: 1891 empty strings: 5 average string length: 24.2
File gsoc-2018.json is valid --- bad strings: 0 good strings: 34128 empty strings: 0 average string length: 86.3
File instruments.json is valid --- bad strings: 0 good strings: 6889 empty strings: 366 average string length: 10.1
File marine_ik.json is valid --- bad strings: 0 good strings: 38268 empty strings: 0 average string length: 3.3
File mesh.json is valid --- bad strings: 0 good strings: 11 empty strings: 0 average string length: 8.4
File mesh.pretty.json is valid --- bad strings: 0 good strings: 11 empty strings: 0 average string length: 8.4
File random.json is valid --- bad strings: 0 good strings: 33005 empty strings: 0 average string length: 10.1
File twitter.json is valid --- bad strings: 0 good strings: 18099 empty strings: 143 average string length: 20.3
File twitterescaped.json is valid --- bad strings: 0 good strings: 18099 empty strings: 143 average string length: 20.3
File update-center.json is valid --- bad strings: 0 good strings: 27229 empty strings: 27 average string length: 16.2
209535 strings checked.
All ok.
./jsoncheck
Usage: ./jsoncheck <directorywithjsonfiles>
We are going to assume you mean to use the 'jsonchecker' directory.
validating: file fail01_EXCLUDE.json ok
validating: file fail02.json invalid
validating: file fail03.json invalid
validating: file fail04.json invalid
validating: file fail05.json invalid
validating: file fail06.json invalid
validating: file fail07.json invalid
validating: file fail08.json invalid
validating: file fail09.json invalid
validating: file fail10.json invalid
validating: file fail11.json invalid
validating: file fail12.json invalid
validating: file fail13.json invalid
validating: file fail14.json invalid
validating: file fail15.json invalid
validating: file fail16.json invalid
validating: file fail17.json invalid
validating: file fail18_EXCLUDE.json ok
validating: file fail19.json invalid
validating: file fail20.json invalid
validating: file fail21.json invalid
validating: file fail22.json invalid
validating: file fail23.json invalid
validating: file fail24.json invalid
validating: file fail25.json invalid
validating: file fail26.json invalid
validating: file fail27.json invalid
validating: file fail28.json invalid
validating: file fail29.json invalid
validating: file fail30.json invalid
validating: file fail31.json invalid
validating: file fail32.json invalid
validating: file fail33.json invalid
validating: file fail34.json invalid
validating: file fail35.json invalid
validating: file fail36.json invalid
validating: file fail37.json invalid
validating: file fail38.json invalid
validating: file fail39_EXCLUDE.json ok
validating: file fail40_s64boverflow.json include/simdjson/numberparsing.h:393:9: runtime error: signed integer overflow: 9223372036854775800 + 8 cannot be represented in type 'long int'
invalid
validating: file fail41_toolarge.json include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1844674407370955161 * 10 cannot be represented in type 'long int'
invalid
validating: file fail42.json invalid
validating: file fail43.json invalid
validating: file fail44.json invalid
validating: file fail45.json invalid
validating: file fail46.json invalid
validating: file fail47.json invalid
validating: file fail48.json invalid
validating: file fail49.json invalid
validating: file fail50.json invalid
validating: file fail51.json invalid
validating: file fail52.json invalid
validating: file fail53.json invalid
validating: file fail54.json invalid
validating: file fail55.json invalid
validating: file fail56.json invalid
validating: file pass01.json ok
validating: file pass02.json ok
validating: file pass03.json ok
validating: file pass04.json include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
ok
validating: file pass05.json include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
ok
validating: file pass06.json ok
validating: file pass07.json ok
validating: file pass08.json ok
validating: file pass09.json ok
validating: file pass10.json ok
validating: file pass11.json ok
validating: file pass12.json ok
validating: file pass13.json ok
validating: file pass14.json ok
67 files checked.
All ok!
./scripts/testjson2json.sh
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/home/szydell/development/simdjson'
make[1]: 'minify' is up to date.
make[1]: 'json2json' is up to date.
make[1]: Leaving directory '/home/szydell/development/simdjson'
apache_builds.json
canada.json
citm_catalog.json
github_events.json
gsoc-2018.json
instruments.json
marine_ik.json
mesh.json
mesh.pretty.json
numbers.json
random.json
twitterescaped.json
twitter.json
update-center.json
pass01.json
pass02.json
pass03.json
pass04.json
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: 3141592653589793238 * 10 cannot be represented in type 'long int'
pass05.json
include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1234567890000000217 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:393:14: runtime error: signed integer overflow: 1234567890000000217 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:406:13: runtime error: signed integer overflow: -4611686018427387904 * 10 cannot be represented in type 'long int'
include/simdjson/numberparsing.h:425:13: runtime error: signed integer overflow: -9223372036854775808 * 10 cannot be represented in type 'long int'
pass06.json
pass07.json
pass08.json
pass09.json
pass10.json
pass11.json
pass12.json
pass13.json
pass14.json
test successful
It looks like the code is good!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment