This benchmark has been misleading for a while. It was originally made to demonstrate how JIT compilers can do all sorts of crazy stuff to your code - especially LuaJIT - and was meant to be a starting point of discussion about what exactly LuaJIT does and how.
As a result, its not indicative of what its performance may be on more realistic data. Differences can be expected because
- the text will not consist of hard-coded constants
- the number of words (and therefore the dictionary) would be larger, and JIT compilers for JS and Lua often have special optimizations for small dictionaries/tables
- the words wont be pre-split, and allocating new words adds significant performance penalty (in that case a trie would probably outperform other approaches)
Here is a faster (than luajit) C++ version. There was no need to use std::string since all strings in this micro-benchmark are constants and not using any of std::string's facilities. Same thing with std::vector, it could be replaced with std::array.
I also changed the hash function's implementation to use the trick @funny-falcon mentioned.