so, for the above script, we store the values in a text file that we read into mysql.
we then have 1000 unique URLs in a table. We read one row at a time into a redis set, then pop values off the set whenever we need a new url. When the set is empty, we read the next row from mysql into the set.
The only thing to look out for is your collation in the db. A lookup is utf8_ci
(case-insnsitive) for queries unless you set your collation to utf8_bin
collation-server = utf8_bin
init-connect='SET NAMES utf8'
init_connect='SET collation_connection = utf8_bin'
skip-character-set-client-handshake
character-set-server = utf8, so you have to set that up in your db.