so, for the above script, we store the values in a text file that we read into mysql.
we then have 1000 unique URLs in a table. We read one row at a time into a redis set, then pop values off the set whenever we need a new url. When the set is empty, we read the next row from mysql into the set.
The only thing to look out for is your collation in the db. A lookup is
utf8_ci (case-insnsitive) for queries unless you set your collation to
collation-server = utf8_bin init-connect='SET NAMES utf8' init_connect='SET collation_connection = utf8_bin' skip-character-set-client-handshake character-set-server = utf8, so you have to set that up in your db.