so, for the below script, we store the values in a text file that we read into mysql.
we then have 1000 unique URLs per row in a table, separated by a colon. We read and split one row at a time into a redis set, then pop values off the set whenever we need a new url. When the set is empty, we read the next row from mysql into the set.
The only thing to look out for is your collation in the db. A lookup is
utf8_ci (case-insnsitive) for queries unless you set your collation to
collation-server = utf8_bin init-connect='SET NAMES utf8' init_connect='SET collation_connection = utf8_bin'