Skip to content

Instantly share code, notes, and snippets.

@derwiki
Created June 4, 2012 22:31
Show Gist options
  • Save derwiki/2871216 to your computer and use it in GitHub Desktop.
Save derwiki/2871216 to your computer and use it in GitHub Desktop.
Bulk loading 500m rows into MySQL
for month_table in action*.txt; do
echo "$(date) splitting $month_table..."
split -l 1000000 $month_table curmonth_
for segment in curmonth_*; do
echo "On segment $segment"
time mysql -hlocalhost action_credit_silo <<-SQL
SET FOREIGN_KEY_CHECKS = 0;
SET UNIQUE_CHECKS = 0;
SET SESSION tx_isolation='READ-UNCOMMITTED';
SET sql_log_bin = 0;
LOAD DATA INFILE '$segment' INTO TABLE actioncredits
FIELDS TERMINATED by '\t' ENCLOSED BY '\"'
(@throwawayId, action, user_id, target_user_id, cause_id, item_type, item_id, activity_id, created_at, utm_campaign) ;
SQL
rm $segment
done
mv $month_table $month_table.done
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment