Filter Service Batch Job

FilterServiceBatchUpdateV4.ipynb

{"nbformat_minor": 0, "cells": [{"execution_count": 1, "cell_type": "code", "source": "import moztelemetry.filter_service as filter_service\n\nfrom datetime import datetime\nfrom dateutil.relativedelta import relativedelta", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 2, "cell_type": "code", "source": "yesterday = (datetime.now() - relativedelta(days=1)).strftime(\"%Y%m%d\")\none_week_ago = (datetime.now() - relativedelta(days=7)).strftime(\"%Y%m%d\")", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 3, "cell_type": "code", "source": "%time filter_service.main(from_date=one_week_ago, to_date=yesterday, dataset=\"telemetry\")", "outputs": [{"output_type": "stream", "name": "stdout", "text": "Bucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151002\nLooked at 0 total records in 1.451616 seconds, added 0\nLooked at 100000 total records in 233.610911 seconds, added 100000\nLooked at 200000 total records in 477.807268 seconds, added 200000\nOverall, added 236021 of 236021 in 608.027914 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151002 - previous: 236021, current: 236021, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151002 - lambda 236021, total 236021, (0.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151003\nLooked at 0 total records in 1.110643 seconds, added 0\nLooked at 100000 total records in 235.457071 seconds, added 100000\nLooked at 200000 total records in 473.270766 seconds, added 200000\nOverall, added 233150 of 233150 in 596.780438 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151003 - previous: 233150, current: 233150, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151003 - lambda 233150, total 233150, (0.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151004\nLooked at 0 total records in 1.13572 seconds, added 0\nLooked at 100000 total records in 232.688548 seconds, added 100000\nLooked at 200000 total records in 475.343551 seconds, added 200000\nOverall, added 235111 of 235111 in 606.934739 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151004 - previous: 235111, current: 235111, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151004 - lambda 235111, total 235111, (0.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151005\nLooked at 0 total records in 1.038756 seconds, added 0\nLooked at 100000 total records in 235.245512 seconds, added 100000\nLooked at 200000 total records in 477.941177 seconds, added 200000\nOverall, added 251470 of 251470 in 610.210706 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151005 - previous: 251470, current: 251470, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151005 - lambda 251470, total 251470, (0.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151006\nLooked at 0 total records in 1.019273 seconds, added 0\nLooked at 100000 total records in 240.478825 seconds, added 100000\nLooked at 200000 total records in 477.998343 seconds, added 200000\nOverall, added 253259 of 253259 in 593.220003 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151006 - previous: 253259, current: 253259, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151006 - lambda 253259, total 253259, (0.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151007\nLooked at 0 total records in 0.949276 seconds, added 0\nLooked at 100000 total records in 234.240323 seconds, added 100000\nLooked at 200000 total records in 474.700411 seconds, added 200000\nOverall, added 229196 of 229196 in 587.12282 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151007 - previous: 229196, current: 229196, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151007 - lambda 224545, total 229196, (2.02926752648% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-2 - Date: 20151008\nLooked at 0 total records in 1.161397 seconds, added 0\nLooked at 100000 total records in 238.349519 seconds, added 100000\nLooked at 200000 total records in 482.880717 seconds, added 200000\nOverall, added 250529 of 250529 in 600.42429 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151008 - previous: 250529, current: 250529, added: 0 (0.0% missing)\nAWS lambda stats:\nDay 20151008 - lambda 250529, total 250529, (0.0% missing)\n\nCPU times: user 592 ms, sys: 624 ms, total: 1.22 s\nWall time: 10min 18s\n"}, {"output_type": "stream", "name": "stderr", "text": "ERROR:boto:400 Bad Request\nERROR:boto:<?xml version=\"1.0\"?>\n<Response><Errors><Error><Code>MissingParameter</Code><Message>No items specified</Message><BoxUsage>0.0000000000</BoxUsage></Error></Errors><RequestID>1dc32598-4a5d-86c4-5aa7-cfe5be06dac3</RequestID></Response>\n"}], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 4, "cell_type": "code", "source": "%time filter_service.main(from_date=one_week_ago, to_date=yesterday, dataset=\"telemetry-release\")", "outputs": [{"output_type": "stream", "name": "stdout", "text": "Bucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151002\nLooked at 0 total records in 1.300597 seconds, added 0\nLooked at 100000 total records in 338.020727 seconds, added 100000\nLooked at 200000 total records in 683.664686 seconds, added 200000\nLooked at 300000 total records in 1024.683393 seconds, added 300000\nLooked at 400000 total records in 1360.000518 seconds, added 400000\nLooked at 500000 total records in 1690.792948 seconds, added 500000\nLooked at 600000 total records in 2038.29741 seconds, added 600000\nLooked at 700000 total records in 2357.026726 seconds, added 700000\nLooked at 800000 total records in 2684.049019 seconds, added 800000\nLooked at 900000 total records in 2996.521813 seconds, added 900000\nLooked at 1000000 total records in 3315.840354 seconds, added 1000000\nLooked at 1100000 total records in 3628.943946 seconds, added 1100000\nLooked at 1200000 total records in 3969.149524 seconds, added 1200000\nLooked at 1300000 total records in 4305.763672 seconds, added 1300000\nLooked at 1400000 total records in 4615.893811 seconds, added 1400000\nLooked at 1500000 total records in 4966.937563 seconds, added 1500000\nLooked at 1600000 total records in 5328.776605 seconds, added 1600000\nLooked at 1700000 total records in 5705.570749 seconds, added 1700000\nLooked at 1800000 total records in 6144.749973 seconds, added 1800000\nLooked at 1900000 total records in 6538.010904 seconds, added 1900000\nLooked at 2000000 total records in 6864.744661 seconds, added 2000000\nLooked at 2100000 total records in 7200.882931 seconds, added 2100000\nLooked at 2200000 total records in 7533.957755 seconds, added 2200000\nLooked at 2300000 total records in 7859.032334 seconds, added 2300000\nLooked at 2400000 total records in 8196.48085 seconds, added 2400000\nLooked at 2500000 total records in 8476.041597 seconds, added 2500000\nLooked at 2600000 total records in 8771.934378 seconds, added 2600000\nLooked at 2700000 total records in 9092.9351 seconds, added 2700000\nLooked at 2800000 total records in 9400.329091 seconds, added 2800000\nLooked at 2900000 total records in 9680.063249 seconds, added 2900000\nOverall, added 2939983 of 2939983 in 9827.859046 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151002 - previous: 0, current: 2595583, added: 2595583 (100.0% missing)\nAWS lambda stats:\nDay 20151002 - lambda 0, total 2595583, (100.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151003\nLooked at 0 total records in 1.259282 seconds, added 0\nLooked at 100000 total records in 329.404242 seconds, added 100000\nLooked at 200000 total records in 644.075432 seconds, added 200000\nLooked at 300000 total records in 1000.318887 seconds, added 300000\nLooked at 400000 total records in 1338.615297 seconds, added 400000\nLooked at 500000 total records in 1662.73126 seconds, added 500000\nLooked at 600000 total records in 2003.830746 seconds, added 600000\nLooked at 700000 total records in 2327.566902 seconds, added 700000\nLooked at 800000 total records in 2652.0854 seconds, added 800000\nLooked at 900000 total records in 2967.820198 seconds, added 900000\nLooked at 1000000 total records in 3273.768904 seconds, added 1000000\nLooked at 1100000 total records in 3606.760202 seconds, added 1100000\nLooked at 1200000 total records in 3942.600231 seconds, added 1200000\nLooked at 1300000 total records in 4283.888702 seconds, added 1300000\nLooked at 1400000 total records in 4603.952839 seconds, added 1400000\nLooked at 1500000 total records in 4951.49371 seconds, added 1500000\nLooked at 1600000 total records in 5311.504328 seconds, added 1600000\nLooked at 1700000 total records in 5683.014011 seconds, added 1700000\nLooked at 1800000 total records in 6119.195973 seconds, added 1800000\nLooked at 1900000 total records in 6522.009457 seconds, added 1900000\nLooked at 2000000 total records in 6841.106261 seconds, added 2000000\nLooked at 2100000 total records in 7179.742611 seconds, added 2100000\nLooked at 2200000 total records in 7519.417606 seconds, added 2200000\nLooked at 2300000 total records in 7842.09352 seconds, added 2300000\nLooked at 2400000 total records in 8176.472838 seconds, added 2400000\nLooked at 2500000 total records in 8455.525023 seconds, added 2500000\nLooked at 2600000 total records in 8747.396086 seconds, added 2600000\nLooked at 2700000 total records in 9073.122679 seconds, added 2700000\nLooked at 2800000 total records in 9371.277431 seconds, added 2800000\nLooked at 2900000 total records in 9655.676949 seconds, added 2900000\nLooked at 3000000 total records in 9859.314755 seconds, added 3000000\nOverall, added 3005594 of 3005594 in 9872.655113 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151003 - previous: 0, current: 2665169, added: 2665169 (100.0% missing)\nAWS lambda stats:\nDay 20151003 - lambda 0, total 2665169, (100.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151004\nLooked at 0 total records in 1.265391 seconds, added 0\nLooked at 100000 total records in 336.885577 seconds, added 100000\nLooked at 200000 total records in 673.593766 seconds, added 200000\nLooked at 300000 total records in 1016.623252 seconds, added 300000\nLooked at 400000 total records in 1348.644685 seconds, added 400000\nLooked at 500000 total records in 1674.68894 seconds, added 500000\nLooked at 600000 total records in 2023.055601 seconds, added 600000\nLooked at 700000 total records in 2342.335556 seconds, added 700000\nLooked at 800000 total records in 2667.38035 seconds, added 800000\nLooked at 900000 total records in 2978.092655 seconds, added 900000\nLooked at 1000000 total records in 3291.209696 seconds, added 1000000\nLooked at 1100000 total records in 3604.759897 seconds, added 1100000\nLooked at 1200000 total records in 3928.016131 seconds, added 1200000\nLooked at 1300000 total records in 4277.425832 seconds, added 1300000\nLooked at 1400000 total records in 4591.510118 seconds, added 1400000\nLooked at 1500000 total records in 4929.029632 seconds, added 1500000\nLooked at 1600000 total records in 5294.858215 seconds, added 1600000\nLooked at 1700000 total records in 5666.603706 seconds, added 1700000\nLooked at 1800000 total records in 6100.580878 seconds, added 1800000\nLooked at 1900000 total records in 6507.588731 seconds, added 1900000\nLooked at 2000000 total records in 6824.350515 seconds, added 2000000\nLooked at 2100000 total records in 7156.653025 seconds, added 2100000\nLooked at 2200000 total records in 7485.036377 seconds, added 2200000\nLooked at 2300000 total records in 7802.492827 seconds, added 2300000\nLooked at 2400000 total records in 8126.234891 seconds, added 2400000\nLooked at 2500000 total records in 8419.463069 seconds, added 2500000\nLooked at 2600000 total records in 8701.243964 seconds, added 2600000\nLooked at 2700000 total records in 9028.464815 seconds, added 2700000\nLooked at 2800000 total records in 9335.569786 seconds, added 2800000\nLooked at 2900000 total records in 9624.385967 seconds, added 2900000\nOverall, added 2942456 of 2942456 in 9794.617557 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151004 - previous: 0, current: 2599306, added: 2599306 (100.0% missing)\nAWS lambda stats:\nDay 20151004 - lambda 0, total 2599306, (100.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151005\nLooked at 0 total records in 1.204175 seconds, added 0\nLooked at 100000 total records in 332.832938 seconds, added 100000\nLooked at 200000 total records in 673.689902 seconds, added 200000\nLooked at 300000 total records in 1021.411837 seconds, added 300000\nLooked at 400000 total records in 1357.872294 seconds, added 400000\nLooked at 500000 total records in 1691.212788 seconds, added 500000\nLooked at 600000 total records in 2033.021145 seconds, added 600000\nLooked at 700000 total records in 2343.593225 seconds, added 700000\nLooked at 800000 total records in 2662.604683 seconds, added 800000\nLooked at 900000 total records in 2971.707666 seconds, added 900000\nLooked at 1000000 total records in 3291.382357 seconds, added 1000000\nLooked at 1100000 total records in 3616.480086 seconds, added 1100000\nLooked at 1200000 total records in 3968.787431 seconds, added 1200000\nLooked at 1300000 total records in 4312.237262 seconds, added 1300000\nLooked at 1400000 total records in 4624.221639 seconds, added 1400000\nLooked at 1500000 total records in 4981.176181 seconds, added 1500000\nLooked at 1600000 total records in 5348.030149 seconds, added 1600000\nLooked at 1700000 total records in 5723.190388 seconds, added 1700000\nLooked at 1800000 total records in 6174.073967 seconds, added 1800000\nLooked at 1900000 total records in 6551.693972 seconds, added 1900000\nLooked at 2000000 total records in 6871.711381 seconds, added 2000000\nLooked at 2100000 total records in 7207.996115 seconds, added 2100000\nLooked at 2200000 total records in 7541.992743 seconds, added 2200000\nLooked at 2300000 total records in 7861.804016 seconds, added 2300000\nLooked at 2400000 total records in 8192.694472 seconds, added 2400000\nLooked at 2500000 total records in 8471.247433 seconds, added 2500000\nLooked at 2600000 total records in 8761.36198 seconds, added 2600000\nLooked at 2700000 total records in 9085.409573 seconds, added 2700000\nLooked at 2800000 total records in 9391.606059 seconds, added 2800000\nLooked at 2900000 total records in 9670.819268 seconds, added 2900000\nLooked at 3000000 total records in 9870.149381 seconds, added 3000000\nLooked at 3100000 total records in 10052.603538 seconds, added 3100000\nLooked at 3200000 total records in 10227.973829 seconds, added 3200000\nOverall, added 3208853 of 3208853 in 10243.838648 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151005 - previous: 0, current: 2866353, added: 2866353 (100.0% missing)\nAWS lambda stats:\nDay 20151005 - lambda 0, total 2866353, (100.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151006\nLooked at 0 total records in 1.13371 seconds, added 0\nLooked at 100000 total records in 332.82465 seconds, added 100000\nLooked at 200000 total records in 665.876071 seconds, added 200000\nLooked at 300000 total records in 1023.594126 seconds, added 300000\nLooked at 400000 total records in 1358.22378 seconds, added 400000\nLooked at 500000 total records in 1689.382981 seconds, added 500000\nLooked at 600000 total records in 2028.496352 seconds, added 600000\nLooked at 700000 total records in 2347.519901 seconds, added 700000\nLooked at 800000 total records in 2671.948398 seconds, added 800000\nLooked at 900000 total records in 2988.255824 seconds, added 900000\nLooked at 1000000 total records in 3311.192708 seconds, added 1000000\nLooked at 1100000 total records in 3624.991713 seconds, added 1100000\nLooked at 1200000 total records in 3965.614504 seconds, added 1200000\nLooked at 1300000 total records in 4301.64517 seconds, added 1300000\nLooked at 1400000 total records in 4610.75381 seconds, added 1400000\nLooked at 1500000 total records in 4960.908401 seconds, added 1500000\nLooked at 1600000 total records in 5319.594101 seconds, added 1600000\nLooked at 1700000 total records in 5689.356829 seconds, added 1700000\nLooked at 1800000 total records in 6125.764854 seconds, added 1800000\nLooked at 1900000 total records in 6519.842979 seconds, added 1900000\nLooked at 2000000 total records in 6842.506786 seconds, added 2000000\nLooked at 2100000 total records in 7183.321501 seconds, added 2100000\nLooked at 2200000 total records in 7518.618167 seconds, added 2200000\nLooked at 2300000 total records in 7837.944043 seconds, added 2300000\nLooked at 2400000 total records in 8175.84103 seconds, added 2400000\nLooked at 2500000 total records in 8458.707217 seconds, added 2500000\nLooked at 2600000 total records in 8762.315547 seconds, added 2600000\nLooked at 2700000 total records in 9089.727033 seconds, added 2700000\nLooked at 2800000 total records in 9391.962668 seconds, added 2800000\nLooked at 2900000 total records in 9669.496105 seconds, added 2900000\nLooked at 3000000 total records in 9869.303938 seconds, added 3000000\nLooked at 3100000 total records in 10053.666761 seconds, added 3100000\nOverall, added 3140775 of 3140775 in 10160.870404 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151006 - previous: 0, current: 2795325, added: 2795325 (100.0% missing)\nAWS lambda stats:\nDay 20151006 - lambda 0, total 2795325, (100.0% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151007\nLooked at 0 total records in 1.202166 seconds, added 0\nLooked at 100000 total records in 332.083748 seconds, added 100000\nLooked at 200000 total records in 665.207925 seconds, added 200000\nLooked at 300000 total records in 1005.930262 seconds, added 300000\nLooked at 400000 total records in 1344.936647 seconds, added 400000\nLooked at 500000 total records in 1673.039158 seconds, added 500000\nLooked at 600000 total records in 2003.342875 seconds, added 600000\nLooked at 700000 total records in 2314.779781 seconds, added 700000\nLooked at 800000 total records in 2629.562658 seconds, added 800000\nLooked at 900000 total records in 2938.141879 seconds, added 900000\nLooked at 1000000 total records in 3245.069185 seconds, added 1000000\nLooked at 1100000 total records in 3567.688973 seconds, added 1100000\nLooked at 1200000 total records in 3876.90304 seconds, added 1200000\nLooked at 1300000 total records in 4227.74919 seconds, added 1300000\nLooked at 1400000 total records in 4534.259582 seconds, added 1400000\nLooked at 1500000 total records in 4857.533458 seconds, added 1500000\nLooked at 1600000 total records in 5220.758194 seconds, added 1600000\nLooked at 1700000 total records in 5577.957619 seconds, added 1700000\nLooked at 1800000 total records in 5973.655685 seconds, added 1800000\nLooked at 1900000 total records in 6415.107432 seconds, added 1900000\nLooked at 2000000 total records in 6749.215588 seconds, added 2000000\nLooked at 2100000 total records in 7083.848189 seconds, added 2100000\nLooked at 2200000 total records in 7402.05582 seconds, added 2200000\nLooked at 2300000 total records in 7719.226057 seconds, added 2300000\nLooked at 2400000 total records in 8047.695759 seconds, added 2400000\nLooked at 2500000 total records in 8357.231001 seconds, added 2500000\nLooked at 2600000 total records in 8642.123475 seconds, added 2600000\nLooked at 2700000 total records in 8943.036599 seconds, added 2700000\nLooked at 2800000 total records in 9265.483551 seconds, added 2800000\nLooked at 2900000 total records in 9545.284923 seconds, added 2900000\nOverall, added 2906883 of 2906883 in 9679.940665 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151007 - previous: 976685, current: 2679339, added: 1702654 (63.5475391505% missing)\nAWS lambda stats:\nDay 20151007 - lambda 976685, total 2679339, (63.5475391505% missing)\n\nBucket: net-mozaws-prod-us-west-2-pipeline-data - Prefix: telemetry-release - Date: 20151008\nLooked at 0 total records in 1.357596 seconds, added 0\nLooked at 100000 total records in 300.850614 seconds, added 100000\nLooked at 200000 total records in 581.019943 seconds, added 200000\nLooked at 300000 total records in 893.708983 seconds, added 300000\nLooked at 400000 total records in 1111.098049 seconds, added 400000\nLooked at 500000 total records in 1368.317758 seconds, added 500000\nLooked at 600000 total records in 1693.139872 seconds, added 600000\nLooked at 700000 total records in 1950.678091 seconds, added 700000\nLooked at 800000 total records in 2187.478365 seconds, added 800000\nLooked at 900000 total records in 2415.518115 seconds, added 900000\nLooked at 1000000 total records in 2662.039214 seconds, added 1000000\nLooked at 1100000 total records in 2924.937298 seconds, added 1100000\nLooked at 1200000 total records in 3189.080593 seconds, added 1200000\nLooked at 1300000 total records in 3436.19422 seconds, added 1300000\nLooked at 1400000 total records in 3667.629476 seconds, added 1400000\nLooked at 1500000 total records in 3913.417222 seconds, added 1500000\nLooked at 1600000 total records in 4161.668553 seconds, added 1600000\nLooked at 1700000 total records in 4415.736997 seconds, added 1700000\nLooked at 1800000 total records in 4658.16581 seconds, added 1800000\nLooked at 1900000 total records in 4898.603828 seconds, added 1900000\nLooked at 2000000 total records in 5139.631315 seconds, added 2000000\nLooked at 2100000 total records in 5396.251683 seconds, added 2100000\nLooked at 2200000 total records in 5671.016955 seconds, added 2200000\nLooked at 2300000 total records in 5897.528303 seconds, added 2300000\nLooked at 2400000 total records in 6152.99746 seconds, added 2400000\nLooked at 2500000 total records in 6449.33931 seconds, added 2500000\nLooked at 2600000 total records in 6765.608871 seconds, added 2600000\nLooked at 2700000 total records in 7028.739251 seconds, added 2700000\nLooked at 2800000 total records in 7308.822868 seconds, added 2800000\nLooked at 2900000 total records in 7541.360479 seconds, added 2900000\nLooked at 3000000 total records in 7808.168993 seconds, added 3000000\nLooked at 3100000 total records in 8071.117797 seconds, added 3100000\nOverall, added 3142721 of 3142721 in 8230.555173 seconds\nFilter service stats:\nNote that the following numbers are correct only if there ins't another entity concurrently pushing new submissions:\nDay 20151008 - previous: 3142715, current: 3142721, added: 6 (0.000190917361102% missing)\nAWS lambda stats:\nDay 20151008 - lambda 3142715, total 3142721, (0.000190917361102% missing)\n\nCPU times: user 9.71 s, sys: 7.05 s, total: 16.8 s\nWall time: 2h 52min 39s\n"}, {"output_type": "stream", "name": "stderr", "text": "ERROR:boto:400 Bad Request\nERROR:boto:<?xml version=\"1.0\"?>\n<Response><Errors><Error><Code>MissingParameter</Code><Message>No items specified</Message><BoxUsage>0.0000000000</BoxUsage></Error></Errors><RequestID>1ff93cb4-4800-33bc-ba1a-ed4c7cae08f7</RequestID></Response>\n"}], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.9", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}