Estimated time calculator for Getty Re indexing

estimated-time-for-getty-reindex.py

# Put this file on scripts folder and run using following command
# sudo DJANGO_SETTINGS_MODULE=theplatform.settings  /root/python-env/bin/python estimated-time-for-getty-reindex.py  /home/ubuntu/source-72-combine-getty-assetgroups.checkpoint

import sys
import datetime


def print_progress(filename):
    with open(filename, 'r') as f:
        content = f.read()
        (dur, checkpoint_info) = eval(content)
        (_, _, checkpoint_id, num_err, num_suc) = checkpoint_info

    from theplatform.newscred.models import Source, Image
    GETTY_SOURCE = Source.objects.get(name='Getty Images')
    remn = Image.objects.filter(source=GETTY_SOURCE, id__lt=checkpoint_id).count()
    rdur = dur*remn/(num_err+num_suc)
    print "Estimated remaining time to complete: %s" % rdur


def main():
    print_progress(sys.argv[1])


if __name__ == '__main__':
    main()

Today I executed it.

print_progress(64888937L, datetime.timedelta(0, 12641, 354171))
Estimated remaining time to complete: 90 days, 11:46:23.744982

Checkpoint file content

[datetime.timedelta(0, 12641, 354171), ('theplatform.newscred.models.image', 'Image', 64888937L, 10, 47303)]

After 10 hours.

Estimated remaining time to complete: 37 days, 1:40:26.497849

Estimated remaining time to complete: 35 days, 22:16:02.392597

Estimated remaining time to complete: 27 days, 4:53:26.189401

Estimated remaining time to complete: 25 days, 21:56:56.987482

Estimated remaining time to complete: 24 days, 16:43:26.354451

Estimated remaining time to complete: 22 days, 8:20:59.484419

Checkpoint file content

[datetime.timedelta(10, 39264, 842892), ('theplatform.newscred.models.image', 'Image', 45468250L, 10, 10930461)]

[datetime.timedelta(11, 36965, 643737), ('theplatform.newscred.models.image', 'Image', 42212783L, 10, 13119968)]


Estimated remaining time to complete: 18 days, 10:41:50.924602

Wed Nov  5 08:47:52 UTC 2014
[datetime.timedelta(11, 51730, 817895), ('theplatform.newscred.models.image', 'Image', 41916685L, 10, 13319818)]
Estimated remaining time to complete: 18 days, 6:21:16.829798

Thu Nov  6 06:56:45 UTC 2014
[datetime.timedelta(12, 45005, 701027), ('theplatform.newscred.models.image', 'Image', 39274555L, 10, 15260202)]
Estimated remaining time to complete: 15 days, 14:49:36.770005

Sun Nov  9 11:18:37 UTC 2014
[datetime.timedelta(15, 59694, 894247), ('theplatform.newscred.models.image', 'Image', 21917683L, 10, 22911520)]
Estimated remaining time to complete: 7 days, 19:06:00.577613

Wed Nov 12 07:42:19 UTC 2014
[datetime.timedelta(18, 47659, 520068), ('theplatform.newscred.models.image', 'Image', 13151317L, 10, 30149640)]
Estimated remaining time to complete: 2 days, 13:12:52.935378

Thu Nov 13 03:47:50 UTC 2014
[datetime.timedelta(19, 33546, 400610), ('theplatform.newscred.models.image', 'Image', 9952649L, 10, 32288263)]
Estimated remaining time to complete: 1 day, 4:55:02.921159

Thu Nov 13 05:48:24 UTC 2014
[datetime.timedelta(19, 40793, 689414), ('theplatform.newscred.models.image', 'Image', 9709388L, 10, 32507338)]
Estimated remaining time to complete: 1 day, 1:41:50.633443

The back pop is completed. Content of the checkpoint after completion is

[datetime.timedelta(20, 7796, 861578), ('theplatform.newscred.models.image', 'Image', 4063647L, 10, 34294634)]