ggreg

First Article (Addressing the problem)

Rationale

Here at Botify

• batch processing of TeraBytes of web server logs involves storing temporary data
• bulk write/read GigaBytes data loads that would not fit in server's main memory (we preferred to improve performance on a single host at the beginning)
• need for persistence

ggreg

@task                                                                                             
def fetch(pattern, src):                                                                                  
    fetcher = LogFetcher(pattern)                                                                         
    return fetcher.register(src, fetcher.fetch(src)).id                                                   
                                                                                                          
                                                                                                          
@task                                                                                             
def convert_tsv_blocks(tsv_id_list):                                                                                                                    
    return group([aggregate.s(tsv_id) for tsv_id in                             
                  tsv_id_list])()