individual_vars.txt

4 workers, 15 cores each, another shell with 40 cores running at the same time.
2tags, 100k securities, 2.5k pvs

I: input - the size of data input to all the tasks
O: ouput - the size of data output from all tasks
SR: shuffle read - the size of data read from other partitions
SW: shuffle write - the size of data written to other partitions


baseline: read from data table and write back to another table
    1s	20/20 I: 1932.5KB	O: 1932.5KB
    spark smart enough to push down the select into cassandra, so not reading all the pvs


basic VaR
	3 s	20/20 I: 1909.2MB	SW: 2.3MB


conditional VaR
    3s	20/20 I: 1909.2MB	SW: 2.3MB	


marginal VaR
    read, rollup and collect:
        5s 	T: 2/2 	S: 80/80 :
            4s	    20/20  I: 1909.2 MB     SW: 48.1 MB
            0.4s	60/60  SR: 48.1 MB 
    marginal:
        5s 	T: 2/2 	S: 80/80 :
            2s	T: 20/20    I: 1909.2 MB	SW: 2018.8 MB
            3s  T: 60/60    O: 2.3 MB       SR: 2018.8 MB

incremental VaR
    read, rollup and collect:
        5s 	T: 2/2 	S: 80/80 :
            4s	    20/20  I: 1909.2 MB     SW: 48.1 MB
            0.4s	60/60  SR: 48.1 MB 
    incremental:
        5s 	T: 2/2 	S: 80/80 :
            2s	20/20    I: 1909.2 MB	SW: 2018.8 MB
            2s  60/60    O: 2.3 MB	    SR: 2018.3 MB

all VaR:
    read, rollup and collect:
        5 s 	2/2 	80/80 
    VaR calcs:
        8 s 	6/6 	240/240 
        1 - persist:
            4s  20/20   I: 2.8GB    SW: 2.0GB
        2 - save (bvar):
            2s  60/60   SR: 2.0GB   SW: 1549.7KB
        3 - save (cvar):
            2s  60/60   I: 2.8GB    SW: 1105.2KB
        4 - save (ivar):
            2s  60/60   I: 2.8GB    SW: 1874.8KB
        5 - save (mvar):
            2s  60/60   I: 2.8GB    SW: 1872.0KB
        6 - runJob (join and save):
            1s  60/60   O: 3.4MB    SR: 6.3MB