RegressionNotes.txt

gistfile1.txt

Two R Packages

   TiosQ => Core Quant Systems (Lion, Forecasts, etc)
   tiosmodelr => Shadow Regressions
   
   
30 jobs queued up in Downloads.  They take 10-60 seconds each

State:
  25 jobs in queue Downloads
   5 workers working on one job each from Downloads
   
Someone queues up 2 jobs in Downloadshighfreq
   2 of the earlier workers finish their 1 job
   
State:
   3 workers working on their prior Downloads job
   2 workers pick up the Downloadsighfrequency
   25 still in Downloads         
   
Model Group Cloning
   1 Job for each model group clone effort
      -> these jobs are put into "miso_model_group_duplicate"  
       -> These are picked up by energyrunner-resqmodelgroup-* servers
          which then use the MDAs, gather all the data, prepare 
          R dataframes, and then spawn off new quant jobs
             -> Spawns 1 QuantJob to do the main regression
                 It tells the QuantJob where the data is, on S3
                 Job gets queued up into "miso_regression_primary"
                  ~> QuantJob for the regression runs on a QuantJob server...
                     Does the regression analysis
                     Notifies that it's done with the QuantJob by putting a job into
                       regression_updates (might be 
                         miso_regression_updates,miso_regression_updates_high) 
                  ~> One of the EnergyCore servers pick up the completion update
                     It spawns multiple QuantJobs to do secondary analysis (scatter, CV, error)
                       Those jobs get queued up into "miso_regression_secondary" 
                       
                       ~> Quantjob for the secondary analysis runs on a QuantJob server...
                          Does the analysis, and then notifies updates          
                           regression_updates (might be 
                             miso_regression_updates,miso_regression_updates_high)
                
                
Model Group Rerun
   1 Job for each rerun request
      -> these jobs are put into "miso_model_group_rerun"      
       -> These are picked up by energyrunner-resqmodelgroup-* servers