Created
July 3, 2015 10:41
-
-
Save breznak/fb12394015fb4108a069 to your computer and use it in GitHub Desktop.
NuPIC OPF swarming - is really best model chosen?
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ####>> UPDATED JOB RESULTS: | |
| { u'bestModel': 104195, | |
| u'bestValue': 4.43, | |
| u'metrics': { u"multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg": 4.43, | |
| u"multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.46635898904319556, | |
| u"prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg": 3.752, | |
| u"prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.39498395640859363}, | |
| u'saved': True} (elapsed time: 31110.6 secs) | |
| ##>> UPDATED STATUS: | |
| Completed generation #2 of swarm 'modelParams|sensorParams|encoders|ecg' with a best errScore of 4.601. Matured swarm(s): [u'modelParams|sensorParams|encoders|ecg'] | |
| ####>> UPDATED JOB RESULTS: | |
| { u'absoluteFieldContributions': { u'ecg': 0.0}, | |
| u'bestModel': 104195, | |
| u'bestValue': 4.43, | |
| u'fieldContributions': { u'ecg': 0.0}, | |
| u'metrics': { u"multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg": 4.43, | |
| u"multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.46635898904319556, | |
| u"prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg": 3.752, | |
| u"prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.39498395640859363}, | |
| u'saved': True} (elapsed time: 31111.6 secs) | |
| Evaluated 20 models | |
| HyperSearch finished! | |
| Worker completion message: Exiting because we've evaluated all possible field combinations | |
| Results from all experiments: | |
| ---------------------------------------------------------------- | |
| [0] Experiment _NupicModelInfo(jobID=2460, modelID=104149, status=completed, completionReason=eof, updateCounter=44, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.482 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.47183318033670485 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [1] Experiment _NupicModelInfo(jobID=2460, modelID=104153, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.02.modelParams|tpParams|minThreshold_9.modelParams|tpParams|activationThreshold_12.modelParams|tpParams|pamLength_10.modelParams|sensorParams|encoders|ecg:w_51): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 6.261 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.659113686320417 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [2] Experiment _NupicModelInfo(jobID=2460, modelID=104171, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_12.modelParams|tpParams|activationThreshold_16.modelParams|tpParams|pamLength_50.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.599 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4841501107471008 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [3] Experiment _NupicModelInfo(jobID=2460, modelID=104182, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.5.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_13.modelParams|tpParams|pamLength_100.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.623 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4866766605748743 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [4] Experiment _NupicModelInfo(jobID=2460, modelID=104183, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_360.modelParams|sensorParams|encoders|ecg:w_51): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.592 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4834132003806668 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [5] Experiment _NupicModelInfo(jobID=2460, modelID=104184, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.599 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4841501107471008 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [6] Experiment _NupicModelInfo(jobID=2460, modelID=104185, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.655 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4900453936785723 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [7] Experiment _NupicModelInfo(jobID=2460, modelID=104186, status=completed, completionReason=eof, updateCounter=27, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.447 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.46814862850453515 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [8] Experiment _NupicModelInfo(jobID=2460, modelID=104187, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_13.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.454 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4688855388709691 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [9] Experiment _NupicModelInfo(jobID=2460, modelID=104188, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.604 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4846764752945536 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [10] Experiment _NupicModelInfo(jobID=2460, modelID=104189, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.461 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.46962244923740304 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [11] Experiment _NupicModelInfo(jobID=2460, modelID=104190, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_16.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.601 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4843606565660819 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [12] Experiment _NupicModelInfo(jobID=2460, modelID=104191, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.621 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4864661147558932 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [13] Experiment _NupicModelInfo(jobID=2460, modelID=104192, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 5.141 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.5412080276909861 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [14] Experiment _NupicModelInfo(jobID=2460, modelID=104193, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 5.139 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.540997481872005 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [15] Experiment _NupicModelInfo(jobID=2460, modelID=104194, status=completed, completionReason=eof, updateCounter=25, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 5.13 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.5400500256865899 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [16] Experiment _NupicModelInfo(jobID=2460, modelID=104195, status=completed, completionReason=eof, updateCounter=27, numRecords=2000) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.43 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.46635898904319556 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 3.752 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.39498395640859363 | |
| [17] Experiment _NupicModelInfo(jobID=2460, modelID=104196, status=completed, completionReason=killed, updateCounter=14, numRecords=1050) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_12.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 8.725 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.9135867362912252 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 4.422 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.46302355849625193 | |
| [18] Experiment _NupicModelInfo(jobID=2460, modelID=104197, status=completed, completionReason=killed, updateCounter=12, numRecords=800) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.5.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 8.774718397997496 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.9194139146103015 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 4.264080100125156 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.4467897884862783 | |
| [19] Experiment _NupicModelInfo(jobID=2460, modelID=104198, status=completed, completionReason=killed, updateCounter=5, numRecords=150) | |
| (modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_12.modelParams|tpParams|activationThreshold_16.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_91): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 2.872483221476511 (*) | |
| multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.3045092989171423 | |
| prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg: 2.1476510067114094 | |
| prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg: 0.22767050386328386 | |
| -------------------------------------------------------------- | |
| 20 experiments total (all completed successfully). | |
| WaitingToStart: 0 | |
| Running: 0 | |
| Completed: 20 | |
| ran to EOF: 17 | |
| ran to stop signal: 0 | |
| were orphaned: 0 | |
| killed off: 3 | |
| failed: 0 | |
| Field Contributions: | |
| { u'ecg': 0.0} | |
| Best results on the optimization metric multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg (maximize=False): | |
| [16] Experiment _NupicModelInfo(jobID=2460, modelID=104195, status=completed, completionReason=eof, updateCounter=27, numRecords=2000) (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31): | |
| multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg: 4.43 | |
| Total number of Records processed: 36000 | |
| Total wall time for all models: 124434 | |
| Generating description files for top 1 models... | |
| Generating description file for model 104195 at /home/mmm/devel/ecg/nupic.biodat/ECG/MIT-BIH-Arrhythmia/model/swarm/model_0 | |
| Generating model params file... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@rhyolight why was this model killed? https://gist.github.com/breznak/fb12394015fb4108a069#file-swarming_models-L160
It had best so far performance!