| # Initial setups. | |
| cp /home/yadunand/swiftdemo.v04.tgz ./ | |
| tar -xvzf swiftdemo.v04.tgz | |
| cd swiftdemo | |
| source ./setup.sh | |
| TEST_HOME=$PWD | |
| # Tests | |
| cd test.local |
| Test-Results | |
| ========================================================================== | |
| Test home : /scratch/midway/yadunand/swift-0.94RC2/run-2013-08-30/remote_driver-083939/out | |
| Sites successful : 6 | |
| ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | |
| Results for site : run-2013-08-30-uc3-sub.uchicago.edu | |
| GROUPS |RUN |PASS |FAIL |TIME | |
| G1: Language-Working Tests |98 |98 |0 | 199 seconds | |
| G2: Local Tests |7 |7 |0 | 16 seconds | |
| G3: Should-Not-Work Tests |10 |10 |0 | 5 seconds |
| import files; | |
| import io; | |
| import random; | |
| import string; | |
| import sys; | |
| import location; | |
| import assert; | |
| app (file out, file err) gen_data (file run, int recsize){ | |
| "/bin/bash" run recsize @stdout=out @stderr=err |
| Results from running a simulation with $5000 USD with : | |
| Starting date : 2012-12-31 16:00:00 | |
| Ending date : 2013-12-16 05:00:00 | |
| Simulation on 1 hour candles from MtGox. | |
| And basic buy threshold set to 1% of price, and sell threshold set to 3% | |
| Format : EMA1, EMA2, EMA3, Final sum in USD |
| I ran a parameter sweep backtest with 2 EMA crosses, with EMAs in range 0,70 on | |
| $5000 USD on MtGox. | |
| Starting date : 2012-12-31 16:00:00 | |
| Ending date : 2013-12-16 05:00:00 | |
| Simulation on 1 hour candles from MtGox. | |
| Here's the results from running simple 2 EMA crosses, sorted by final USD balance | |
| EMA1, EMA2, FINAL_USD | |
| 9, 29, 840301.79202999303 |
#Hadoop
#SwiftReduce
| #!/bin/bash | |
| get_last_swift() | |
| { | |
| ps -u $USER | grep swift | head -n 1 | awk -F ' ' '{print $1}' | |
| } | |
| timestamp() | |
| { |
| Number of files, File size, no_strings.swift-Used_Heap, no_strings.swift-Crt_Heap, strings.swift-Used_Heap, strings.swift-Crt_Heap | |
| 1, 1, 93Mb, 362M, 95Mb, 362Mb | |
| 10, 1, 95Mb, 362M, 175Mb, 457Mb | |
| 100, 1, 95Mb, 362M, ---Mb, 362Mb | |
| Experiment 2: Lazy ranges. | |
| Assumptions: An int in swift, is 8 Bytes (64 bit). Array is a hashtable with unknown memory pattern. |
The purpose of this guide is to help you setup swift on a hadoop cluster. This would let you execute swiftscripts on the cluster allowing you to go beyond the limitations of MapReduce style workflows.
In current configurations Swift runs in persistent coasters mode on Hadoop. What this means is that, there are two separate Swift entities required to execute a swift run on a hadoop cluster, the Swift-coaster-service which deals with managing workers on the Hadoop cluster and the swift runtime which executes swiftscripts via the Swift-coaster-service.
Coasters is the resource manager for swift. By default, swift uses automatic-coasters, which provisions compute resources for the duration of a swift run based on the directives in the sites file. Persistent coasters, on the other hand allows you to retain an active service separate from the swift runtime across several swift runs. This can be used to hold on to resources in a queue based system, string together a collection of nodes for use as a quick ad-hoc cluster, or help work-around networking issues.