Skip to content

Instantly share code, notes, and snippets.

@yhilpisch

yhilpisch/00_epat_aug_2018.md Secret

Last active December 18, 2018 14:05
Show Gist options
  • Save yhilpisch/7dc47060eb51954d7aa2a576e6380d5f to your computer and use it in GitHub Desktop.
Save yhilpisch/7dc47060eb51954d7aa2a576e6380d5f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
00_epat_aug_2018.md

Executive Program in Algorithmic Trading (QuantInsti)

Python Sessions by Dr. Yves J. Hilpisch | The Python Quants GmbH

Online, 25. & 26. August 2018

Short Link

http://bit.ly/epat_aug_2018

Resources

Slides & Materials

You find the introduction slides under http://hilpisch.com/epat.pdf

You find the materials about OOP under http://hilpisch.com/py4fi_oop_epat.html

Python

If you have either Miniconda or Anaconda already installed, there is no need to install anything new.

The code that follows uses Python 3.6. For example, download and install Miniconda 3.6 from https://conda.io/miniconda.html if you do not have conda already installed.

In any case, for Linux/Mac you should execute the following lines on the shell to create a new environment with the needed packages:

conda create -n epat python=3.6
source activate epat
conda install numpy pandas matplotlib statsmodels
pip install plotly==2.4.0 cufflinks
conda install ipython jupyter
jupyter notebook

On Windows, execute the following lines on the command prompt:

conda create -n epat python=3.6
activate epat
conda install numpy pandas matplotlib statsmodels
pip install plotly==2.4.0 cufflinks
pip install win-unicode-console
set PYTHONIOENCODING=UTF-8
conda install ipython jupyter
jupyter notebook

Read more about the management of environments under https://conda.io/docs/using/envs.html

Docker

To install Docker see https://docs.docker.com/install/.

To run a Ubuntu-based Docker container, execute on the shell the following:

docker run -ti -p 9000:9000 -h epat -v /Users/yves/Temp/:/root/tmp ubuntu:latest /bin/bash

Make sure to adjust the folder to be mounted accordingly.

ZeroMQ

The major resource for the ZeroMQ distributed messaging package based on sockets is http://zeromq.org/

Cloud

Use this link to get a 10 USD bonus on DigitalOcean when signing up for a new account.

Books & Resources

An overview of the Python Data Model is found under: Python Data Model

Good book about everything important in Python data analysis: Python Data Science Handbook, O'Reilly

Good book covering object-oriented programming in Python: Fluent Python, O'Reilly

Display the source blob
Display the rendered blob
Raw
01_vecback_oop.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
02_object_orientation.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
03_event_based_backtesting.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
03_event_based_backtesting_live.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"http://hilpisch.com/tpq_logo.png\" width=\"350px\" align=\"right\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# EPAT Session 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Executive Program in Algorithmic Trading**\n",
"\n",
"**_Event-based Backtesting_ &mdash; Live**\n",
"\n",
"Dr. Yves J. Hilpisch | The Python Quants GmbH | http://tpq.io\n",
"\n",
"<img src=\"http://hilpisch.com/images/tpq_bootcamp.png\" width=\"350px\" align=\"left\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Basic Imports"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import math\n",
"import numpy as np\n",
"import pandas as pd\n",
"from pylab import plt\n",
"plt.style.use('seaborn')\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Financial Data Class"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class FinancialData(object):\n",
" def __init__(self, symbol):\n",
" self.symbol = symbol\n",
" self.data_url = 'http://hilpisch.com/tr_eikon_eod_data.csv'\n",
" self.prepare_data()\n",
" \n",
" def prepare_data(self):\n",
" self.raw = pd.read_csv(self.data_url, index_col=0, parse_dates=True)\n",
" self.data = pd.DataFrame(self.raw[self.symbol])\n",
" self.data['Returns'] = np.log(self.data / self.data.shift(1))\n",
" self.data.dropna(inplace=True)\n",
" \n",
" def plot_data(self, cols=None):\n",
" if cols is None:\n",
" cols = [self.symbol]\n",
" self.data[cols].plot(figsize=(10, 6), title=self.symbol)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd = FinancialData('AAPL.O')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd.data.info()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd.data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd.plot_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Event-based View on Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# vectorized data handling = complete data set in a single step\n",
"# fd.data['AAPL.O'].plot(figsize=(10, 6));\n",
"fd.plot_data()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for bar in range(10):\n",
" print(bar)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# event-based view on data = going bar by bar \"through time\"\n",
"for bar in range(10):\n",
" print(bar, fd.data['AAPL.O'].iloc[bar])\n",
" time.sleep(1) # pause for 1 second"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# event-based view on data = going bar by bar \"through time\"\n",
"for bar in range(10):\n",
" print(bar, str(fd.data['AAPL.O'].index[bar])[:10], fd.data['AAPL.O'].iloc[bar])\n",
" time.sleep(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Backtesting Base Class"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are going to implement a **base class** for event-based backtesting with:\n",
"\n",
"* `__init__`\n",
"* `prepare_data` (`FinancialBase`)\n",
"* `plot_data` (`FinancialBase`)\n",
"* `get_date_price`\n",
"* `print_balance`\n",
"* `place_buy_order`\n",
"* `place_sell_order`\n",
"* `close_out`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amount = 5000"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"S0 = fd.data.iloc[0]['AAPL.O']\n",
"S0"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"units = 5000 / S0 # theoretical\n",
"units"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"units = int(5000 // (S0 * 1.01)) # practical (incl. prop. transactions costs)\n",
"units"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"units = int(5000 // S0) # practical (without transactions costs)\n",
"units"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cash = amount - units * S0 # transactions costs?!\n",
"cash"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class BacktestingBase(FinancialData):\n",
" def __init__(self, symbol, amount, verbose=True):\n",
" super(BacktestingBase, self).__init__(symbol)\n",
" self.initial_amount = amount # initial cash\n",
" self.amount = amount # current cash balance\n",
" self.position = 0\n",
" self.units = 0\n",
" self.trades = 0\n",
" self.verbose = verbose\n",
" def get_date_price(self, bar):\n",
" date = str(self.data[self.symbol].index[bar])[:10]\n",
" price = self.data[self.symbol].iloc[bar]\n",
" return date, price\n",
" def print_balance(self, bar):\n",
" date, price = self.get_date_price(bar)\n",
" print('{} | current cash balance is {:.2f}'.format(date, self.amount))\n",
" def print_net_wealth(self, bar):\n",
" date, price = self.get_date_price(bar)\n",
" nw = self.amount + self.units * price\n",
" print('{} | current net wealth is {:.2f}'.format(date, nw))\n",
" def place_buy_order(self, bar, units=None, amount=None):\n",
" date, price = self.get_date_price(bar)\n",
" if units is None:\n",
" units = int(amount // price) # tc?!\n",
" self.amount -= units * price # tc?!\n",
" self.units += units\n",
" self.trades += 1\n",
" if self.verbose:\n",
" print('{} | bought {} units for {:.2f}'.format(date, units, price))\n",
" self.print_balance(bar)\n",
" self.print_net_wealth(bar)\n",
" def place_sell_order(self, bar, units=None, amount=None):\n",
" date, price = self.get_date_price(bar)\n",
" if units is None:\n",
" units = int(amount // price) # tc?!\n",
" self.amount += units * price # tc?! & margin?!\n",
" self.units -= units\n",
" self.trades += 1\n",
" if self.verbose:\n",
" print('{} | sold {} units for {:.2f}'.format(date, units, price))\n",
" self.print_balance(bar)\n",
" self.print_net_wealth(bar)\n",
" def close_out(self, bar):\n",
" # self.position in [-1, 0, 1]\n",
" date, price = self.get_date_price(bar)\n",
" self.amount += (self.units * price) * self.position\n",
" self.units -= self.units\n",
" self.trades += 1\n",
" print('\\n{} | *** CLOSING OUT ***'.format(date))\n",
" print(66 * '=')\n",
" print('{} | number of trades {}'.format(date, self.trades))\n",
" print('{} | initial cash balance {}'.format(date, self.initial_amount))\n",
" self.print_balance(bar)\n",
" perf = (self.amount - self.initial_amount) / self.initial_amount * 100\n",
" print('{} | net performance [%] {:.2f}'.format(date, perf))\n",
" print(66 * '=')\n",
" self.position = 0\n",
" self.trades = 0"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb = BacktestingBase('AAPL.O', 5000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.amount"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.get_date_price(100)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.print_balance(500)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_buy_order(501, units=50)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_buy_order(502, units=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.units"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_buy_order(1500, amount=1500)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.units"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_sell_order(1750, units=34)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.units"
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"bb.place_sell_order(2000, units=bb.units) # close out"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.position = 1\n",
"bb.close_out(2000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## SMA Backtesting Class"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class SMABacktester(BacktestingBase):\n",
" \n",
" def prepare_statistics(self):\n",
" self.data['SMA1'] = self.data[self.symbol].rolling(self.SMA1).mean()\n",
" self.data['SMA2'] = self.data[self.symbol].rolling(self.SMA2).mean()\n",
" \n",
" def run_strategy(self, SMA1, SMA2):\n",
" self.SMA1 = SMA1\n",
" self.SMA2 = SMA2\n",
" self.prepare_statistics()\n",
" \n",
" for bar in range(self.SMA2, len(self.data)):\n",
" date, price = self.get_date_price(bar)\n",
" \n",
" if self.position in [0, -1]:\n",
" if self.data['SMA1'].iloc[bar] > self.data['SMA2'].iloc[bar]:\n",
" if self.verbose:\n",
" print('\\n{} | *** PLACING BUY ORDER ***'.format(date))\n",
" # going long based on units\n",
" # self.place_buy_order(bar, units=(1 - self.position) * 50)\n",
" \n",
" # going long based on amount\n",
" if self.position == -1:\n",
" self.place_buy_order(bar, units=-self.units)\n",
" self.place_buy_order(bar, amount=self.amount * 0.95)\n",
" \n",
" self.position = 1\n",
" \n",
" elif self.position in [0, 1]:\n",
" if self.data['SMA1'].iloc[bar] < self.data['SMA2'].iloc[bar]:\n",
" if self.verbose:\n",
" print('\\n{} | *** PLACING SELL ORDER ***'.format(date))\n",
" # going short based on units\n",
" # self.place_sell_order(bar, units=(1 + self.position) * 50)\n",
" \n",
" # going short based on amount\n",
" if self.position == 1:\n",
" self.place_sell_order(bar, units=self.units)\n",
" self.place_sell_order(bar, amount=self.amount * 0.95)\n",
" \n",
" self.position = -1\n",
" \n",
" self.close_out(bar)\n",
" S0 = self.data.iloc[self.SMA2][self.symbol]\n",
" ST = self.data.iloc[bar][self.symbol]\n",
" ben_perf = (ST - S0) / S0 * 100\n",
" print('benchmark performance [%] {:.2f}'.format(ben_perf))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma = SMABacktester('AAPL.O', 10000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma.run_strategy(42, 252)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"symbols = ['AAPL.O', 'MSFT.O']\n",
"SMA1_list = [30, 42]\n",
"SMA2_list = [180, 252]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from itertools import product"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for sym, SMA1, SMA2 in product(symbols, SMA1_list, SMA2_list):\n",
" print(sym, SMA1, SMA2)\n",
" print(66 * '=')\n",
" sma = SMABacktester(sym, 10000, verbose=False)\n",
" sma.run_strategy(SMA1, SMA2)\n",
" print('\\n')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"http://hilpisch.com/tpq_logo.png\" width=\"350px\" align=\"right\">"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Raw
epat_sma_trader.py
#
# SMA-based Online Trading Algorithm
# ("Algorithmic Trading Simulator")
#
import zmq
import pandas as pd
import datetime as dt
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, '')
raw = pd.DataFrame()
SMA1 = 5
SMA2 = 10
min_length = SMA2 + 1
position = 0
ticks = 0
while True:
ticks += 1
msg = socket.recv_string()
t = dt.datetime.now()
print('{} | '.format(str(t)) + msg)
symbol, price = msg.split()
price = float(price)
raw = raw.append(pd.DataFrame({symbol: price}, index=[t,]))
data = raw.resample('3s', label='right').last().ffill()
if len(data) > min_length:
min_length += 1
data['SMA1'] = data[symbol].rolling(SMA1).mean()
data['SMA2'] = data[symbol].rolling(SMA2).mean()
if position in [0, -1]:
if data['SMA1'].iloc[-2] > data['SMA2'].iloc[-2]:
print('\n*** GOING LONG ***\n')
position = 1
# place your trading logic/code
print(data.tail(), '\n')
elif position in [0, 1]:
if data['SMA1'].iloc[-2] < data['SMA2'].iloc[-2]:
print('\n*** GOING SHORT ***\n')
position = -1
# place your trading logic/code
print(data.tail(), '\n')
Raw
epat_tick_client.py
#
# Simple Tick Data Client
# ("Algorithmic Trading Simulator")
#
import zmq
import datetime as dt
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, '')
while True:
msg = socket.recv_string()
t = dt.datetime.now()
print('{} | '.format(str(t)) + msg)
Raw
epat_tick_collector.py
#
# Simple Tick Data Collector
# ("Algorithmic Trading Simulator")
#
import zmq
import pandas as pd
import datetime as dt
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, '')
raw = pd.DataFrame()
ticks = 0
while True:
ticks += 1
msg = socket.recv_string()
t = dt.datetime.now()
print('{} | '.format(str(t)) + msg)
symbol, price = msg.split()
price = float(price)
raw = raw.append(pd.DataFrame({symbol: price}, index=[t,]))
#if ticks % 100 == 0:
# h5 = pd.HDFStore('tick_data.h5', 'a')
# h5[symbol] = raw
# h5.close()
Raw
epat_tick_server.py
#
# Simple Tick Data Server
# ("Financial Market Simulator")
#
import zmq
import time
import random
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind('tcp://127.0.0.1:5555')
AAPL = 100.
while True:
AAPL += random.gauss(0, 1) * 0.5
msg = 'AAPL {:.2f}'.format(AAPL)
socket.send_string(msg)
print(msg)
time.sleep(random.random() * 2)
Raw
tick_client.py
#
# Simple Tick Data Client
#
import zmq
import datetime
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, '')
while True:
msg = socket.recv_string()
t = datetime.datetime.now()
print(str(t) + ' | ' + msg)
Raw
tick_collector.py
#
# Simple Tick Data Collector
#
import zmq
import datetime
import pandas as pd
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, '')
raw = pd.DataFrame()
while True:
msg = socket.recv_string()
t = datetime.datetime.now()
print(str(t) + ' | ' + msg)
symbol, price = msg.split()
raw = raw.append(pd.DataFrame({'SYM': symbol, 'PRICE': float(price)}, index=[t]))
data = raw.resample('5s', label='right').last()
if len(data) % 4 == 0:
print(50 * '=')
print(data.tail())
print(50 * '=')
# simple way of storing data, needs to be adjusted for your purposes
if len(data) % 20 == 0:
# h5 = pd.HDFStore('database.h5', 'a')
# h5['data'] = data
# h5['raw'] = raw
# h5.close()
pass
Raw
tick_plot.py
#
# Simple Tick Data Plotter with ZeroMQ & http://plot.ly
#
import zmq
import datetime
import plotly.plotly as ply
from plotly.graph_objs import *
import configparser
# credentials
c = configparser.ConfigParser()
c.read('../pyalgo.cfg')
stream_ids = c['plotly']['api_tokens'].split(',')
# socket
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, '')
# plotting
s = Stream(maxpoints=100, token=stream_ids[0])
tr = Scatter(x=[], y=[], name='tick data', mode='lines+markers', stream=s)
d = Data([tr])
l = Layout(title='EPAT Tick Data Example')
f = Figure(data=d, layout=l)
ply.plot(f, filename='epat_example', auto_open=True)
st = ply.Stream(stream_ids[0])
st.open()
while True:
msg = socket.recv_string()
t = datetime.datetime.now()
print(str(t) + ' | ' + msg)
sym, value = msg.split()
st.write({'x': t, 'y': float(value)})
Raw
tick_server.py
#
# Simple Tick Data Server
#
import zmq
import time
import random
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind('tcp://127.0.0.1:5555')
AAPL = 100.
while True:
AAPL += random.gauss(0, 1) * 0.5
msg = 'AAPL %.3f' % AAPL
socket.send_string(msg)
print(msg)
time.sleep(random.random() * 2)
Raw
update.sh
gist -u 7dc47060eb51954d7aa2a576e6380d5f *
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment