Skip to content

Instantly share code, notes, and snippets.

@yhilpisch
Last active December 6, 2024 08:03
Show Gist options
  • Select an option

  • Save yhilpisch/4ba8d5b3e17d8480e8a1d552c8b4567e to your computer and use it in GitHub Desktop.

Select an option

Save yhilpisch/4ba8d5b3e17d8480e8a1d552c8b4567e to your computer and use it in GitHub Desktop.

Executive Program in Algorithmic Trading (QuantInsti)

Python Sessions by Dr. Yves J. Hilpisch | The Python Quants GmbH

Online, 08. & 09. December 2018

Python for Finance (2nd ed.)

Sign up under http://py4fi.pqp.io to access all the Jupyter Notebooks and codes and execute them on our Quant Platform.

Short Link

http://bit.ly/epat_dec_2018

Resources

Slides & Materials

You find the introduction slides under http://hilpisch.com/epat.pdf

You find the materials about OOP under http://hilpisch.com/py4fi_oop_epat.html

Python

If you have either Miniconda or Anaconda already installed, there is no need to install anything new.

The code that follows uses Python 3.6. For example, download and install Miniconda 3.6 from https://conda.io/miniconda.html if you do not have conda already installed.

In any case, for Linux/Mac you should execute the following lines on the shell to create a new environment with the needed packages:

conda create -n epat python=3.6
source activate epat
conda install numpy pandas matplotlib statsmodels
pip install plotly==2.4.0 cufflinks
conda install ipython jupyter
jupyter notebook

On Windows, execute the following lines on the Anaconda prompt:

conda create -n epat python=3.6
activate epat
conda install numpy pandas matplotlib statsmodels
pip install plotly==2.4.0 cufflinks
pip install win-unicode-console
set PYTHONIOENCODING=UTF-8
conda install ipython jupyter
jupyter notebook

Read more about the management of environments under https://conda.io/docs/using/envs.html

Docker

To install Docker see https://docs.docker.com/install/.

To run a Ubuntu-based Docker container, execute on the shell the following:

docker run -ti -p 9000:9000 -h epat -v /Users/yves/Temp/:/root/tmp ubuntu:latest /bin/bash

Make sure to adjust the folder to be mounted accordingly.

ZeroMQ

The major resource for the ZeroMQ distributed messaging package based on sockets is http://zeromq.org/

Cloud

Use this link to get a 10 USD bonus on DigitalOcean when signing up for a new account.

Books & Resources

An overview of the Python Data Model is found under: Python Data Model

Good book about everything important in Python data analysis: Python Data Science Handbook, O'Reilly

Good book covering object-oriented programming in Python: Fluent Python, O'Reilly

Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"http://hilpisch.com/tpq_logo.png\" width=\"350px\" align=\"right\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# EPAT Session 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Executive Program in Algorithmic Trading**\n",
"\n",
"**_Event-based Backtesting_**\n",
"\n",
"Dr. Yves J. Hilpisch | The Python Quants GmbH | http://tpq.io\n",
"\n",
"<img src=\"http://hilpisch.com/images/tpq_bootcamp.png\" width=\"350px\" align=\"left\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Basic Imports"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from pylab import plt\n",
"plt.style.use('seaborn')\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Financial Data Class"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class FinancialData(object):\n",
" def __init__(self, symbol):\n",
" self.symbol = symbol\n",
" self.prepare_data()\n",
" \n",
" def prepare_data(self):\n",
" self.raw = pd.read_csv('http://hilpisch.com/tr_eikon_eod_data.csv',\n",
" index_col=0, parse_dates=True).dropna()\n",
" self.data = pd.DataFrame(self.raw[self.symbol])\n",
" self.data['Returns'] = np.log(self.data / self.data.shift(1))\n",
" \n",
" def plot_data(self, cols=None):\n",
" if cols is None:\n",
" cols = [self.symbol]\n",
" self.data[cols].plot(figsize=(10, 6), title=self.symbol)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd = FinancialData('AAPL.O')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd.data.info()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd.data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fd.plot_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Event-based View on Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# vectorized data handling = complete data set in a single step\n",
"# fd.data['AAPL.O'].plot(figsize=(10, 6));\n",
"fd.plot_data()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for bar in range(10):\n",
" print(bar)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# event-based view on data = going bar by bar \"through time\"\n",
"for bar in range(10):\n",
" print(bar, fd.data['AAPL.O'].iloc[bar])\n",
" time.sleep(1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# event-based view on data = going bar by bar \"through time\"\n",
"for bar in range(10):\n",
" print(bar, str(fd.data['AAPL.O'].index[bar])[:10], fd.data['AAPL.O'].iloc[bar])\n",
" time.sleep(.5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Backtesting Base Class"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are going to implement a **base class** for event-based backtesting with:\n",
"\n",
"* `__init__`\n",
"* `prepare_data` (`FinancialBase`)\n",
"* `plot_data` (`FinancialBase`)\n",
"* `get_date_price`\n",
"* `print_balance`\n",
"* `place_buy_order`\n",
"* `place_sell_order`\n",
"* `close_out`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import math"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amount = 5000\n",
"price = 27.85"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amount / price # --> vectorized backtesting"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"units = math.floor(amount / price) # --> event-based backtesting\n",
"units"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cost = units * price\n",
"cost"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amount - cost # cash left"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class BacktestingBase(FinancialData):\n",
" def __init__(self, symbol, amount, verbose=True):\n",
" super(BacktestingBase, self).__init__(symbol)\n",
" self.amount = amount # current cash balance\n",
" self.initial_amount = amount # initial invest/cash\n",
" self.verbose = verbose\n",
" self.units = 0\n",
" self.trades = 0\n",
" \n",
" def get_date_price(self, bar):\n",
" date = str(self.data[self.symbol].index[bar])[:10]\n",
" price = self.data[self.symbol].iloc[bar]\n",
" return date, price\n",
" \n",
" def print_balance(self, bar):\n",
" date, price = self.get_date_price(bar)\n",
" print('%s | current cash balance is %8.2f' % (date, self.amount))\n",
" \n",
" def place_buy_order(self, bar, units=None, amount=None):\n",
" date, price = self.get_date_price(bar)\n",
" if amount is not None:\n",
" units = math.floor(amount / price)\n",
" self.amount -= units * price # here tc can be included\n",
" self.units += units\n",
" self.trades += 1\n",
" if self.verbose is True:\n",
" print('%s | buying %3d units for %8.2f' % (date, units, price))\n",
" self.print_balance(bar)\n",
" \n",
" def place_sell_order(self, bar, units=None, amount=None):\n",
" date, price = self.get_date_price(bar)\n",
" if amount is not None:\n",
" units = math.floor(amount / price)\n",
" self.amount += units * price\n",
" self.units -= units\n",
" self.trades += 1\n",
" if self.verbose is True:\n",
" print('%s | selling %3d units for %8.2f' % (date, units, price))\n",
" self.print_balance(bar)\n",
" \n",
" def close_out(self, bar):\n",
" date, price = self.get_date_price(bar)\n",
" self.amount += self.units * price\n",
" print(50 * '=')\n",
" print('Closing out the position.')\n",
" print(50 * '=')\n",
" if self.units != 0:\n",
" self.trades += 1\n",
" print('%s | selling %3d units for %8.2f' % (date, self.units, price))\n",
" self.units -= self.units\n",
" self.print_balance(bar)\n",
" perf = ((self.amount - self.initial_amount) / self.initial_amount) * 100\n",
" print('%s | net performance %8.2f' % (date, perf))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb = BacktestingBase('AAPL.O', 10000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.data.info()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.get_date_price(177)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.print_balance(210)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_buy_order(209, units=15)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(bb.units, bb.trades)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_buy_order(260, amount=2000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(bb.units, bb.trades)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_sell_order(300, units=40)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.place_sell_order(350, amount=500)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(bb.units, bb.trades)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bb.close_out(400)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Long Only Backtesting Class"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class LongOnlyBacktest(BacktestingBase):\n",
" # def __init__(self, *args):\n",
" # super(LongOnlyBacktest, self).__init__(*args)\n",
" \n",
" def run_strategy(self, SMA1, SMA2):\n",
" print('\\n\\nRunning strategy for %s | SMA1=%d | SMA2=%d' % (self.symbol, SMA1, SMA2))\n",
" print(50 * '=')\n",
" self.units = 0\n",
" self.trades = 0\n",
" self.position = 0\n",
" self.amount = self.initial_amount\n",
" self.results = self.data.copy()\n",
" self.results['SMA1'] = self.results[self.symbol].rolling(SMA1).mean()\n",
" self.results['SMA2'] = self.results[self.symbol].rolling(SMA2).mean()\n",
" \n",
" for bar in range(SMA2 - 1, len(self.results)):\n",
" \n",
" if self.position == 0:\n",
" if self.results['SMA1'].iloc[bar] > self.results['SMA2'].iloc[bar]:\n",
" # self.place_buy_order(bar, units=100)\n",
" self.place_buy_order(bar, amount=self.amount * 0.8)\n",
" # self.place_buy_order(bar, amount=5000)\n",
" date, price = self.get_date_price(bar)\n",
" self.entry_cost = self.units * price\n",
" # place whatever logic reflects your strategy\n",
" self.position = 1\n",
" \n",
" elif self.position == 1:\n",
" if self.results['SMA1'].iloc[bar] < self.results['SMA2'].iloc[bar]:\n",
" # self.place_sell_order(bar, units=100)\n",
" self.place_sell_order(bar, units=self.units)\n",
" self.position = 0\n",
" # stop loss logic\n",
" else:\n",
" date, price = self.get_date_price(bar)\n",
" current_position_value = self.units * price\n",
" if (current_position_value - self.entry_cost) / self.entry_cost <= -0.05:\n",
" self.place_sell_order(bar, units=self.units)\n",
" self.position = -2 # position indicating a previous stop\n",
" self.entry_cost = 0\n",
" self.trades += 1\n",
" self.wait_days = 10\n",
" if self.verbose:\n",
" print('Closing out due to stop loss.')\n",
" \n",
" elif self.position == -2 and self.wait_days > 0:\n",
" self.wait_days -= 1\n",
" if self.wait_days == 0:\n",
" self.position = 0\n",
" \n",
" self.close_out(bar)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma = LongOnlyBacktest('AAPL.O', 10000, True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma.run_strategy(42, 252)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma = LongOnlyBacktest('AAPL.O', 10000, True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma.run_strategy(42, 252)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma.run_strategy(30, 180)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from itertools import product"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for sym in sma.raw.columns.values:\n",
" print(sym)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for sym in ['AAPL.O', 'MSFT.O']:\n",
" sma = LongOnlyBacktest(sym, 10000, False)\n",
" for SMA1, SMA2 in product([30, 42], [180, 252]):\n",
" sma.run_strategy(SMA1, SMA2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Long-Short Strategies"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class LongShortBacktest(BacktestingBase):\n",
" \n",
" def run_strategy(self, SMA1, SMA2):\n",
" print('\\n\\nRunning strategy for %s | SMA1=%d | SMA2=%d' % (self.symbol, SMA1, SMA2))\n",
" print(50 * '=')\n",
" self.units = 0\n",
" self.trades = 0\n",
" self.position = 0\n",
" self.entry_value = 0\n",
" self.amount = self.initial_amount\n",
" self.results = self.data.copy()\n",
" self.results['SMA1'] = self.results[self.symbol].rolling(SMA1).mean()\n",
" self.results['SMA2'] = self.results[self.symbol].rolling(SMA2).mean()\n",
" \n",
" for bar in range(SMA2 - 1, len(self.results)):\n",
" date, price = self.get_date_price(bar)\n",
" current_position_value = self.units * price\n",
" diff = current_position_value - self.entry_value\n",
" rdiff = diff / self.entry_value\n",
" rdiff = rdiff if self.position >= 0 else -rdiff\n",
" if self.verbose:\n",
" print('%s | %8.2f | %8.2f | %8.3f | %7.3f' %\n",
" (date, self.entry_value, current_position_value, diff, rdiff))\n",
" \n",
" if self.position in [0, -1, -2]:\n",
" if self.results['SMA1'].iloc[bar] > self.results['SMA2'].iloc[bar]:\n",
" if self.position == -1:\n",
" self.place_buy_order(bar, amount=-self.units)\n",
" # self.place_buy_order(bar, amount=5000)\n",
" self.place_buy_order(bar, amount=self.amount * 0.8)\n",
" date, price = self.get_date_price(bar)\n",
" self.entry_value = self.units * price\n",
" self.position = 1\n",
" elif self.entry_value != 0:\n",
" if (current_position_value - self.entry_value) / -self.entry_value <= -0.075:\n",
" self.place_buy_order(bar, units=-self.units)\n",
" self.position = -2\n",
" self.entry_value = 0\n",
" if self.verbose:\n",
" print('Closing out short position due to stop loss.')\n",
" \n",
" elif self.position in [0, 1, 2]:\n",
" if self.results['SMA1'].iloc[bar] < self.results['SMA2'].iloc[bar]:\n",
" if self.position == 1:\n",
" self.place_sell_order(bar, amount=self.units)\n",
" # self.place_sell_order(bar, amount=5000)\n",
" self.place_sell_order(bar, amount=self.amount * 0.8)\n",
" self.entry_value = self.units * price\n",
" self.position = -1\n",
" elif self.entry_value != 0:\n",
" if (current_position_value - self.entry_value) / self.entry_value <= -0.075:\n",
" self.place_sell_order(bar, units=self.units)\n",
" self.position = 2\n",
" self.entry_value = 0\n",
" if self.verbose:\n",
" print('Closing out long position due to stop loss.')\n",
" \n",
" self.close_out(bar)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma = LongShortBacktest('AAPL.O', 10000, False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sma.run_strategy(42, 252)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for sym in ['AAPL.O', 'MSFT.O']:\n",
" sma = LongShortBacktest(sym, 10000, False)\n",
" for SMA1, SMA2 in product([30, 42], [180, 252]):\n",
" sma.run_strategy(SMA1, SMA2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some improvements (as an exercise):\n",
"\n",
"* include different signals (momentum)\n",
"* include proportional and fixed transaction costs\n",
"* allow for different time periods for the backtest"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"http://hilpisch.com/tpq_logo.png\" width=\"350px\" align=\"right\">"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
#
# Simple Tick Data Client
#
import zmq
import datetime
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')
while True:
msg = socket.recv_string()
t = datetime.datetime.now()
print(str(t.time()) + ' | ' + msg)
#
# Simple Tick Data Collector
#
import zmq
import datetime
import pandas as pd
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')
raw = pd.DataFrame()
while True:
msg = socket.recv_string()
tr = datetime.datetime.now()
print(str(tr.time()) + ' | ' + msg)
sym, price, ts = msg.split()
raw = raw.append(pd.DataFrame(
{'ts': pd.Timestamp(ts),
'sym': sym, 'price': float(price)},
index=[tr,]))
#
# Simple Tick Data Server
#
import zmq
import time
import random
import datetime
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind('tcp://127.0.0.1:5555')
AAPL = 100.
while True:
AAPL += random.gauss(0, 1) * 0.5
t = datetime.datetime.now()
msg = 'AAPL {:.3f} {}'.format(AAPL, t.time())
socket.send_string(msg)
print(msg)
time.sleep(random.random() * 2)
#
# Simple Tick Data Collector
#
import zmq
import datetime
import pandas as pd
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://127.0.0.1:5555')
socket.setsockopt_string(zmq.SUBSCRIBE, 'AAPL')
raw = pd.DataFrame()
SMA1 = 3
SMA2 = 6
min_length = SMA2 + 1
position = 0
while True:
msg = socket.recv_string()
tr = datetime.datetime.now()
print(str(tr.time()) + ' | ' + msg)
sym, price, ts = msg.split()
raw = raw.append(pd.DataFrame(
{'ts': pd.Timestamp(ts),
'sym': sym, 'price': float(price)},
index=[tr,]))
data = raw.resample('5s', label='right').last().ffill()
if len(data) > min_length:
min_length += 1
data['SMA1'] = data['price'].rolling(SMA1).mean()
data['SMA2'] = data['price'].rolling(SMA2).mean()
if data['SMA1'].iloc[-2] > data['SMA2'].iloc[-2]:
print('\n**** GOING/STAYING LONG ****')
elif data['SMA1'].iloc[-2] < data['SMA2'].iloc[-2]:
print('\n**** GOING/STAYING SHORT ****')
print(data[['SMA1', 'SMA2']].tail(), '\n\n')
gist -u 4ba8d5b3e17d8480e8a1d552c8b4567e *
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment