00_fxcm_webinar.md

FXCM Webinar

Algo Trading: REST API & Python Wrapper

Tick Data, Historical Data, Real-Time Data, Backtesting, Orders

Dr. Yves J. Hilpisch | The Python Quants GmbH

Online, 04. April 2018

(short link to this Gist: https://goo.gl/oS1ZYw)

TPQ Resources

• http://tpq.io
• http://hilpisch.com
• http://twitter.com/dyjh
• http://pyalgo.tpq.io
• http://certificate.tpq.io

FXCM Resources

• FXCM REST API page
• FXCM REST API documentation

Risk Disclaimer

Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits. Leverage can work against you. Due to the certain restrictions imposed by the local law and regulation, German resident retail client(s) could sustain a total loss of deposited funds but are not subject to subsequent payment obligations beyond the deposited funds. Be aware and fully understand all risks associated with the market and trading. Prior to trading any products, carefully consider your financial situation and experience level. Any opinions, news, research, analyses, prices, or other information is provided as general market commentary, and does not constitute investment advice. FXCM will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information.

Slides & Codes

You find the slides under http://hilpisch.com/fxcm_algo_trading.pdf

You find HTML versions of the Jupyter notebooks under:

• http://hilpisch.com/01_fxcm_hist_data.html
• http://hilpisch.com/02_fxcm_rest_api.html

Agenda

The webinar covers the following topics:

Introduction

• The Python Quants Group
• Data-Driven Finance
• AI-First Finance

Live Demo

• Historical Tick Data
• Historical Candle Data
• Visualization
• Backtesting of AI-Based Strategies
• Streaming Data
• Order Placement

Python

If you have either Miniconda or Anaconda already installed, there is no need to install anything new.

The code that follows uses Python 3.6. For example, download and install Miniconda 3.6 from https://conda.io/miniconda.html if you do not have conda already installed.

In any case, for Linux/Mac you should execute the following lines on the shell to create a new environment with the needed packages:

conda create -n fxcm python=3.6
source activate fxcm
conda install numpy pandas matplotlib statsmodels
pip install plotly cufflinks
pip install fxcmpy
conda install ipython jupyter
jupyter notebook

On Windows, execute the following lines on the command prompt:

conda create -n fxcm python=3.6
activate fxcm
conda install numpy pandas matplotlib statsmodels
pip install plotly cufflinks
pip install fxcmpy
pip install win-unicode-console
set PYTHONIOENCODING=UTF-8
conda install ipython jupyter
jupyter notebook

Read more about the management of environments under https://conda.io/docs/using/envs.html

Books

The following book is recommended for Python data analysis: Python Data Science Handbook, O'Reilly

01_fxcm_hist_data.ipynb

02_fxcm_rest_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://hilpisch.com/tpq_logo.png\" alt=\"The Python Quants\" width=\"35%\" align=\"right\" border=\"0\"><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FXCM Algorithmic Trading Initiative\n",
    "\n",
    "## Algo Trading: REST API & Python Wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dr. Yves J. Hilpisch**\n",
    "\n",
    "\n",
    "The Python Quants GmbH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://hilpisch.com/images/finaince_visual_low.png\" width=\"300px\" align=\"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Risk Disclaimer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits. Leverage can work against you. Due to the certain restrictions imposed by the local law and regulation, German resident retail client(s) could sustain a total loss of deposited funds but are not subject to subsequent payment obligations beyond the deposited funds. Be aware and fully understand all risks associated with the market and trading. Prior to trading any products, carefully consider your financial situation and experience level. Any opinions, news, research, analyses, prices, or other information is provided as general market commentary, and does not constitute investment advice. FXCM & TPQ will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Speaker Disclaimer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The speaker is neither an employee, agent nor representative of FXCM and is therefore acting independently. The opinions given are their own, constitute general market commentary, and do not constitute the opinion or advice of FXCM or any form of personal or investment advice. FXCM assumes no responsibility for any loss or damage, including but not limited to, any loss or gain arising out of the direct or indirect use of this or any other content. Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Basic Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime as dt\n",
    "import cufflinks as cf\n",
    "from pylab import plt\n",
    "cf.set_config_file(offline=True)\n",
    "plt.style.use('seaborn')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can install `fxcmpy` via\n",
    "\n",
    "    pip install fxcmpy\n",
    "    \n",
    "The documentation is currently found under http://fxcmpy.tpq.io\n",
    "\n",
    "Read the **[license & risk warning](http://fxcmpy.tpq.io/00_quick_start.html#License)** carefully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fxcmpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fxcmpy.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting to the FXCM RESTful API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this approach, a **configuration file** is needed (see [Quick Start](http://fxcmpy.tpq.io/00_quick_start.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api = fxcmpy.fxcmpy(config_file='../fxcm.cfg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instruments = api.get_instruments()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(instruments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Historical Data via RESTful API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candles = api.get_candles('USD/JPY', period='D1', number=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10 most recent days | daily\n",
    "candles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = dt.datetime(2017, 1, 1)\n",
    "end = dt.datetime(2018, 1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candles = api.get_candles('USD/JPY', period='D1',\n",
    "                         start=start, stop=end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candles['askclose'].iplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameter `period` must be one of `m1, m5, m15, m30, H1, H2, H3, H4, H6, H8, D1, W1` or `M1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 50 most recent 1 minute bars\n",
    "candles = api.get_candles('EUR/USD', period='m1', number=150)\n",
    "candles.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving Streaming Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def output(data, dataframe):\n",
    "    print('%3d | %s | %s | %6.4f, %6.4f, %6.4f, %6.4f' \n",
    "          % (len(dataframe), data['Symbol'],\n",
    "             pd.to_datetime(int(data['Updated']), unit='ms'), \n",
    "             data['Rates'][0], data['Rates'][1],\n",
    "             data['Rates'][2], data['Rates'][3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.subscribe_market_data('EUR/USD', (output,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_last_price('EUR/USD')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.unsubscribe_market_data('EUR/USD')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Placing Orders via the RESTful API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = api.create_market_buy_order('EUR/USD', 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sel = ['tradeId', 'amountK', 'currency', 'grossPL', 'isBuy']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()[sel]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = api.create_market_buy_order('USD/JPY', 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()[sel]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = api.create_market_sell_order('EUR/USD', 25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = api.create_market_buy_order('USD/JPY', 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()[sel]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = api.create_market_sell_order('EUR/USD', 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()[sel]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.close_all_for_symbol('USD/JPY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()[sel]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.close_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.get_open_positions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backtesting AI-Based Algorithmic Trading Strategy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b style=\"color: red;\">The following example is simplified and for illustration purposes only. Among others, it does not consider transactions costs or bid-ask spreads.</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candles = api.get_candles('EUR/USD', period='m5',\n",
    "                         start=dt.datetime(2018, 3, 26),\n",
    "                          stop=dt.datetime(2018, 3, 30))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame(candles[['askclose', 'bidclose']].mean(axis=1),\n",
    "                    columns=['midclose'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.iplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['returns'] = np.log(data / data.shift(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lags = 5\n",
    "cols = []\n",
    "for lag in range(1, lags + 1):\n",
    "    col = 'lag_%s' % lag\n",
    "    data[col] = data['returns'].shift(lag)\n",
    "    cols.append(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the \"patterns\" = 2 ** lags\n",
    "np.digitize(data[cols], bins=[0])[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "2 ** len(cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = svm.SVC(C=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x, test_x, train_y, test_y = train_test_split(\n",
    "    data[cols].apply(lambda x: np.digitize(x, bins=[0])),                   \n",
    "    np.sign(data['returns']),\n",
    "    test_size=0.50, random_state=111)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x.sort_index(inplace=True)\n",
    "train_y.sort_index(inplace=True)\n",
    "test_x.sort_index(inplace=True)\n",
    "test_y.sort_index(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Fitting & Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_pred = model.predict(train_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(train_y, train_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred = model.predict(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(test_y, test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorized Backtesting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['position'] = model.predict(data[cols].apply(lambda x: np.digitize(x, bins=[0])).dropna())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['strategy'] = data['position'] * data['returns']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# in-sample | unleveraged | no bid-ask spread or transaction costs\n",
    "data.loc[train_x.index][['returns', 'strategy']].cumsum().apply(np.exp).iplot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# out-of-sample | unleveraged | no bid-ask spread or transaction costs\n",
    "data.loc[test_x.index][['returns', 'strategy']].cumsum().apply(np.exp).iplot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of trades\n",
    "sum(data['position'].diff() != 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automating A Simple Time-Based Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in range(5):\n",
    "    print(50 * '=')\n",
    "    print('TRADE NO {}'.format(_))\n",
    "    order = api.create_market_buy_order('EUR/USD', 100)\n",
    "    time.sleep(1)\n",
    "    print('POSITIONS\\n', api.get_open_positions()[sel])\n",
    "    time.sleep(7)\n",
    "    api.close_all_for_symbol('EUR/USD')\n",
    "    print('POSITIONS\\n', api.get_open_positions())\n",
    "    time.sleep(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://hilpisch.com/tpq_logo.png\" alt=\"The Python Quants\" width=\"35%\" align=\"right\" border=\"0\"><br>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

nice work sir , but from which source can I check for the spread-ask cost , or using which type of data / transformation which can help provide a reliable estimate of the bid ask spread on the above intraday positions ?
input is highly appreciated

Regarding accounts, bid-ask spreads, etc. you need to reach out to FXCM directly.