00_fxcm_webinar.md

FXCM Webinar Series

on Algorithmic Trading

Python & Historical Tick Data

Dr. Yves J. Hilpisch | The Python Quants GmbH

Online, 24. October 2017

(short link to this Gist: https://goo.gl/C1WD8r)

Resources

• http://tpq.io
• http://hilpisch.com
• http://twitter.com/dyjh
• http://pyalgo.tpq.io
• http://certificate.tpq.io

Risk Disclaimer

Trading forex/CFDs on margin carries a high level of risk and may not be suitable for all investors as you could sustain losses in excess of deposits. Leverage can work against you. Due to the certain restrictions imposed by the local law and regulation, German resident retail client(s) could sustain a total loss of deposited funds but are not subject to subsequent payment obligations beyond the deposited funds. Be aware and fully understand all risks associated with the market and trading. Prior to trading any products, carefully consider your financial situation and experience level. Any opinions, news, research, analyses, prices, or other information is provided as general market commentary, and does not constitute investment advice. FXCM will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information.

Slides

You find the slides under http://hilpisch.com/fxcm_webinar_tick_data.pdf

Agenda

The webinar covers the following topics:

Introduction

• The Python Quants Group
• Driving Forces in Algorithmic Trading
• Why Python for Algorithmic Trading?

Live Demo

• Using Python & pandas for Backtesting
• Working with FXCM Historical Tick Data
• Adding Indicators to Data Sets
• Visualization of OHLC Data & Studies

Data

You find the historical EOD data set used under http://hilpisch.com/eurusd.csv (as provided by FXCM Forex Capital Markets Ltd.).

You find further information about the historical tick data source under https://github.com/FXCMAPI/FXCMTickData (as provided by FXCM Forex Capital Markets Ltd.).

Python

If you have either Miniconda or Anaconda already installed, there is no need to install anything new.

The code that follows uses Python 3.6. For example, download and install Miniconda 3.6 from https://conda.io/miniconda.html if you do not have conda already installed.

In any case, for Linux/Mac you should execute the following lines on the shell to create a new environment with the needed packages:

conda create -n fxcm python=3.6
source activate fxcm
conda install numpy pandas matplotlib statsmodels
pip install plotly cufflinks
conda install ipython jupyter
jupyter notebook

On Windows, execute the following lines on the command prompt:

conda create -n fxcm python=3.6
activate fxcm
conda install numpy pandas matplotlib statsmodels
pip install plotly cufflinks
pip install win-unicode-console
set PYTHONIOENCODING=UTF-8
conda install ipython jupyter
jupyter notebook

Read more about the management of environments under https://conda.io/docs/using/envs.html

Books

The following book is recommended for Python data analysis: Python Data Science Handbook, O'Reilly

01_fxcm_tick_data.ipynb

fxcm_ti.py

#
# Python Class for Technical Analysis
# eg Based on Data from FXCM Financial Capital Markets Limited
#
# The Python Quants GmbH
# October 2017
#
# Note that this code and the data (service) accessed
# by the code are for illustration purposes only.
# They come with no warranties or representations,
# to the extent permitted by applicable law.
#
# Read the RISK DISCLAIMER carefully.
#
# Status: Experimental
#
import numpy as np
import pandas as pd
pd.options.mode.chained_assignment = None

class technical_indicators(object):
    """ A class to generate technical, financial indicators 
    based on pandas DataFrame objects """
    def __init__(self, data, index=''):
        if not isinstance(data, pd.DataFrame):
            raise TypeError('data must be a pandas DataFrame')
        self.data = data
        if index != "" and index in self.data.columns:
            self.data.set_index(index, inplace=True)

    def add_columns(self, data, name):
        """ Add columns to the data object """
        self.data[name] = data

    def remove_columns(self, name):
        """ Remove the column with name 'name' """
        self.data = self.data.drop([name], axis=1)

    def get_data(self):
        """ Return the data as pandas DataFrame object """
        return self.data
    
    def check_periods(self, periods):
        try:
            periods = int(periods)
        except:
            raise TypeError('periods must be an integer')
        if periods < 1:
            raise TypeError('periods must be a positive integer')
        return periods

    def sma(self, column_name, periods):
        """ Return the simple moving average (SMA) of the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods: integer
            the length of the time window

        Returns:
        ======== 
        sma: ndarray
            the simple moving average data set
        """
        periods = self.check_periods(periods)
        sma = self.data[column_name].rolling(periods).mean()
        return sma

    def add_sma(self, column_name, periods):
        """ Add simple moving average (SMA) of the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods: integer
            the length of the time window

        Returns:
        ========
        name: string
            the name of the added SMA column
        """
        periods = self.check_periods(periods)
        name = 'sma_%s_%s' % (periods, column_name)
        data = self.sma(column_name, periods)
        self.add_columns(data, name)
        return name

    def ewma(self, column_name, periods):
        """ Return the exponential weighted moving average (EWMA) of the data

        Arguments:
        ==========
        column_name: string,
            the name of the data set's column to use

        periods: integer, 
            the length of the time window

        Returns:
        ======== 
        ewma: ndarray
            the exponentially weighted moving average (EWMA) data set
        """
        periods = self.check_periods(periods)
        ewma = self.data[column_name].ewm(span=periods,
                              min_periods=periods).mean()
        return ewma

    def add_ewma(self, column_name, periods):
        """ Add the exponential weighted moving average (EWMA) of the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods: integer
            the length of the time window

        Returns:
        ======== 
        name: string
            name of the added EWMA column
        """
        periods = self.check_periods(periods)
        name = 'ewma_%s_%s' % (periods, column_name)
        data = self.ewma(column_name, periods)
        self.add_columns(data, name)
        return name

    def bollinger_upper(self, column_name, periods):
        """ Return upper Bollinger band of the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods: integer
            the length of the time window

        Returns:
        ========
        upper_bol: ndarray
            the upper Bollinger band data
        """
        periods = self.check_periods(periods)
        stdev = self.data[column_name].rolling(periods).std()
        upper_bol = self.sma(column_name, periods) + (2 * stdev)
        return upper_bol

    def add_bollinger_upper(self,  column_name, periods):
        """ Add upper Bollinger band to the data

        Arguments:
        ==========

        column_name: string,
            the name of the dateset's column to use.

        periods: integer, 
            the length of the time window.

        Returns:
        ======== 
        name: string
            the name of the added column
        """
        periods = self.check_periods(periods)
        name = 'boll_up_%s_%s' % (periods, column_name)
        data = self.bollinger_upper(column_name, periods)
        self.add_columns(data, name)
        return name

    def bollinger_lower(self, column_name, periods):
        """ Return upper Bollinger band of the data

        Arguments:
        ==========

        column_name: string,
            the name of the dateset's column to use.

        periods: integer, 
            the length of the time window.

        Returns:
        ========
        lower_bol: ndarray
            the lower Bollinger band data
        """
        periods = self.check_periods(periods)
        stdev = self.data[column_name].rolling(periods).std()
        lower_bol = self.sma(column_name, periods) - (2 * stdev)
        return lower_bol

    def add_bollinger_lower(self,  column_name, periods):
        """ Add lower Bollinger band to the data

        Arguments:
        ==========

        column_name: string,
            the name of the dateset's column to use.

        periods: integer, 
            the length of the time window.

        Returns:
        ======== 
        name: string
            the name of the added column
        """
        periods = self.check_periods(periods)
        name = 'boll_low_%s_%s' % (periods, column_name)
        data = self.bollinger_lower(column_name, periods)
        self.add_columns(data, name)
        return name

    def rsi(self, column_name, periods):
        """ Return the relative strength index (RSI) of the data

        Arguments:
        ==========

        column_name: string,
            the name of the dateset's column to use.

        periods: integer,
            the lenght of the time window.

        Returns:
        ========
        rsi: ndarray
            relative strength index (RSI) data
        """
        periods = self.check_periods(periods)
       
        data = self.data[column_name]
        delta = data.diff()
        delta = delta[1:] 

        up, down = delta.copy(), delta.copy()
        up[up < 0] = 0
        down[down > 0] = 0
        down = down.abs()

        sma_up =  up.rolling(periods).mean()
        sma_down = down.rolling(periods).mean()

        rs = sma_up / sma_down
        rsi = 100.0 - (100.0 / (1.0 + rs))
        
        return rsi

    def add_rsi(self, column_name, periods):
        """ Add the relative streng index (RSI) of the data

        Arguments:
        ==========

        column_name: string
            the name of the data set's column to use

        periods: integer
            the length of the time window

        Returns:
        ======== 
        name: string
            the name of the added column
        """

        try:
            periods = int(periods)
        except:
            raise TypeError('periods must be an integer')
        if periods < 1:
            raise TypeError('periods must be positive')

        name = 'rsi_%s_%s' % (periods, column_name)
        data = self.rsi(column_name, periods)
        self.add_columns(data, name)

        return name

    def macd(self, column_name, periods_fast, periods_slow):
        """ Return the moving average convergence/divergence (MACD) of the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods_fast: integer,
            the length of the shorter ewma time window

        periods_slow: integer 
            the length of the longer ewma time window

        Returns:
        ======== 
        macd: ndarray
            the moving average convergence/divergence (MACD) data
        """
        periods_fast = self.check_periods(periods_fast)
        periods_slow = self.check_periods(periods_slow)

        if periods_slow < periods_fast:
            raise ValueError('periods_fast must be smaller/shorter than periods_slow') 

        ewma_fast = self.ewma(column_name, periods_fast)
        ewma_slow = self.ewma(column_name, periods_slow)
        macd = ewma_fast - ewma_slow
        return macd

    def add_macd(self, column_name, periods_fast, periods_slow):
        """ Add moving average convergence/divergence (MACD) to the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods_fast: integer, 
            the length of the shorter ewma time window

        periods_slow: integer, 
            the length of the longer ewma time window

        Returns:
        ======== 
        name: string
            the name of the added column
        """
        periods_fast = self.check_periods(periods_fast)
        periods_slow = self.check_periods(periods_slow)

        if periods_slow < periods_fast:
            raise ValueError('periods_fast must be smaller/shorter than periods_slow') 

        name = 'macd_%sx%s_%s' % (periods_fast, periods_slow, column_name)
        data = self.macd(column_name, periods_fast, periods_slow)
        self.add_columns(data, name)
        return name

    def macd_signal(self, column_name, periods_fast, periods_slow, 
                     periods_signal):
        """ Return the signal generated by the MACD of the data

        Arguments:
        ==========
        column_name: string
            the name of the dat aset's column to use

        periods_fast: integer
            the length of the shorter ewma time window

        periods_slow: integer 
            the length of the longer ewma time window

        periods_signal: integer 
            the length of the time window used for signal generation

        Returns:
        ======== 
        macd_signal: ndarray
            the moving average convergence/divergence (MACD) signal data
        """
        periods_fast = self.check_periods(periods_fast)
        periods_slow = self.check_periods(periods_slow)
        periods_signal = self.check_periods(periods_signal)

        if periods_slow < periods_fast:
            raise ValueError('periods_fast must be smaller/shorter than periods_slow') 

        macd = self.macd(column_name, periods_fast, periods_slow)
        macd_signal = macd.ewm(span=periods_signal, min_periods=periods_signal).mean()
        return macd_signal

    def add_macd_signal(self, column_name, periods_fast, periods_slow,
                        periods_signal):
        """ Add the signal gernerated by the macd of the data

        Arguments:
        ==========
        column_name: string
            the name of the data set's column to use

        periods_fast: integer
            the length of the shorter ewma time window

        periods_slow: integer 
            the length of the longer ewma time window

        periods_signal: interger
            the length of the time window used for signal generation

        Returns:
        ======== 
        name: string
            the name of the added column
        """
        periods_fast = self.check_periods(periods_fast)
        periods_slow = self.check_periods(periods_slow)
        periods_signal = self.check_periods(periods_signal)

        if periods_slow < periods_fast:
            raise ValueError('periods_fast must be smaller/shorter than periods_slow') 

        name = 'macd_signal_%sx%sx%s_%s' % (periods_fast, periods_slow,
                                            period_signal, column_name)
        data = self.macd_signal(column_name, periods_fast, periods_slow,
                                period_signal)
        self.add_columns(data, name)
        return name

fxcm_tick_reader.py

#
# A Class for the Retrieval of Historical Tick Data
# as Provided by FXCM Financial Capital Markets Limited
#
# The Python Quants GmbH
# October 2017
#
# Note that this code and the data service accessed
# by the code are for illustration purposes only.
# They come with no warranties or representations,
# to the extent permitted by applicable law.
#
#
# Read the RISK DISCLAIMER carefully.
#
# Status: Experimental
#
import gzip
import pandas as pd
import urllib.request
import datetime as dt
from io import BytesIO, StringIO


class fxcm_tick_reader(object):
    """ A class to retrieve historical tick data provided by FXCM. """

    symbols = ('AUDCAD', 'AUDCHF', 'AUDJPY', 'AUDNZD', 'CADCHF', 'EURAUD',
               'EURCHF', 'EURGBP', 'EURJPY', 'EURUSD', 'GBPCHF', 'GBPJPY',
               'GBPNZD', 'GBPUSD', 'GBPCHF', 'GBPJPY', 'GBPNZD', 'NZDCAD',
               'NZDCHF', 'NZDJPY', 'NZDUSD', 'USDCAD', 'USDCHF', 'USDJPY')

    def __init__(self, symbol, start, stop):
        """ Constructor of the class.

        Arguments:
        ==========
        symbol: string
            one of symbols

        start: datetime.date
            the first day to retrieve data for

        stop: datetime.date
            the last day to retrieve data for

        """

        if not (isinstance(start, dt.datetime) or isinstance(start, dt.date)):
            raise TypeError('start must be a datetime object')
        else:
            self.start = start

        if not (isinstance(stop, dt.datetime) or isinstance(stop, dt.date)):
            raise TypeError('stop must be a datetime object')
        else:
            self.stop = stop

        if self.start > self.stop:
            raise ValueError('Invalid date range')

        if symbol not in self.symbols:
            msg = 'Symbol %s is not supported. For a list of supported'
            msg += ' symbols, call get_available_symbols()'
            raise ValueError(msg % symbol)
        else:
            self.symbol = symbol

        self.data = None
        self.url = 'https://tickdata.fxcorporate.com/%s/%s/%s.csv.gz'
        self.__fetch_data__()

    def get_raw_data(self):
        """ Returns the raw data set as pandas DataFrame """
        return self.data

    def get_data(self, start=None, end=None):
        """ Returns the requested data set as pandas DataFrame;
        DataFrame index is converted to DatetimeIndex object """
        try:
            self.data_adj
        except:
            data = self.data.copy()
            index = pd.to_datetime(data.index.values,
                                   format='%m/%d/%Y %H:%M:%S.%f')
            data.index = index
            self.data_adj = data
        data = self.data_adj
        if start is not None:
            data = data[data.index >= start]
        if end is not None:
            data = data[data.index <= end]
        return data

    @classmethod
    def get_available_symbols(cls):
        """ Returns all available symbols """
        return cls.symbols

    def __fetch_data__(self):
        """ Retrieve the data for the given symbol and the given time window """
        self.data = pd.DataFrame()
        running_date = self.start
        seven_days = dt.timedelta(days=7)
        while running_date <= self.stop:
            year, week, noop = running_date.isocalendar()
            url = self.url % (self.symbol, year, week)
            data = self.__fetch_dataset__(url)
            if len(self.data) == 0:
                self.data = data
            else:
                self.data = pd.concat((self.data, data))
            running_date = running_date + seven_days

    def __fetch_dataset__(self, url):
        """ Retrieve data for the given symbol for one week """
        print('Fetching data from: %s' % url)
        requests = urllib.request.urlopen(url)
        buf = BytesIO(requests.read())
        f = gzip.GzipFile(fileobj=buf)
        data = f.read()
        data_str = data.decode('utf-16')
        data_pandas = pd.read_csv(StringIO(data_str), index_col=0)
        return data_pandas

Hi!,

Thanks for sharing this Demo! I encountered a problem with Quant Figures, on the following lines:

Basic Quant Figures
qf = cf.QuantFig(df, title='EUR/USD', legend='top',
name='EUR/USD', datalegend=False)
iplot(qf.iplot(asFigure=True))

datalegend=False is throwing me an error : ValueError: Invalid property specified for object of type plotly.graph_objs.candlestick.Decreasing: 'showlegend'

If I replace it with showlegend=False it runs but the legend of course is not showing and I can't activate/deactivate the indicators form the charts, Any suggestions?

Regards!

Potly and also cufflinks are changing the API somehow regularly. Would need to research this topic myself. I do not have an immediate solution/suggestion.

…

On Thu, Jun 18, 2020 at 10:37 PM jlvega ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi!, Thanks for sharing this Demo! I encountered a problem with Quant Figures, on the following lines: Basic Quant Figures qf = cf.QuantFig(df, title='EUR/USD', legend='top', name='EUR/USD', *datalegend=False*) iplot(qf.iplot(asFigure=True)) *datalegend=False* is throwing me an error : *ValueError: Invalid property specified for object of type plotly.graph_objs.candlestick.Decreasing: 'showlegend'* If I replace it with *showlegend=False* it runs but the legend of course is not showing and I can't activate/deactivate the indicators form the charts, Any suggestions? Regards! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://gist.github.com/e802541f8def69a299032c359d0f1008#gistcomment-3346732>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABC4LAFU4UEND7OTCPRTMW3RXJ3JZANCNFSM4OCEFIKA> .

-- Dr Yves J Hilpisch CEO | http://tpq.io | @dyjh <http://twitter.com/dyjh> The Python Quants | The AI Machine