Skip to content

Instantly share code, notes, and snippets.

@degerhan

degerhan/live_sharpe_sortino.ipynb

Last active August 18, 2021 16:07
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save degerhan/172be51b80c3d9b218d3d46c5fad6a69 to your computer and use it in GitHub Desktop.
Save degerhan/172be51b80c3d9b218d3d46c5fad6a69 to your computer and use it in GitHub Desktop.
Download ZIP
live_sharpe_sortino.ipynb
Display the source blob
Display the rendered blob
Raw
live_sharpe_sortino.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"orig_nbformat": 2,
"colab": {
"name": "live_sharpe_sortino.ipynb",
"provenance": [],
"collapsed_sections": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/degerhan/172be51b80c3d9b218d3d46c5fad6a69/live_sharpe_sortino.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iN0nNtRKWMyS"
},
"source": [
"# Live annualized Sharpe and Sortino metrics for Numerai\n",
"Copyright Degerhan Usluel, license: [MIT](https://opensource.org/licenses/MIT)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CxCtAsPdWMyV"
},
"source": [
"## The value of daily scores\n",
"\n",
"Your numerai models earn or burn when a round is resolved. The 20 daily scores you receive in between are for kicks and giggles, and don't mean anything.\n",
"\n",
"Or do they? A model with a slow and steady climb might be more deserving of your hard earned NMRs, even when other models resolve with higher corr and mmc scores.\n",
"\n",
"The historical daily scores are also a high resolution indicator of your model's volatility under different staking regimes, and may influence your choice of stake type.\n",
"\n",
"This notebook calculates the Sharpe and Sortino ratios for a list of models, for the four different staking regimes, based on their daily gyrations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yt1iJX87WMyX"
},
"source": [
"## Scaling daily return series to annual Sharpe ratio\n",
"\n",
"Published hedge fund Sharpe ratios are annualized numbers. Same for mutual funds, see [Morningstar Methodology Paper](https://gladmainnew.morningstar.com/directhelp/Methodology_StDev_Sharpe.pdf).\n",
"\n",
"The financial industry practice is to start with monthly return series. Often arithmetic average is used for returns (12 x monthly mean return) rather than geometric, and monthly series standard deviation is time-scaled to annual by multiplying with $\\sqrt{12}$.\n",
"\n",
"Andrew Lo warns us about the perils of serial correlations and multiplying by $\\sqrt{12}$ in [Statistics of Sharpe Ratios](https://alo.mit.edu/wp-content/uploads/2017/06/The-Statistics-of-Sharpe-Ratios.pdf) ). Yet, warts and all, this is the metric every fund publishes, so we shamelessly use $\\sqrt{252}$ to time-scale from daily to annualized volatility in this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8jpY8yCKWMyY"
},
"source": [
"# Account Performance"
]
},
{
"cell_type": "code",
"metadata": {
"id": "nzFrdhj6WMyb",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "6d038ab6-6b7d-47db-b6e2-9922fee2a365"
},
"source": [
"# install packages\n",
"# !pip install pandas numpy\n",
"!pip install numerapi"
],
"execution_count": 29,
"outputs": [
{
"output_type": "stream",
"text": [
"Requirement already satisfied: numerapi in /usr/local/lib/python3.7/dist-packages (2.4.3)\n",
"Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from numerapi) (2.8.1)\n",
"Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from numerapi) (2018.9)\n",
"Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.7/dist-packages (from numerapi) (7.1.2)\n",
"Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from numerapi) (2.23.0)\n",
"Requirement already satisfied: tqdm>=4.29.1 in /usr/local/lib/python3.7/dist-packages (from numerapi) (4.41.1)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil->numerapi) (1.15.0)\n",
"Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->numerapi) (3.0.4)\n",
"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->numerapi) (1.24.3)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->numerapi) (2020.12.5)\n",
"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->numerapi) (2.10)\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "iKL3MID5WMyc"
},
"source": [
"# Import dependencies\n",
"import pandas as pd\n",
"import numpy as np\n",
"import numerapi"
],
"execution_count": 30,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XqV5_vbkWMyd",
"outputId": "fdf23251-dcbe-4561-8f25-720379e04713"
},
"source": [
"# Connect to numerai\n",
"api = numerapi.NumerAPI()\n",
"\n",
"# Get round resolved states\n",
"resolve_status = pd.DataFrame(api.get_competitions()).set_index(\"number\")[\n",
" \"resolvedGeneral\"\n",
"]"
],
"execution_count": 31,
"outputs": [
{
"output_type": "stream",
"text": [
"2021-03-03 18:00:51,230 INFO numerapi.base_api: getting rounds...\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "sAWcfJhHWMyd"
},
"source": [
"def get_compund_stake(daily_scores, resolved_rounds, weights):\n",
" \"\"\"\n",
" Starts with stake of 1 and compound weekly based on resolved rounds.\n",
" Numerai tournament compounds every four weeks, so while not entirely accurate,\n",
" a pretty good approximation.\n",
" \"\"\"\n",
" stake = pd.Series(data=1, index=daily_scores.index)\n",
" stake = stake.multiply(\n",
" 1\n",
" + weights[0] * resolved_rounds[\"correlation\"]\n",
" + weights[1] * resolved_rounds[\"mmc\"],\n",
" fill_value=1,\n",
" ).cumprod()\n",
"\n",
" return stake"
],
"execution_count": 32,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "naC-QeoTWMye"
},
"source": [
"def get_account(account_name):\n",
" \"\"\"\n",
" Retrieve daily scores and calculate hypotethical 'unrealized brokerage\n",
" account value' for an account\n",
" \"\"\"\n",
" daily_performances = pd.DataFrame(api.daily_submissions_performances(account_name))\n",
"\n",
" # Each day, each active round submission promises a payout seperate from the others,\n",
" # therefore sum() of all rounds scores for a day are used instead of mean()\n",
" daily_scores = (\n",
" daily_performances.dropna(subset=[\"mmc\",\"correlation\"])\n",
" .sort_values(by=\"date\")\n",
" .groupby(\"date\")\n",
" .sum()[[\"correlation\", \"mmc\"]]\n",
" )\n",
"\n",
" # Last known date for each round submission\n",
" round_lastday = (\n",
" daily_performances.dropna()\n",
" .sort_values(by=\"date\")\n",
" .groupby(\"roundNumber\")\n",
" .last()\n",
" .reset_index()\n",
" .set_index(\"date\")\n",
" )\n",
"\n",
" # Pick resolved rounds\n",
" round_lastday[\"resolved\"] = round_lastday[\"roundNumber\"].map(resolve_status)\n",
" resolved_rounds = round_lastday[round_lastday.resolved]\n",
"\n",
" # Build the hypothethical stake that grows with each resolved round\n",
" money = pd.DataFrame(index=daily_scores.index)\n",
" money[\"stake_corr\"] = get_compund_stake(daily_scores, resolved_rounds, (1, 0))\n",
" money[\"stake_halfmmc\"] = get_compund_stake(daily_scores, resolved_rounds, (1, 0.5))\n",
" money[\"stake_1mmc\"] = get_compund_stake(daily_scores, resolved_rounds, (1, 1))\n",
" money[\"stake_2mmc\"] = get_compund_stake(daily_scores, resolved_rounds, (1, 2))\n",
"\n",
" # Daily scores are akin to daily changes in your brokerage account\n",
" money[\"score_corr\"] = daily_scores.correlation\n",
" money[\"score_halfmmc\"] = daily_scores.correlation + 0.5 * daily_scores.mmc\n",
" money[\"score_1mmc\"] = daily_scores.correlation + 1.0 * daily_scores.mmc\n",
" money[\"score_2mmc\"] = daily_scores.correlation + 2.0 * daily_scores.mmc\n",
"\n",
" # The paper value of your account changes with the daily score applied to the stake\n",
" money[\"account_corr\"] = money[\"stake_corr\"] * (1 + money[\"score_corr\"])\n",
" money[\"account_halfmmc\"] = money[\"stake_halfmmc\"] * (1 + money[\"score_halfmmc\"])\n",
" money[\"account_1mmc\"] = money[\"stake_1mmc\"] * (1 + money[\"score_1mmc\"])\n",
" money[\"account_2mmc\"] = money[\"stake_2mmc\"] * (1 + money[\"score_2mmc\"])\n",
"\n",
" # All this work was to get our daily 'brokerage account value' equivalent\n",
" money = money[[\"account_corr\", \"account_halfmmc\", \"account_1mmc\", \"account_2mmc\"]]\n",
"\n",
" # Log returns and percentage change are approximately same for small numbers\n",
" # I use log as a creature of habit, no judgment if you prefer percentages\n",
" account_returns = pd.concat(\n",
" [\n",
" daily_scores,\n",
" money,\n",
" # money.pct_change().add_prefix(\"roc_\"),\n",
" (np.log(money) - np.log(money.shift(1))).add_prefix(\"log_\"),\n",
" ],\n",
" axis=1,\n",
" ).dropna()\n",
"\n",
" account_returns.name = account_name\n",
"\n",
" return account_returns"
],
"execution_count": 33,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "KeUANHDgWMyf"
},
"source": [
"def get_performance(account_returns, lookback_days):\n",
" \"\"\"\n",
" Calculate annualized Sharpe and Sortino ratios\n",
" \"\"\"\n",
" # Extract the log return columns of interest\n",
" log_returns = account_returns[\n",
" [column for column in account_returns.columns if column.startswith(\"log\")]\n",
" ].tail(lookback_days)\n",
" log_returns.columns = [\"corr\", \"halfmmc\", \"1mmc\", \"2mmc\"]\n",
"\n",
" # Financial industry practice is 12 x monthly mean return\n",
" # for daily returns, use 252 trading days per year of US stock markets\n",
" annual_return = 252 * log_returns.mean()\n",
"\n",
" # could use monthly compounding if you feel like bending those boundaries\n",
" # annual_return = np.power(1 + 21 * log_returns.mean(), 12) - 1\n",
"\n",
" # Annual volatility is calculated by multiplying daily volatility with sqrt of time \n",
" sharpe_ratio = annual_return / (np.sqrt(252) * log_returns.std())\n",
"\n",
" # Sortino ratio considers stdev of negative days, positive suprises are a blessing\n",
" sortino_ratio = annual_return / (np.sqrt(252) * log_returns[log_returns < 0].std())\n",
"\n",
" # Let's see how long this track record is really for\n",
" actual_days = pd.Series({\"rows\": len(log_returns)})\n",
"\n",
" return annual_return.add_prefix(\"annual_return_\").append(\n",
" sharpe_ratio.add_prefix(\"sharpe_\")\n",
" .append(sortino_ratio.add_prefix(\"sortino_\"))\n",
" .append(actual_days)\n",
" )"
],
"execution_count": 34,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "GVyjoFHEWMyg"
},
"source": [
"# Default lookback is 6 months\n",
"DEFAULT_LOOKBACK = 130\n",
"\n",
"# helper function to hit the API and compute performances in a single call\n",
"def get_account_performance(account_name, lookback_days = DEFAULT_LOOKBACK):\n",
" account_returns = get_account(account_name)\n",
" return get_performance(account_returns, lookback_days).rename(account_name)"
],
"execution_count": 35,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JvH77NmzWMyh",
"outputId": "6448ed70-620b-463e-bd9e-b43a83c0ab9e"
},
"source": [
"# lets see how the integration_test is doing\n",
"get_account_performance(\"integration_test\")"
],
"execution_count": 36,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"annual_return_corr 1.402745\n",
"annual_return_halfmmc 1.377893\n",
"annual_return_1mmc 1.352570\n",
"annual_return_2mmc 1.300451\n",
"sharpe_corr 1.980763\n",
"sharpe_halfmmc 1.763217\n",
"sharpe_1mmc 1.565455\n",
"sharpe_2mmc 1.238449\n",
"sortino_corr 3.190071\n",
"sortino_halfmmc 2.731549\n",
"sortino_1mmc 2.305267\n",
"sortino_2mmc 1.733450\n",
"rows 130.000000\n",
"Name: integration_test, dtype: float64"
]
},
"metadata": {
"tags": []
},
"execution_count": 36
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "IHYMfcpUWMyi"
},
"source": [
"# your models and others you want to bencmark\n",
"account_list = [\n",
" \"degerhan\",\n",
" \"era__mix__2000\",\n",
" \"hiryuu\",\n",
" \"integration_test\",\n",
" \"mdo\",\n",
" \"sorios\",\n",
" \"themicon\",\n",
" \"arbitrage\",\n",
"]"
],
"execution_count": 37,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "GNB8JdTmWMyi"
},
"source": [
"# Rather than use get_account_performance helper,\n",
"# lets hit the API once for all accounts, which takes a few seconds\n",
"account_returns = [get_account(account) for account in account_list]"
],
"execution_count": 38,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 314
},
"id": "6-PxdWqlWMyi",
"outputId": "25cd4c39-5f8e-4d82-ae0c-add7b5b71c08"
},
"source": [
"# then explore with lookback periods without hitting the API again\n",
"LOOKBACK = 130\n",
"MIN_ROWS = 30\n",
"\n",
"account_performances = pd.DataFrame(\n",
" [\n",
" get_performance(account_df, LOOKBACK).rename(account_df.name)\n",
" for account_df in account_returns\n",
" ]\n",
")\n",
"\n",
"account_performances[account_performances.rows > MIN_ROWS].sort_values(\n",
" by=\"sortino_2mmc\", ascending=False\n",
")"
],
"execution_count": 39,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>annual_return_corr</th>\n",
" <th>annual_return_halfmmc</th>\n",
" <th>annual_return_1mmc</th>\n",
" <th>annual_return_2mmc</th>\n",
" <th>sharpe_corr</th>\n",
" <th>sharpe_halfmmc</th>\n",
" <th>sharpe_1mmc</th>\n",
" <th>sharpe_2mmc</th>\n",
" <th>sortino_corr</th>\n",
" <th>sortino_halfmmc</th>\n",
" <th>sortino_1mmc</th>\n",
" <th>sortino_2mmc</th>\n",
" <th>rows</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>degerhan</th>\n",
" <td>1.712360</td>\n",
" <td>2.158780</td>\n",
" <td>2.598229</td>\n",
" <td>3.457280</td>\n",
" <td>2.615735</td>\n",
" <td>2.687938</td>\n",
" <td>2.718909</td>\n",
" <td>2.735669</td>\n",
" <td>4.291405</td>\n",
" <td>4.741018</td>\n",
" <td>4.950057</td>\n",
" <td>4.995605</td>\n",
" <td>59.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>hiryuu</th>\n",
" <td>1.603670</td>\n",
" <td>2.060966</td>\n",
" <td>2.512689</td>\n",
" <td>3.400038</td>\n",
" <td>2.753744</td>\n",
" <td>2.619258</td>\n",
" <td>2.515279</td>\n",
" <td>2.374324</td>\n",
" <td>4.784889</td>\n",
" <td>4.454316</td>\n",
" <td>4.170092</td>\n",
" <td>3.822496</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>era__mix__2000</th>\n",
" <td>1.812259</td>\n",
" <td>2.224693</td>\n",
" <td>2.630952</td>\n",
" <td>3.425525</td>\n",
" <td>3.069789</td>\n",
" <td>3.011930</td>\n",
" <td>2.926086</td>\n",
" <td>2.780924</td>\n",
" <td>4.068755</td>\n",
" <td>3.911257</td>\n",
" <td>3.721287</td>\n",
" <td>3.433898</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>arbitrage</th>\n",
" <td>1.439797</td>\n",
" <td>1.556383</td>\n",
" <td>1.672687</td>\n",
" <td>1.905020</td>\n",
" <td>2.670219</td>\n",
" <td>2.565972</td>\n",
" <td>2.407299</td>\n",
" <td>2.088494</td>\n",
" <td>4.629703</td>\n",
" <td>4.325707</td>\n",
" <td>3.774225</td>\n",
" <td>2.990578</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mdo</th>\n",
" <td>1.343288</td>\n",
" <td>1.458944</td>\n",
" <td>1.570011</td>\n",
" <td>1.778377</td>\n",
" <td>2.573783</td>\n",
" <td>2.518394</td>\n",
" <td>2.375061</td>\n",
" <td>2.053527</td>\n",
" <td>4.369704</td>\n",
" <td>4.171902</td>\n",
" <td>3.709542</td>\n",
" <td>2.961006</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>sorios</th>\n",
" <td>1.507010</td>\n",
" <td>1.636998</td>\n",
" <td>1.763865</td>\n",
" <td>2.008631</td>\n",
" <td>2.943613</td>\n",
" <td>2.685931</td>\n",
" <td>2.369338</td>\n",
" <td>1.888087</td>\n",
" <td>3.980116</td>\n",
" <td>3.656240</td>\n",
" <td>3.322387</td>\n",
" <td>2.703564</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>themicon</th>\n",
" <td>1.216351</td>\n",
" <td>1.301826</td>\n",
" <td>1.389449</td>\n",
" <td>1.568699</td>\n",
" <td>2.266811</td>\n",
" <td>2.093298</td>\n",
" <td>1.910954</td>\n",
" <td>1.627119</td>\n",
" <td>3.609768</td>\n",
" <td>3.333851</td>\n",
" <td>3.041291</td>\n",
" <td>2.654933</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>integration_test</th>\n",
" <td>1.402745</td>\n",
" <td>1.377893</td>\n",
" <td>1.352570</td>\n",
" <td>1.300451</td>\n",
" <td>1.980763</td>\n",
" <td>1.763217</td>\n",
" <td>1.565455</td>\n",
" <td>1.238449</td>\n",
" <td>3.190071</td>\n",
" <td>2.731549</td>\n",
" <td>2.305267</td>\n",
" <td>1.733450</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" annual_return_corr ... rows\n",
"degerhan 1.712360 ... 59.0\n",
"hiryuu 1.603670 ... 130.0\n",
"era__mix__2000 1.812259 ... 130.0\n",
"arbitrage 1.439797 ... 130.0\n",
"mdo 1.343288 ... 130.0\n",
"sorios 1.507010 ... 130.0\n",
"themicon 1.216351 ... 130.0\n",
"integration_test 1.402745 ... 130.0\n",
"\n",
"[8 rows x 13 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 39
}
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment