notebook12.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0a3f14e3",
   "metadata": {},
   "source": [
    "## Optimal Stopping\n",
    "\\begin{equation}\n",
    "\t\tp_{n} = \\max\\{f_{n}, \\mathbb{E}_{Q}[\\alpha p_{n+1}\\vert \\mathcal{F}_{n}]\\}\n",
    "\\end{equation}\n",
    "\n",
    "Approximate $\\mathbb{E}_{Q}[\\alpha p_{n+1}\\vert \\mathcal{F}_{n}]$ with $c_{n,\\theta}(X_1,\\dots,X_n)$\n",
    "such that at each time $n$\n",
    "\\begin{equation}\n",
    "\t\t\\min_{\\theta} \\mathbb{E}_{Q} \\Big[\\big(c_{n,\\theta}(X_1,\\dots,X_n) - \\alpha p_{n+1}\\big)^{2}\\Big]\n",
    "\t\\end{equation}\n",
    "    \n",
    "\\begin{equation}\n",
    "\t\t\\min_{\\theta} \\sum_{i=1}^{m} \\Big[\\big(c_{n,\\theta}(x^i_1,\\dots,x^i_n) - \\alpha p^{i}_{n+1}\\big)^{2}\\Big]\n",
    "\t\\end{equation}\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccde6b59",
   "metadata": {},
   "source": [
    "\n",
    "### Backward induction\n",
    "$$\n",
    "p^i_n = g(x^i_n)\\cdot s_{n,\\theta}(x^i_n) + \\alpha p^i_{n+1}\\cdot\\big(1-s_{n,\\theta}(x^i_n)\\big)\n",
    "$$\n",
    "- $p^i_n$: values\n",
    "- $g(x^i_n)$ : immediate_exercise_value\n",
    "- $\\alpha p^i_{n+1}$: discounted_next_values\n",
    "- $s_{n,\\theta}(\\cdot)$: stop(...)\n",
    "- $s_{n,\\theta}(x^i_n)$: stopping_rule\n",
    "\n",
    "#### DOS (Deep Optimal Stopping):\n",
    "$$s_{n,\\theta}(x^i_n) = f_{\\theta}(x^{i}_n) \\in [0,1]$$\n",
    "- $f_{\\theta}(\\cdot)$: a neural network\n",
    "\n",
    "#### LSM (Least Squares Monte Carlo):\n",
    "$$\n",
    "s_{n,\\theta}(x^i_n) = \\mathbb{1}_{\\{g(x^i_n) \\geq c_{n,\\theta}(x^i_n)\\}}\n",
    "$$\n",
    "- $c_{n,\\theta}(x^i_n)$: continuation_value\n",
    "- $c_{n,\\theta}(\\cdot)$: calculate_continuation_value(...)\n",
    "$$\n",
    "c_{n,\\theta}(\\cdot) = \\theta_n^{T}\\phi(\\cdot) = \\sum_{j=1}^{K}\\theta_{n,j} \\phi_{j}(\\cdot)\n",
    "$$\n",
    "\n",
    "- **LSM** (Least Square Monte Carlo): $\\phi_j$ be polynomials\n",
    "- **NLSM** (Neural Least Square Monte Carlo): $\\phi_j$ be neural networks\n",
    "- **RLSM** (Randomized Least Square Monte Carlo): $\\phi_j$ be randomized neural networks\n",
    "- **RRLSM** (Randomized Recurrent Least Square Monte Carlo): $\\phi_j$ be randomized neural networks taking one more hidden input $h_n$\n",
    "\\begin{equation}\n",
    "\\left\\{\\begin{aligned}\n",
    "    h_n &= \\sigma(A_x x_n + A_h h_{n-1} + b)\\\\\n",
    "    c_{n,\\theta}(h_n) &= A_n^{T}h_n + b_n = \\theta_n^{T}\\phi(x_n,h_{n-1})\n",
    "\\end{aligned}\n",
    "\\right.\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "496d7d48",
   "metadata": {},
   "source": [
    "### Reinforcement Learning \n",
    "$$\n",
    "p^i_n = \\max \\{ g(x^i_n), c_{n,\\theta}(x^i_1,\\dots,x^i_n) \\}\n",
    "$$\n",
    "\n",
    "\\begin{equation}\n",
    "\t\t\\min_{\\theta} \\mathbb{E}_{Q} \\Big[\\sum_{n=1}^{N}\\big(c_{n,\\theta}(X_1,\\dots,X_n) - \\alpha p_{n+1}\\big)^{2}\\Big]\n",
    "\t\\end{equation}\n",
    "    \n",
    "\\begin{equation}\n",
    "\t\t\\min_{\\theta}  \\Big[\\sum_{i=1}^{m}\\sum_{n=1}^{N}\\big(c_{n,\\theta}(x^i_1,\\dots,x^i_n) - \\alpha p^{i}_{n+1}\\big)^{2}\\Big]\n",
    "\t\\end{equation}\n",
    "\n",
    "    \n",
    "#### Fitted Q-Iteration\n",
    "\n",
    "$$\n",
    "c_{n,\\theta}(\\cdot) = \\theta^{T}\\phi(\\cdot, n) = \\sum_{j=1}^{K}\\theta_{j} \\phi_{j}(\\cdot,n)\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "- $\\phi_{j}$: self.bf\n",
    "\n",
    "\n",
    "- **FQI** (fitted Q-Iteration): $\\phi_j$ \n",
    "be polynomials\n",
    "- **RFQI** (randomized fitted Q-Iteration): $\\phi_j$ be randomized neural networks\n",
    "\n",
    "$$\n",
    "\\phi_{j}(x,n) = (\\sigma(A\\tilde{x} + b) , 1)\n",
    "$$\n",
    "where\n",
    "$$\n",
    "\\tilde{x} = (x, n, N-n) \\in \\mathbb{R}^{d+2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1033a887",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:deephedge] *",
   "language": "python",
   "name": "conda-env-deephedge-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}