Error propagation

error_propagation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the model predicts a sequence of length $n$, denoted as\n",
    "\n",
    "$$\\mathbf{\\hat{x}_n} = (x̂_{t-1}, x̂_{t-2}, \\dots, x̂_{t-n})$$\n",
    "\n",
    "Each predicted token $\\hat{x}_t$ is computed based on the previous predicted tokens:\n",
    "\n",
    "$$\\hat{x}_t = \\varphi_1\\hat{x}_{t-1} + \\varphi_2\\hat{x}_{t-2} + \\dots + \\varphi_p\\hat{x}_{t-n} + b = \\sum_{i=1}^n \\varphi_i\\hat{x}_{t-i} + b$$\n",
    "\n",
    "Suppose the model predicts a token $\\hat{x}_{t-i}$ with an error $\\epsilon_{t-i}$.  The next token will be predicted based on the erroneous token $\\hat{x}_{t-i}$, leading to a new error. This process continues, and the errors accumulate, leading to a significant deviation from the true sequence.\n",
    "We can define this error propagation as:\n",
    "\n",
    "$$\\hat{x}_t = (\\varphi_1\\hat{x}_{t-1} + \\epsilon_{t-1}) + (\\varphi_2\\hat{x}_{t-2} + \\epsilon_{t-2}) + \\dots + (\\varphi_p\\hat{x}_{t-n} + \\epsilon_{t-n}) + b$$\n",
    "\n",
    "Adding the growing error as a separated sum helps to visualize the harmful impact of the error propagation term:\n",
    "\n",
    "$$\\hat{x}_t = \\sum_{i=1}^n \\varphi_i\\hat{x}_{t-i} + \\sum_{i=1}^n \\epsilon_{t-i} + b$$\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}