Example of timeout handling for the SRBench Competition (https://cavalab.org/srbench/competition-2022/)

timeout_handling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b2573b03",
   "metadata": {},
   "source": [
    "# example of timeout handling for SRBench Competition\n",
    "\n",
    "- refer to https://cavalab.org/srbench/competition-guide/ for up-to-date guidelines. \n",
    "\n",
    "**Note**: The preferred approach is that participants design their algorithms to terminate comfortably in less than an hour for 1000x100 datasets, and less than 10 hours for 10000x100 datasets.**\n",
    "Participants are also encouraged to include a `max_time` parameter and set it appropriately. \n",
    "\n",
    "Instead of managing runtime internally, participants may choose to handle the `SIGALRM` signal or the `TimeOutException` sent from the evaluation routine.\n",
    "However, *proceed with caution*: the exception handling runs the risk of taking an unexpected amount of time and leading to the job being killed. \n",
    "These examples are for illustration purposes only and should be independently verified for compatibility with user's code submissions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69608a69",
   "metadata": {},
   "source": [
    "## evaluation\n",
    "This is an example of how the TimeOutException is raised in model fitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c020108e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import signal\n",
    "import sys\n",
    "import time\n",
    "\n",
    "##########\n",
    "class TimeOutException(Exception):\n",
    "    pass\n",
    "# model evaluation:\n",
    "MAXTIME=2\n",
    "\n",
    "def alarm_handler(signum, frame):    \n",
    "    print(f\"raising TimeOutException\")\n",
    "    raise TimeOutException\n",
    "\n",
    "def evaluate(est):\n",
    "    \"\"\"model evaluation\"\"\"\n",
    "    # alarm that sends SIGALRM\n",
    "    signal.signal(signal.SIGALRM, alarm_handler)\n",
    "    signal.alarm(MAXTIME)\n",
    "    try:\n",
    "        est.fit() \n",
    "    except TimeOutException:\n",
    "        print('evaluate timeout')\n",
    "    print('estimator value:',est.predict())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "848e177d",
   "metadata": {},
   "source": [
    "# handle exception"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1b961494",
   "metadata": {},
   "outputs": [],
   "source": [
    "##########\n",
    "# submission codebase\n",
    "\n",
    "class HandleExceptionAlg:\n",
    "    def __init__(self):\n",
    "        self.value = 'initial value' \n",
    "        \n",
    "    def fit(self):\n",
    "        try:\n",
    "            while True:\n",
    "                print('fitting...')\n",
    "                time.sleep(1)\n",
    "        except TimeOutException as e:\n",
    "            print('TimeOutException raised')\n",
    "            self.value = 'external timeout'\n",
    "            return self\n",
    "        \n",
    "        return self \n",
    "    \n",
    "    def predict(self):\n",
    "        return self.value\n",
    "    \n",
    "##########"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0e76cf5",
   "metadata": {},
   "source": [
    "# handle SIGALRM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e7602542",
   "metadata": {},
   "outputs": [],
   "source": [
    "##########\n",
    "# submission codebase\n",
    "\n",
    "class InternalTimeOutException(Exception):\n",
    "    pass\n",
    "\n",
    "class HandleSignalAlg:\n",
    "    def __init__(self):\n",
    "        self.value = 'initial value' \n",
    "        \n",
    "    def alarm_handler(self,signum, frame):    \n",
    "        print(f\"raising InternalTimeOutException\")\n",
    "        raise InternalTimeOutException\n",
    "    \n",
    "    def fit(self):\n",
    "        # define an internal signal handler\n",
    "        signal.signal(signal.SIGALRM, self.alarm_handler)\n",
    "        try:\n",
    "            while True:\n",
    "                print('fitting...')\n",
    "                time.sleep(1)\n",
    "        except InternalTimeOutException as e:\n",
    "            print('InternalTimeOutException raised')\n",
    "            self.value = 'internal timeout'\n",
    "            return self\n",
    "        \n",
    "        return self \n",
    "    \n",
    "    def predict(self):\n",
    "        return self.value\n",
    "##########"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4032d2c",
   "metadata": {},
   "source": [
    "# No timeout handling "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b7d649df",
   "metadata": {},
   "outputs": [],
   "source": [
    "##########\n",
    "# submission codebase\n",
    "\n",
    "class NoTimeoutAlg:\n",
    "    def __init__(self):\n",
    "        self.value = 'initial value' \n",
    "        \n",
    "    def fit(self):\n",
    "        while True:\n",
    "            print('fitting...')\n",
    "            time.sleep(1)\n",
    "        \n",
    "        return self \n",
    "    \n",
    "    def predict(self):\n",
    "        return self.value\n",
    "    \n",
    "##########"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a3dfe65",
   "metadata": {},
   "source": [
    "# comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "06f9c208",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##########\n",
      "Exception Handling\n",
      "fitting...\n",
      "fitting...\n",
      "raising TimeOutException\n",
      "TimeOutException raised\n",
      "estimator value: external timeout\n",
      "##########\n",
      "Signal Handling\n",
      "fitting...\n",
      "fitting...\n",
      "raising InternalTimeOutException\n",
      "InternalTimeOutException raised\n",
      "estimator value: internal timeout\n",
      "##########\n",
      "No Handling\n",
      "fitting...\n",
      "fitting...\n",
      "raising TimeOutException\n",
      "evaluate timeout\n",
      "estimator value: initial value\n"
     ]
    }
   ],
   "source": [
    "    \n",
    "# Signal and Timeout Handling \n",
    "print('##########')\n",
    "print('Exception Handling')\n",
    "est = HandleExceptionAlg()\n",
    "evaluate(est)\n",
    "print('##########')\n",
    "print('Signal Handling')\n",
    "est = HandleSignalAlg()\n",
    "evaluate(est)\n",
    "print('##########')\n",
    "print('No Handling')\n",
    "est = NoTimeoutAlg()\n",
    "evaluate(est)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}