Vision2020-ex02.ipynb

vision2020-ex02.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Vision2020-ex02.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMbTQlPclnGMk41ibDnjLZW",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/takatakamanbou/302ec26a3a06028092a7f58c41774bd2/vision2020-ex02.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dZcF12ojSrJ",
        "colab_type": "text"
      },
      "source": [
        "# Vision2020 演習その2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SrIIY0RtpBYO",
        "colab_type": "text"
      },
      "source": [
        "課題の期限や提出の方法などについては，Visionチーム内に書いてます．"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZSx7Ng2FkdUB",
        "colab_type": "text"
      },
      "source": [
        "## 準備"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BXtjmCk3jNaH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zd9VnEQRjtgq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#####  最小二乗法の正規方程式を解く関数\n",
        "### solving the linear least squares problem (1)\n",
        "#\n",
        "# X.shape is assumed to be (N, D+1)\n",
        "# y.shape is assumed to be (N,)\n",
        "#\n",
        "def solve(X, y):\n",
        "\n",
        "    A = np.dot(X.T, X)\n",
        "    b = np.dot(X.T, y)\n",
        "\n",
        "    # x is the solution of the equation Ax = b\n",
        "    x = np.linalg.solve(A, b)\n",
        "\n",
        "    return x\n",
        "\n",
        "\n",
        "### solving the linear least squares problem (2)\n",
        "#\n",
        "def solve2(X, y):\n",
        "\n",
        "    A = np.dot(X.T, X)\n",
        "    b = np.dot(X.T, y)\n",
        "\n",
        "    # rv[0] is the solution minimizing ||Ax - b||^2\n",
        "    rv = np.linalg.lstsq(A, b)\n",
        "\n",
        "    return rv[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UG8GQ8uFnzld",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### 1, x, x^2, x^3, ..., X^D をならべたデータ行列（N x (D+1)）をつくる\n",
        "#     授業の説明のものとは転置の関係になっていることに注意\n",
        "#\n",
        "def makeDataMatrix(x, D):\n",
        "\n",
        "    N = x.shape[0]\n",
        "    X = np.zeros((N, D+1))\n",
        "    X[:, 0] = 1\n",
        "    for i in range(1, D+1):\n",
        "        X[:, i] = x**i\n",
        "\n",
        "    return X"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M6wk0zlylJPt",
        "colab_type": "text"
      },
      "source": [
        "## 課題A\n",
        "\n",
        "以下のプログラムの中の関数 gendat は，ある3時間数にノイズをのせたデータを生成するものである．どんな3次関数を使っているか，プログラムを読み解きなさい．また，どんなノイズをのせているか，関数名などを頼りに調べなさい．\n",
        "\n",
        "提出するもの： 実行結果のグラフと，上記について調べた結果"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RzU58z-6lSMX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def gendat(n, seed = 0, sigma = 0.0):\n",
        "\n",
        "    x = np.linspace(-1.5, 2.5, num = n)\n",
        "    y = x*x*x - x*x - x + 1.0\n",
        "    \n",
        "    np.random.seed(seed)\n",
        "    y += sigma * np.random.randn(n)\n",
        "\n",
        "    X = np.vstack((x, y)).T\n",
        "\n",
        "    return X"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_S-ebk07lWth",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### 乱数の種(seed)を指定．適当に変えれば結果も変わる\n",
        "seed = 0\n",
        "\n",
        "data_noisy = gendat(30, seed=seed, sigma=0.5)\n",
        "data_true  = gendat(1000, seed=seed, sigma=0)\n",
        "\n",
        "fig, ax = plt.subplots(facecolor=\"white\", figsize=(8, 6))\n",
        "ax.set_xlim(-2, 3)\n",
        "ax.set_ylim(-5, 10)\n",
        "ax.plot(data_true[:, 0], data_true[:, 1], color=\"blue\", label=\"true function\")\n",
        "ax.scatter(data_noisy[:, 0], data_noisy[:, 1], color=\"red\", label=\"noisy data\")\n",
        "ax.legend()\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bVig4szUoayG",
        "colab_type": "text"
      },
      "source": [
        "## 課題B\n",
        "\n",
        "以下のプログラムは，課題Aのプログラムを用いて生成したデータに D 次多項式をあてはめるものである．次のことをやりなさい．\n",
        "\n",
        "1. 実行すると表示される mean squared error (MSE) が何を表しているか，プログラムを読解しなさい．\n",
        "1. D を 1 から 20 まで変えたとき MSE の値やグラフがどのように変化するか観察しなさい．\n",
        "1. 横軸を D，縦軸を学習データおよびテストデータの MSE としてグラフを描きなさい（Colab上でやってもよいし，他の手段を使ってもよい）．D が小さいときの MSE は他に比べてとても大きくなるので，縦軸の範囲を狭めてグラフを描き，D がいくつのときにテストデータの MSE が最小となっているか読み取れるようなものにすること．\n",
        "\n",
        "提出するもの：上記の最後のステップで描いたグラフと，それに対する説明・考察．\n",
        "\n",
        "**注意**\n",
        "この課題と次の課題では，テストデータを使って次元数 $D$ や正則化パラメータ $\\alpha$ の最適な値を決めようとしてます．授業で説明したように，未知のはずのテストデータを使うのは本当はいんちきです．\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cUjBtohNoeB_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### generating the data\n",
        "\n",
        "ndata = 30\n",
        "sigma = 0.5\n",
        "\n",
        "# learning (training) data\n",
        "datL = gendat(ndata, seed=0, sigma=sigma)\n",
        "\n",
        "# test data\n",
        "datT = gendat(ndata, seed=1, sigma=sigma)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JguJE_agosud",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### fitting a D-th degree polynomial by least squares method\n",
        "\n",
        "D = 1\n",
        "XL = makeDataMatrix(datL[:, 0], D)\n",
        "yL = datL[:, 1]\n",
        "print('# X.shape =', XL.shape)\n",
        "print('# y.shape =', yL.shape)\n",
        "w = solve(XL, yL)\n",
        "print('# estimated coefficients = ', w)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H97ob5-co5OE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### mean squared error for the learning data\n",
        "\n",
        "yL_est = np.dot(XL, w)\n",
        "msqeL = np.mean((yL - yL_est)**2)\n",
        "print('# mean squared error (L) = {:.3f}'.format(msqeL))\n",
        "\n",
        "\n",
        "##### mean squared error for the test data\n",
        "\n",
        "XT = makeDataMatrix(datT[:, 0], D)\n",
        "yT = datT[:, 1]\n",
        "yT_est = np.dot(XT, w)\n",
        "msqeT = np.mean((yT - yT_est)**2)\n",
        "print('# mean squared error (T) = {:.3f}'.format(msqeT))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4pVjrB5dpkGG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### making the true (noiseless) data & more...\n",
        "\n",
        "datTrue = gendat(1000)\n",
        "X = makeDataMatrix(datTrue[:, 0], D)\n",
        "y = datTrue[:, 1]\n",
        "y_est = np.dot(X, w)\n",
        "\n",
        "\n",
        "##### plotting the results\n",
        "\n",
        "fig, ax = plt.subplots(facecolor=\"white\", figsize=(8, 6))\n",
        "ax.set_xlim(-2, 3)\n",
        "ax.set_ylim(-5, 10)\n",
        "ax.plot(datTrue[:, 0], datTrue[:, 1], color=\"blue\", label=\"true function\")\n",
        "ax.scatter(datL[:, 0], datL[:, 1], color=\"red\", label=\"noisy data\")\n",
        "ax.plot(datTrue[:, 0], y_est, color=\"green\", label=\"estimated curve\")\n",
        "ax.legend()\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "95N9Ray9sr7z",
        "colab_type": "text"
      },
      "source": [
        "## 課題C\n",
        "\n",
        "上記の問題を，$L_2$ 正則化を用いて解いてみよう．次のことをやりなさい．\n",
        "\n",
        "1. 関数 solve をもとにして，正則化を考慮した関数 solve3 を作りなさい．solve3 は X, y に加えて alpha という名前の引数で正則化項の重み $\\alpha$ の値を受け取るようにしよう．パラメータのうち定数の係数（バイアス項）には正則化を効かせないようにすること．\n",
        "1. D を 20 として，$\\alpha$ を様々に変えて実験を行い，横軸を $\\alpha$，縦軸を学習データおよびテストデータに対する誤差としてグラフを描いてみよう．\n",
        "    - $\\alpha$ は $0$ 以上の任意の値をとるので，いくつか適当な値を選んで実行 → 結果を見て範囲を変える／狭める，というステップを何回か繰り返すのがよい．とりあえず今回は [0, 10] からスタートしたらよい．\n",
        "    - 余裕があれば，1つの $\\alpha$ に対してノイズ生成の乱数の種を何通りか変えて，それらの誤差の平均でグラフを描くとなおよい\n",
        "    - さらに余裕があれば，学習データの数を増減して同じ実験をやると傾向が変わるかどうか調べるとよい．\n",
        "    - もっと余裕があれば，真の値に対する誤差も描いてみるとよい．\n",
        "\n",
        "提出するもの：関数 solve3 の定義，描いたグラフとそれに対する説明・考察．"
      ]
    }
  ]
}