Skip to content

Instantly share code, notes, and snippets.

@jonasrauber

jonasrauber/scipy_lbfgs_mwe.ipynb

Last active May 15, 2016 08:14
Show Gist options
  • Save jonasrauber/41da8b80342c437ecc3e41c743aff98f to your computer and use it in GitHub Desktop.
Save jonasrauber/41da8b80342c437ecc3e41c743aff98f to your computer and use it in GitHub Desktop.
Download ZIP
scipy_lbfgs_mwe.ipynb
Display the source blob
Display the rendered blob
Raw
scipy_lbfgs_mwe.ipynb
{
"cells": [
{
"metadata": {},
"cell_type": "markdown",
"source": "**See this bug report for details: https://github.com/scipy/scipy/issues/6162**"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "# Minimal Working Example\n\nscipy.optimize.fmin_l_bfgs_b returns inconsistent results and can return arbitrarily bad results even when better ones have been seen before"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:21.369800",
"end_time": "2016-05-15T08:03:21.618389"
},
"collapsed": true,
"trusted": true
},
"cell_type": "code",
"source": "from scipy.optimize import fmin_l_bfgs_b\nfrom scipy.optimize import rosen\nfrom scipy.optimize import rosen_der\nimport numpy as np\nimport scipy\nimport sys",
"execution_count": 1,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:22.153469",
"end_time": "2016-05-15T08:03:22.158030"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "print(sys.version)",
"execution_count": 2,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "3.4.3 (default, Oct 14 2015, 20:28:29) \n[GCC 4.8.4]\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:22.639786",
"end_time": "2016-05-15T08:03:22.643491"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "print(scipy.__version__)",
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "0.18.0.dev0+02d3ab0\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:23.407743",
"end_time": "2016-05-15T08:03:23.411502"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "print(np.__version__)",
"execution_count": 4,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": "1.12.0.dev0+3509704\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:24.825667",
"end_time": "2016-05-15T08:03:24.831660"
},
"collapsed": true,
"trusted": true
},
"cell_type": "code",
"source": "def verbose_rosen(x):\n global function_calls\n function_calls += 1\n f = rosen(x)\n print(' call {:5d} returned: {:20.2f}'.format(function_calls, f))\n return f",
"execution_count": 5,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:25.251111",
"end_time": "2016-05-15T08:03:25.256239"
},
"collapsed": true,
"trusted": true
},
"cell_type": "code",
"source": "def verbose_iteration(x):\n global iteration\n iteration += 1\n f = rosen(x)\n print('f after iteration {:5d}: {:20.2f} !'.format(iteration, f))",
"execution_count": 6,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Problem 1: Inconsistent Results"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Whenever the optimization is stopped because of maxfun, fmin_l_bfgs_b returns inconsistent results, i.e. f(xmin) ≠ fmin!"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:26.368086",
"end_time": "2016-05-15T08:03:26.374986"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "function_calls = 0\niteration = 0\nx0 = 1000 * np.ones(7)\nxmin, fmin, d = fmin_l_bfgs_b(verbose_rosen, x0, fprime=rosen_der, callback=verbose_iteration, maxfun=10)",
"execution_count": 7,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": " call 1 returned: 598800605988006.00\n call 2 returned: 597822796988103.88\n call 3 returned: 593923530746706.62\n call 4 returned: 578516888022267.38\n call 5 returned: 519867673288622.69\nf after iteration 1: 519867673288622.69 !\n call 6 returned: 110073536397771.98\nf after iteration 2: 110073536397771.98 !\n call 7 returned: 41749330662696.73\nf after iteration 3: 41749330662696.73 !\n call 8 returned: 12747616470460.33\nf after iteration 4: 12747616470460.33 !\n call 9 returned: 4233695601672.95\nf after iteration 5: 4233695601672.95 !\n call 10 returned: 1360528312225.51\nf after iteration 6: 1360528312225.51 !\n call 11 returned: 442244013728.31\nf after iteration 7: 442244013728.31 !\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:26.768328",
"end_time": "2016-05-15T08:03:26.782911"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "xmin",
"execution_count": 8,
"outputs": [
{
"execution_count": 8,
"output_type": "execute_result",
"data": {
"text/plain": "array([ 124.86585533, 125.01193097, 125.01223917, 125.01224328,\n 125.01516302, 126.82982827, 1001.34208776])"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:27.247458",
"end_time": "2016-05-15T08:03:27.251853"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "rosen(xmin)",
"execution_count": 9,
"outputs": [
{
"execution_count": 9,
"output_type": "execute_result",
"data": {
"text/plain": "142809546529.3916"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:27.689428",
"end_time": "2016-05-15T08:03:27.693392"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "fmin",
"execution_count": 10,
"outputs": [
{
"execution_count": 10,
"output_type": "execute_result",
"data": {
"text/plain": "442244013728.3136"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "**rosen(xmin) and fmin should be the same**"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### Compare this to limiting the number of iterations instead of function evaluations"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:29.657534",
"end_time": "2016-05-15T08:03:29.664958"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "function_calls = 0\niteration = 0\nx0 = 1000 * np.ones(7)\nxmin, fmin, d = fmin_l_bfgs_b(verbose_rosen, x0, fprime=rosen_der, callback=verbose_iteration, maxiter=6)",
"execution_count": 11,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": " call 1 returned: 598800605988006.00\n call 2 returned: 597822796988103.88\n call 3 returned: 593923530746706.62\n call 4 returned: 578516888022267.38\n call 5 returned: 519867673288622.69\nf after iteration 1: 519867673288622.69 !\n call 6 returned: 110073536397771.98\nf after iteration 2: 110073536397771.98 !\n call 7 returned: 41749330662696.73\nf after iteration 3: 41749330662696.73 !\n call 8 returned: 12747616470460.33\nf after iteration 4: 12747616470460.33 !\n call 9 returned: 4233695601672.95\nf after iteration 5: 4233695601672.95 !\n call 10 returned: 1360528312225.51\nf after iteration 6: 1360528312225.51 !\n call 11 returned: 442244013728.31\nf after iteration 7: 442244013728.31 !\n call 12 returned: 142809546529.39\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:30.044531",
"end_time": "2016-05-15T08:03:30.050224"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "xmin",
"execution_count": 12,
"outputs": [
{
"execution_count": 12,
"output_type": "execute_result",
"data": {
"text/plain": "array([ 124.86585533, 125.01193097, 125.01223917, 125.01224328,\n 125.01516302, 126.82982827, 1001.34208776])"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:30.598319",
"end_time": "2016-05-15T08:03:30.602528"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "rosen(xmin)",
"execution_count": 13,
"outputs": [
{
"execution_count": 13,
"output_type": "execute_result",
"data": {
"text/plain": "142809546529.3916"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:30.981833",
"end_time": "2016-05-15T08:03:30.986725"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "fmin",
"execution_count": 14,
"outputs": [
{
"execution_count": 14,
"output_type": "execute_result",
"data": {
"text/plain": "142809546529.3916"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "**maxiter works fine**"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Problem 2: Can return arbitrarily bad function values"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Because fmin ≠ f(xmin) as shown before, we can look at two cases: 1) fmin bad, 2) xmin bad. Note: in contrast to the first problem, this is only a problem for specific function-dependent choices of maxfun."
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### bad fmin"
},
{
"metadata": {
"scrolled": false,
"ExecuteTime": {
"start_time": "2016-05-15T08:03:51.615905",
"end_time": "2016-05-15T08:03:51.637530"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "function_calls = 0\niteration = 0\nx0 = 1000 * np.ones(7)\nxmin, fmin, d = fmin_l_bfgs_b(verbose_rosen, x0, fprime=rosen_der, callback=verbose_iteration, maxfun=115)",
"execution_count": 15,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": " call 1 returned: 598800605988006.00\n call 2 returned: 597822796988103.88\n call 3 returned: 593923530746706.62\n call 4 returned: 578516888022267.38\n call 5 returned: 519867673288622.69\nf after iteration 1: 519867673288622.69 !\n call 6 returned: 110073536397771.98\nf after iteration 2: 110073536397771.98 !\n call 7 returned: 41749330662696.73\nf after iteration 3: 41749330662696.73 !\n call 8 returned: 12747616470460.33\nf after iteration 4: 12747616470460.33 !\n call 9 returned: 4233695601672.95\nf after iteration 5: 4233695601672.95 !\n call 10 returned: 1360528312225.51\nf after iteration 6: 1360528312225.51 !\n call 11 returned: 442244013728.31\nf after iteration 7: 442244013728.31 !\n call 12 returned: 142809546529.39\nf after iteration 8: 142809546529.39 !\n call 13 returned: 46062613345.75\nf after iteration 9: 46062613345.75 !\n call 14 returned: 14773724518.19\nf after iteration 10: 14773724518.19 !\n call 15 returned: 4703898661.48\nf after iteration 11: 4703898661.48 !\n call 16 returned: 1482180832.60\nf after iteration 12: 1482180832.60 !\n call 17 returned: 462131193.72\nf after iteration 13: 462131193.72 !\n call 18 returned: 143457174.33\nf after iteration 14: 143457174.33 !\n call 19 returned: 44753186.68\nf after iteration 15: 44753186.68 !\n call 20 returned: 14070681.95\nf after iteration 16: 14070681.95 !\n call 21 returned: 4452497.88\nf after iteration 17: 4452497.88 !\n call 22 returned: 1974293.55\nf after iteration 18: 1974293.55 !\n call 23 returned: 378976.20\nf after iteration 19: 378976.20 !\n call 24 returned: 152593.95\nf after iteration 20: 152593.95 !\n call 25 returned: 62902.97\nf after iteration 21: 62902.97 !\n call 26 returned: 35402.40\nf after iteration 22: 35402.40 !\n call 27 returned: 9061.05\nf after iteration 23: 9061.05 !\n call 28 returned: 3482.27\nf after iteration 24: 3482.27 !\n call 29 returned: 1887.74\nf after iteration 25: 1887.74 !\n call 30 returned: 1307.94\nf after iteration 26: 1307.94 !\n call 31 returned: 1044.36\nf after iteration 27: 1044.36 !\n call 32 returned: 982.37\nf after iteration 28: 982.37 !\n call 33 returned: 970.24\nf after iteration 29: 970.24 !\n call 34 returned: 966.61\nf after iteration 30: 966.61 !\n call 35 returned: 965.37\nf after iteration 31: 965.37 !\n call 36 returned: 964.80\nf after iteration 32: 964.80 !\n call 37 returned: 988.88\n call 38 returned: 964.74\nf after iteration 33: 964.74 !\n call 39 returned: 964.61\nf after iteration 34: 964.61 !\n call 40 returned: 964.54\nf after iteration 35: 964.54 !\n call 41 returned: 964.50\nf after iteration 36: 964.50 !\n call 42 returned: 964.42\nf after iteration 37: 964.42 !\n call 43 returned: 964.15\nf after iteration 38: 964.15 !\n call 44 returned: 964.20\n call 45 returned: 964.04\nf after iteration 39: 964.04 !\n call 46 returned: 963.97\nf after iteration 40: 963.97 !\n call 47 returned: 963.96\nf after iteration 41: 963.96 !\n call 48 returned: 963.96\nf after iteration 42: 963.96 !\n call 49 returned: 963.96\nf after iteration 43: 963.96 !\n call 50 returned: 963.96\nf after iteration 44: 963.96 !\n call 51 returned: 963.96\nf after iteration 45: 963.96 !\n call 52 returned: 963.96\nf after iteration 46: 963.96 !\n call 53 returned: 963.98\n call 54 returned: 963.96\nf after iteration 47: 963.96 !\n call 55 returned: 963.96\nf after iteration 48: 963.96 !\n call 56 returned: 963.96\nf after iteration 49: 963.96 !\n call 57 returned: 963.96\nf after iteration 50: 963.96 !\n call 58 returned: 963.96\nf after iteration 51: 963.96 !\n call 59 returned: 963.96\nf after iteration 52: 963.96 !\n call 60 returned: 963.95\nf after iteration 53: 963.95 !\n call 61 returned: 963.93\nf after iteration 54: 963.93 !\n call 62 returned: 963.92\nf after iteration 55: 963.92 !\n call 63 returned: 963.86\nf after iteration 56: 963.86 !\n call 64 returned: 963.76\nf after iteration 57: 963.76 !\n call 65 returned: 963.52\nf after iteration 58: 963.52 !\n call 66 returned: 963.30\nf after iteration 59: 963.30 !\n call 67 returned: 962.87\nf after iteration 60: 962.87 !\n call 68 returned: 962.57\nf after iteration 61: 962.57 !\n call 69 returned: 962.27\nf after iteration 62: 962.27 !\n call 70 returned: 962.15\n call 71 returned: 961.79\nf after iteration 63: 961.79 !\n call 72 returned: 960.51\nf after iteration 64: 960.51 !\n call 73 returned: 955.57\nf after iteration 65: 955.57 !\n call 74 returned: 950.08\nf after iteration 66: 950.08 !\n call 75 returned: 937.37\nf after iteration 67: 937.37 !\n call 76 returned: 921.56\nf after iteration 68: 921.56 !\n call 77 returned: 902.80\nf after iteration 69: 902.80 !\n call 78 returned: 919.33\n call 79 returned: 896.35\nf after iteration 70: 896.35 !\n call 80 returned: 887.90\nf after iteration 71: 887.90 !\n call 81 returned: 868.63\nf after iteration 72: 868.63 !\n call 82 returned: 851.15\nf after iteration 73: 851.15 !\n call 83 returned: 827.70\nf after iteration 74: 827.70 !\n call 84 returned: 805.88\nf after iteration 75: 805.88 !\n call 85 returned: 802.61\nf after iteration 76: 802.61 !\n call 86 returned: 809.97\n call 87 returned: 797.59\nf after iteration 77: 797.59 !\n call 88 returned: 794.03\nf after iteration 78: 794.03 !\n call 89 returned: 778.43\nf after iteration 79: 778.43 !\n call 90 returned: 768.42\n call 91 returned: 766.10\nf after iteration 80: 766.10 !\n call 92 returned: 762.72\n call 93 returned: 751.91\nf after iteration 81: 751.91 !\n call 94 returned: 735.94\nf after iteration 82: 735.94 !\n call 95 returned: 750.64\n call 96 returned: 731.79\nf after iteration 83: 731.79 !\n call 97 returned: 723.95\nf after iteration 84: 723.95 !\n call 98 returned: 1046.98\n call 99 returned: 716.29\nf after iteration 85: 716.29 !\n call 100 returned: 696.83\nf after iteration 86: 696.83 !\n call 101 returned: 686.71\nf after iteration 87: 686.71 !\n call 102 returned: 676.83\nf after iteration 88: 676.83 !\n call 103 returned: 669.52\nf after iteration 89: 669.52 !\n call 104 returned: 665.31\nf after iteration 90: 665.31 !\n call 105 returned: 679.80\n call 106 returned: 654.77\nf after iteration 91: 654.77 !\n call 107 returned: 644.50\nf after iteration 92: 644.50 !\n call 108 returned: 636.91\n call 109 returned: 634.24\nf after iteration 93: 634.24 !\n call 110 returned: 632.36\n call 111 returned: 626.56\nf after iteration 94: 626.56 !\n call 112 returned: 612.69\nf after iteration 95: 612.69 !\n call 113 returned: 604.85\nf after iteration 96: 604.85 !\n call 114 returned: 594.35\nf after iteration 97: 594.35 !\n call 115 returned: 577.45\nf after iteration 98: 577.45 !\n call 116 returned: 2209.04\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:52.290887",
"end_time": "2016-05-15T08:03:52.296578"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "xmin",
"execution_count": 16,
"outputs": [
{
"execution_count": 16,
"output_type": "execute_result",
"data": {
"text/plain": "array([ 1.08037919, 1.2245496 , 1.43647884, 2.1877428 ,\n 4.92707449, 24.36141377, 593.33069028])"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:53.257390",
"end_time": "2016-05-15T08:03:53.262321"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "rosen(xmin)",
"execution_count": 17,
"outputs": [
{
"execution_count": 17,
"output_type": "execute_result",
"data": {
"text/plain": "570.00286796417458"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:03:53.798155",
"end_time": "2016-05-15T08:03:53.802291"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "fmin",
"execution_count": 18,
"outputs": [
{
"execution_count": 18,
"output_type": "execute_result",
"data": {
"text/plain": "2209.0430396217657"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "**fmin can in principle be arbitrarily bad no matter how good the previously found points already are**"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### bad xmin"
},
{
"metadata": {
"scrolled": false,
"ExecuteTime": {
"start_time": "2016-05-15T08:04:00.556896",
"end_time": "2016-05-15T08:04:00.577820"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "function_calls = 0\niteration = 0\nx0 = 1000 * np.ones(7)\nxmin, fmin, d = fmin_l_bfgs_b(verbose_rosen, x0, fprime=rosen_der, callback=verbose_iteration, maxfun=114)",
"execution_count": 19,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": " call 1 returned: 598800605988006.00\n call 2 returned: 597822796988103.88\n call 3 returned: 593923530746706.62\n call 4 returned: 578516888022267.38\n call 5 returned: 519867673288622.69\nf after iteration 1: 519867673288622.69 !\n call 6 returned: 110073536397771.98\nf after iteration 2: 110073536397771.98 !\n call 7 returned: 41749330662696.73\nf after iteration 3: 41749330662696.73 !\n call 8 returned: 12747616470460.33\nf after iteration 4: 12747616470460.33 !\n call 9 returned: 4233695601672.95\nf after iteration 5: 4233695601672.95 !\n call 10 returned: 1360528312225.51\nf after iteration 6: 1360528312225.51 !\n call 11 returned: 442244013728.31\nf after iteration 7: 442244013728.31 !\n call 12 returned: 142809546529.39\nf after iteration 8: 142809546529.39 !\n call 13 returned: 46062613345.75\nf after iteration 9: 46062613345.75 !\n call 14 returned: 14773724518.19\nf after iteration 10: 14773724518.19 !\n call 15 returned: 4703898661.48\nf after iteration 11: 4703898661.48 !\n call 16 returned: 1482180832.60\nf after iteration 12: 1482180832.60 !\n call 17 returned: 462131193.72\nf after iteration 13: 462131193.72 !\n call 18 returned: 143457174.33\nf after iteration 14: 143457174.33 !\n call 19 returned: 44753186.68\nf after iteration 15: 44753186.68 !\n call 20 returned: 14070681.95\nf after iteration 16: 14070681.95 !\n call 21 returned: 4452497.88\nf after iteration 17: 4452497.88 !\n call 22 returned: 1974293.55\nf after iteration 18: 1974293.55 !\n call 23 returned: 378976.20\nf after iteration 19: 378976.20 !\n call 24 returned: 152593.95\nf after iteration 20: 152593.95 !\n call 25 returned: 62902.97\nf after iteration 21: 62902.97 !\n call 26 returned: 35402.40\nf after iteration 22: 35402.40 !\n call 27 returned: 9061.05\nf after iteration 23: 9061.05 !\n call 28 returned: 3482.27\nf after iteration 24: 3482.27 !\n call 29 returned: 1887.74\nf after iteration 25: 1887.74 !\n call 30 returned: 1307.94\nf after iteration 26: 1307.94 !\n call 31 returned: 1044.36\nf after iteration 27: 1044.36 !\n call 32 returned: 982.37\nf after iteration 28: 982.37 !\n call 33 returned: 970.24\nf after iteration 29: 970.24 !\n call 34 returned: 966.61\nf after iteration 30: 966.61 !\n call 35 returned: 965.37\nf after iteration 31: 965.37 !\n call 36 returned: 964.80\nf after iteration 32: 964.80 !\n call 37 returned: 988.88\n call 38 returned: 964.74\nf after iteration 33: 964.74 !\n call 39 returned: 964.61\nf after iteration 34: 964.61 !\n call 40 returned: 964.54\nf after iteration 35: 964.54 !\n call 41 returned: 964.50\nf after iteration 36: 964.50 !\n call 42 returned: 964.42\nf after iteration 37: 964.42 !\n call 43 returned: 964.15\nf after iteration 38: 964.15 !\n call 44 returned: 964.20\n call 45 returned: 964.04\nf after iteration 39: 964.04 !\n call 46 returned: 963.97\nf after iteration 40: 963.97 !\n call 47 returned: 963.96\nf after iteration 41: 963.96 !\n call 48 returned: 963.96\nf after iteration 42: 963.96 !\n call 49 returned: 963.96\nf after iteration 43: 963.96 !\n call 50 returned: 963.96\nf after iteration 44: 963.96 !\n call 51 returned: 963.96\nf after iteration 45: 963.96 !\n call 52 returned: 963.96\nf after iteration 46: 963.96 !\n call 53 returned: 963.98\n call 54 returned: 963.96\nf after iteration 47: 963.96 !\n call 55 returned: 963.96\nf after iteration 48: 963.96 !\n call 56 returned: 963.96\nf after iteration 49: 963.96 !\n call 57 returned: 963.96\nf after iteration 50: 963.96 !\n call 58 returned: 963.96\nf after iteration 51: 963.96 !\n call 59 returned: 963.96\nf after iteration 52: 963.96 !\n call 60 returned: 963.95\nf after iteration 53: 963.95 !\n call 61 returned: 963.93\nf after iteration 54: 963.93 !\n call 62 returned: 963.92\nf after iteration 55: 963.92 !\n call 63 returned: 963.86\nf after iteration 56: 963.86 !\n call 64 returned: 963.76\nf after iteration 57: 963.76 !\n call 65 returned: 963.52\nf after iteration 58: 963.52 !\n call 66 returned: 963.30\nf after iteration 59: 963.30 !\n call 67 returned: 962.87\nf after iteration 60: 962.87 !\n call 68 returned: 962.57\nf after iteration 61: 962.57 !\n call 69 returned: 962.27\nf after iteration 62: 962.27 !\n call 70 returned: 962.15\n call 71 returned: 961.79\nf after iteration 63: 961.79 !\n call 72 returned: 960.51\nf after iteration 64: 960.51 !\n call 73 returned: 955.57\nf after iteration 65: 955.57 !\n call 74 returned: 950.08\nf after iteration 66: 950.08 !\n call 75 returned: 937.37\nf after iteration 67: 937.37 !\n call 76 returned: 921.56\nf after iteration 68: 921.56 !\n call 77 returned: 902.80\nf after iteration 69: 902.80 !\n call 78 returned: 919.33\n call 79 returned: 896.35\nf after iteration 70: 896.35 !\n call 80 returned: 887.90\nf after iteration 71: 887.90 !\n call 81 returned: 868.63\nf after iteration 72: 868.63 !\n call 82 returned: 851.15\nf after iteration 73: 851.15 !\n call 83 returned: 827.70\nf after iteration 74: 827.70 !\n call 84 returned: 805.88\nf after iteration 75: 805.88 !\n call 85 returned: 802.61\nf after iteration 76: 802.61 !\n call 86 returned: 809.97\n call 87 returned: 797.59\nf after iteration 77: 797.59 !\n call 88 returned: 794.03\nf after iteration 78: 794.03 !\n call 89 returned: 778.43\nf after iteration 79: 778.43 !\n call 90 returned: 768.42\n call 91 returned: 766.10\nf after iteration 80: 766.10 !\n call 92 returned: 762.72\n call 93 returned: 751.91\nf after iteration 81: 751.91 !\n call 94 returned: 735.94\nf after iteration 82: 735.94 !\n call 95 returned: 750.64\n call 96 returned: 731.79\nf after iteration 83: 731.79 !\n call 97 returned: 723.95\nf after iteration 84: 723.95 !\n call 98 returned: 1046.98\n call 99 returned: 716.29\nf after iteration 85: 716.29 !\n call 100 returned: 696.83\nf after iteration 86: 696.83 !\n call 101 returned: 686.71\nf after iteration 87: 686.71 !\n call 102 returned: 676.83\nf after iteration 88: 676.83 !\n call 103 returned: 669.52\nf after iteration 89: 669.52 !\n call 104 returned: 665.31\nf after iteration 90: 665.31 !\n call 105 returned: 679.80\n call 106 returned: 654.77\nf after iteration 91: 654.77 !\n call 107 returned: 644.50\nf after iteration 92: 644.50 !\n call 108 returned: 636.91\n call 109 returned: 634.24\nf after iteration 93: 634.24 !\n call 110 returned: 632.36\n call 111 returned: 626.56\nf after iteration 94: 626.56 !\n call 112 returned: 612.69\nf after iteration 95: 612.69 !\n call 113 returned: 604.85\nf after iteration 96: 604.85 !\n call 114 returned: 594.35\nf after iteration 97: 594.35 !\n call 115 returned: 577.45\nf after iteration 98: 577.45 !\n"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:04:01.506846",
"end_time": "2016-05-15T08:04:01.511877"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "xmin",
"execution_count": 20,
"outputs": [
{
"execution_count": 20,
"output_type": "execute_result",
"data": {
"text/plain": "array([ 1.09556217, 1.19096181, 1.29048555, 2.08932217,\n 4.74801924, 22.86724499, 518.82339562])"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:04:02.335927",
"end_time": "2016-05-15T08:04:02.341329"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "rosen(xmin)",
"execution_count": 21,
"outputs": [
{
"execution_count": 21,
"output_type": "execute_result",
"data": {
"text/plain": "2209.0430396217657"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2016-05-15T08:04:02.911712",
"end_time": "2016-05-15T08:04:02.916751"
},
"collapsed": false,
"trusted": true
},
"cell_type": "code",
"source": "fmin",
"execution_count": 22,
"outputs": [
{
"execution_count": 22,
"output_type": "execute_result",
"data": {
"text/plain": "577.45032867813188"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "**xmin can in principle be arbitrarily bad no matter how good the previously found points already are**"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Reason for both problems"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "If the number of function evaluations reaches maxfun, the algorithm returns the function value returned by last function call as fmin and the x that would be used for the next function call as xmin. Both are wrong, because they are from within the line search and thus neither the current nor the next function call are guaranteed to represent the optimimum of the function evaluations done so far.\n\nInstead, the algorithm should only return results of completed iterations, i.e. by either restoring xmin and fmin of the last completed iteration or completing the current iteration despite reaching maxfun."
}
],
"metadata": {
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"notify_time": "5",
"language_info": {
"mimetype": "text/x-python",
"nbconvert_exporter": "python",
"name": "python",
"file_extension": ".py",
"version": "3.4.3",
"pygments_lexer": "ipython3",
"codemirror_mode": {
"version": 3,
"name": "ipython"
}
},
"gist": {
"id": "41da8b80342c437ecc3e41c743aff98f",
"data": {
"description": "scipy_lbfgs_mwe.ipynb",
"public": false
}
},
"_draft": {
"nbviewer_url": "https://gist.github.com/41da8b80342c437ecc3e41c743aff98f"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment