Skip to content

Instantly share code, notes, and snippets.

@CarstVaartjes

CarstVaartjes/fb_prophet_chalice.py

Created April 8, 2018 19:24
Show Gist options
  • Star 6 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save CarstVaartjes/77dbe8249d171e592bd17847f7009272 to your computer and use it in GitHub Desktop.
Save CarstVaartjes/77dbe8249d171e592bd17847f7009272 to your computer and use it in GitHub Desktop.
Download ZIP
fb-prophet transformation for aws lambda
Raw
fb_prophet_chalice.py
# based on https://github.com/ryansb/sklearn-build-lambda
$ mkdir ~/.aws
$ cat >> ~/.aws/config
[default]
aws_access_key_id=YOUR_ACCESS_KEY_HERE
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
region=YOUR_REGION (such as us-west-2, us-west-1, etc)
rm -rf /srv/python/chalice
virtualenv --python=/usr/bin/python2.7 /srv/python/chalice
source /srv/python/chalice/bin/activate
pip install --upgrade pip
pip install -r ~/PycharmProjects/xxx/requirements/requirements.txt
pip install chalice
cd "$VIRTUAL_ENV/local/lib/python2.7/site-packages"
du -sh
du -sh * | sort -rh | head -10
du -xh -d 1 * | sort -rh | head -20
du -xh -d 1 pystan/* | sort -rh | head -20
du -xh -d 1 scipy/* | sort -rh | head -20
du -xh -d 1 fbprophet/* | sort -rh | head -20
# libdir="$VIRTUAL_ENV/lib64/python2.7/site-packages/lib/"
# mkdir -p $VIRTUAL_ENV/lib64/python2.7/site-packages/lib || true
# cp /usr/lib64/atlas/* $libdir
# cp /usr/lib64/libquadmath.so.0 $libdir
# cp /usr/lib64/libgfortran.so.3 $libdir
echo "venv original size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.so" | xargs strip
echo "venv stripped size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "test" | xargs rm -rf
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "tests" | xargs rm -rf
echo "venv tests removed size $(du -sh $VIRTUAL_ENV | cut -f1)"
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/lib/stan_math/lib"
echo "venv source codes removed size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.py" | xargs python -m py_compile
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.py" | xargs rm -f
echo "venv py files compiled size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.json" | xargs rm -f
echo "json files removed size $(du -sh $VIRTUAL_ENV | cut -f1)"
"""
import os
import ctypes
for d, _, files in os.walk('lib'):
for f in files:
if f.endswith('.a'):
continue
ctypes.cdll.LoadLibrary(os.path.join(d, f))
"""
# pushd $VIRTUAL_ENV/lib/python2.7/site-packages/ && zip -r -9 -q /tmp/partial-venv.zip * ; popd
# pushd $VIRTUAL_ENV/local/python2.7/site-packages/ && zip -r -9 --out /outputs/venv.zip -q /tmp/partial-venv.zip * ; popd
# echo "site-packages compressed size $(du -sh /outputs/venv.zip | cut -f1)"
# pushd $VIRTUAL_ENV && zip -r -q /outputs/full-venv.zip * ; popd
# echo "venv compressed size $(du -sh /outputs/full-venv.zip | cut -f1)"
# pushd $VIRTUAL_ENV && zip -r -q /outputs/full-venv.zip * ; popd
# zip -r -q /outputs/full-venv.zip
@Hatter1337
Copy link

Hello! You solved the problem with Lambda and fb-prophet?
How to run this script? You use Python version 2.7?

@CarstVaartjes
Copy link
Author

Hi! Yes this is Python 2.7 but it should also work on 3.6; the latest fb prophet version (0.3) is also smaller

@ys3006
Copy link

ys3006 commented Jul 23, 2018 •
edited

Hey! I followed your way of removing these extra files but on Python 3.6, and it keeps returning this error when I test in Lambda Function:

File "/var/task/pystan/init.py", line 9, in
from pystan.api import stanc, stan
File "/var/task/pystan/api.py", line 13, in
import pystan._api # stanc wrapper
ModuleNotFoundError: No module named 'pystan._api'

Any chances you would know how to solve this issue?

I found in pystan/_api.cpp, it has:
"include_dirs": [
"./pystan",
"pystan/stan/src",
"pystan/stan/lib/stan_math/",
"pystan/stan/lib/stan_math/lib/eigen_3.3.3",
"pystan/stan/lib/stan_math/lib/boost_1.64.0",
"pystan/stan/lib/stan_math/lib/cvodes_2.9.0/include"
],
Is this error raised because of doing the following command?
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/lib/stan_math/lib"

@CarstVaartjes
Copy link
Author

Note to self:
facebook/prophet#179
stan-dev/math#545
https://medium.com/@maebert/machine-learning-on-aws-lambda-5dc57127aee1

@matse004
Copy link

got it working finally. Thanks a lot for the help! Here's all the steps I took:
https://medium.com/@amacenov/how-to-get-fbprophet-work-on-aws-lambda-c3a33a081aaf

@raychew13
Copy link

got it working finally. Thanks a lot for the help! Here's all the steps I took:
https://medium.com/@amacenov/how-to-get-fbprophet-work-on-aws-lambda-c3a33a081aaf

Thanks for your instructions.
Followed your step to run the docker container
docker run --rm -it -v “$PWD”:/var/task lambci/lambda:build-python3.6 bash
Then I created the virtual env (venv), activate venv and upgrade pip. But I'm getting error when installing the fbprophet. This looks like an issue with installing pystan dependency as part of the fbprophet installation. Just wondering if you need to install any gcc libraries within the docker container before you install the fbprophet

pip install fbprophet --no-cache

'    ERROR: Complete output from command /var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_model
    DIAGNOSTIC(S) FROM PARSER:
    Warning: left-hand side variable (name=cp_idx) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    Warning: left-hand side variable (name=m_pr) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_861b75c6337e237650a61ae58c4385ef NOW.
    error: command 'gcc' failed with exit status 4
    ----------------------------------------
ERROR: Command "/var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet" failed with error code 1 in /tmp/pip-install-16ok_rcd/fbprophet/

@matse004
Copy link

matse004 commented Aug 5, 2019

got it working finally. Thanks a lot for the help! Here's all the steps I took:
https://medium.com/@amacenov/how-to-get-fbprophet-work-on-aws-lambda-c3a33a081aaf

Thanks for your instructions.
Followed your step to run the docker container
docker run --rm -it -v “$PWD”:/var/task lambci/lambda:build-python3.6 bash
Then I created the virtual env (venv), activate venv and upgrade pip. But I'm getting error when installing the fbprophet. This looks like an issue with installing pystan dependency as part of the fbprophet installation. Just wondering if you need to install any gcc libraries within the docker container before you install the fbprophet

pip install fbprophet --no-cache

'    ERROR: Complete output from command /var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_model
    DIAGNOSTIC(S) FROM PARSER:
    Warning: left-hand side variable (name=cp_idx) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    Warning: left-hand side variable (name=m_pr) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_861b75c6337e237650a61ae58c4385ef NOW.
    error: command 'gcc' failed with exit status 4
    ----------------------------------------
ERROR: Command "/var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet" failed with error code 1 in /tmp/pip-install-16ok_rcd/fbprophet/

Try pystan==2.18
Also: https://medium.com/@marc.a.metz/docker-run-rm-it-v-pwd-var-task-lambci-lambda-build-python3-7-bash-c7d53f3b7eb2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment