trying to figure out useful info for running python on dreamhost shared hosting - intended for twitter bot makers

dreamhostpython.md

python for botmakers, on dreamhost shared hosting

On a shared hosting service like dreamhost, how do you get your twitter bot up and running? Problems:

• where should I put my script?
• you can't install python modules like tweepy (for twitter access) because you don't have root permission
• once you get that solved, how do you run your script? cron?

I'm still figuring this stuff out myself, so nothing is clear as it should be. Hope this page will be a resource that will improve over time.

where should I put my script?

You almost definitely don't want it in your web directory if you're making a twitterbot. You can put it in your home directory or in a subdirectory of the home dir. When you make the virtual environment for it (see below) it will automatically make a subdirectory for you.

installing modules

Python's virtualenv lets you make your own private python where you can install modules without root access. http://docs.python-guide.org/en/latest/dev/virtualenvs/

You need shell access to get everything set up, but once it's set up, you can edit your actual script over ftp or with Coda or whatever you like.

See dreamhostpython.sh-session below for an actual shell session where I made a virtualenv and installed tweepy. Here's commentary on that session.

The command virtualenv mynewbot will make a new python virtual environment in a new directory called mynewbot. Then cd mynewbot to enter that directory, and source bin/activate to tell python you want to use this virtualenv.

The first thing I tried to do in my new virtualenv was install tweepy, the python twitter module - but it failed because dreamhost supplies an ancient version of pip, the python module installer. Luckily, you can upgrade pip inside the virtualenv with the command pip install --upgrade pip

Now it's possible to install tweepy with pip install tweepy. I made a really simple python script that just tries to import tweepy, and then showed that it works fine in the virtualenv but fails outside of the virtualenv.

running your script

To test your script, you'll probably want to run it from the dreamhost shell, like in my transcript below, but later you'll want to make it run autonomously, probably from a scheduled cron job. Issues:

• you need to tell cron to use the virtualenv
• cron always runs from your home directory; if your script cares about the working directory (because it loads external vocabulary files or some such), you need to take that into account.

The shell script test.sh in the transcript below takes care of both those issues by turning on the virtualenv and moving into your python script's directory. Instead of running the python script directly, run the shell script and let it run the python for you.

crontab on dreamhost

Here's a screenshot from the dreamhost control panel, under goodies / crontab, for running the example test.sh:

dreamhostpython.sh-session

[dreamhost]$ # make a python virtual environment so you can install your own modules
[dreamhost]$ virtualenv mynewbot
New python executable in mynewbot/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
[dreamhost]$ # go into the mynewbot directory
[dreamhost]$ cd mynewbot/
[dreamhost]$ # enter the virtualenv
[dreamhost]$ source bin/activate
(mynewbot)[dreamhost]$ # upgrade pip because dreamhost pip is obsolete, uggghhhhhhh
(mynewbot)[dreamhost]$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa
...
Successfully installed pip
(mynewbot)[dreamhost]$ # now you can install modules like tweepy
(mynewbot)[dreamhost]$ pip install tweepy
...
Successfully installed oauthlib-0.7.2 requests-2.4.3 requests-oauthlib-0.4.1 six-1.7.3 tweepy-3.1.0
(mynewbot)[dreamhost]$ # here is a little test python script
(mynewbot)[dreamhost]$ cat > test.py
import tweepy

print "YAY! I didnt crash"

(mynewbot)[dreamhost]$ python test.py
YAY! I didnt crash
(mynewbot)[dreamhost]$ 
(mynewbot)[dreamhost]$ # if i leave the virtual environment, it won't work
(mynewbot)[dreamhost]$ deactivate 
[dreamhost]$ 
[dreamhost]$ python test.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import tweepy
ImportError: No module named tweepy
[dreamhost]$ 
[dreamhost]$ # I can re-enter the virtual environment
[dreamhost]$ source bin/activate
(mynewbot)[dreamhost]$ 
(mynewbot)[dreamhost]$ # now it works again
(mynewbot)[dreamhost]$ python test.py
YAY! I didnt crash
(mynewbot)[dreamhost]$ deactivate
[dreamhost]$ 
[dreamhost]$ 
[dreamhost]$ # or i can run it like this without activating the virtualenv first
[dreamhost]$ bin/python test.py
YAY! I didnt crash
[dreamhost]$ # that should work from cron too, if I put the whole path in cron
[dreamhost]$ /home/moonmilk/mynewbot/bin/python /home/moonmilk/mynewbot/test.py 
Yay! I didnt crash
[dreamhost]$ 
[dreamhost]$ # Another way to run it without activating it first:
[dreamhost]$ # Make a shell wrapper
[dreamhost]$ # Advantage of this way is it sets the working directory
[dreamhost]$ # -- so if your script uses other files in the same directory 
[dreamhost]$ # -- you don't have to give the full path name of those files
[dreamhost]$ # this is from http://stackoverflow.com/a/4150693
[dreamhost]$ cat > test.sh
#!/bin/bash
cd /home/moonmilk/mynewbot
source bin/activate

python test.py
[dreamhost]$ 
[dreamhost]$ # did it work?
[dreamhost]$ test.sh
-bash: test.sh: command not found
[dreamhost]$ # ugh it has to be executable
[dreamhost]$ chmod +x test.sh
[dreamhost]$ ./test.sh
YAY! I didn't crash

[dreamhost]$ # it should run from crontab ok
[dreamhost]$ # with an entry like this
[dreamhost]$ /home/moonmilk/mynewbot/test.sh
YAY! I didn't crash

[dreamhost]$ # if i want to install more python modules for my script
[dreamhost]$ # i have to remember to activate the virtualenv first
[dreamhost]$ # here I forgot to activate, so it failed to install:
[dreamhost]$ pip install pillow
Downloading/unpacking pillow
...
error: could not create '/usr/local/lib/python2.7/dist-packages/PIL': Permission denied

[dreamhost]$ # OK this time I will activate the virtualenv first!
[dreamhost]$ source bin/activate
(mynewbot)[dreamhost]$ pip install pillow
Installing collected packages: pillow
 ...
Successfully installed pillow-2.7.0
(mynewbot)[dreamhost]$ # YAY!

Thanks for this. Really help me to understand virtualenv.

Had to set my crontab to this for it to work though:
$ sh /home/moonmilk/mynewbot/test.sh
(prefix line 76 with sh command)

I also found this very helpful (4 years later!). In the shell script, do you think the virtual environment should be deactivated after the python script runs?

I also found this very helpful (4 years later!). In the shell script, do you think the virtual environment should be deactivated after the python script runs?

Good question! My guess would be that it's not necessary, because the whole shell will vanish after the cron job finishes, so it doesn't matter whether it's still in the virtual environment or not.

Thank you so much for this! This was super helpful :)

Thanks for posting this, it was very helpful!