How to run Vatic on Windows 10 based Ubuntu

How to run Vatic on Windows 10 based Ubuntu.md

1. Get Linux Bash Shell on Windows 10 (https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/)

   1. Open the Settings app and head to Update & Security > For Developers. Activate the "Developer Mode" switch here to enable Developer Mode.
   2. Next, open the Control Panel, click "Programs," and click "Turn Windows Features On or Off" under Programs and Features. Enable the "Windows Subsystem for Linux (Beta)" option in the list here and click "OK."
   3. After your computer restarts, click the Start button (or press the Windows key), type "bash", and press "Enter."

2. Install dependencies of vatic. The rest of this guide is based on the instruction at https://gitlab.com/EAVISE/publicwiki/wikis/install-vatic, but follow this guide here written for Windows 10's Ubuntu. I highlighted the necessary changes from the original guide in bold.    1. First install the needed packages, make sure that you leave the MySQL password empty (You will be asked to provide it this several times):         sudo apt-get install -y git g++ make python-setuptools python-dev libavcodec-dev libavformat-dev libjpeg62 libjpeg62-dev libfreetype6 libfreetype6-dev apache2 libapache2-mod-wsgi mysql-server-5.6 mysql-client-5.6 libmysqlclient-dev

   2. Next, install these dependencies, this might take a while.
      sudo easy_install pip
sudo pip install cython==0.20
sudo pip install SQLAlchemy wsgilog mysql-python munkres parsedatetime argparse
sudo pip install numpy
sudo pip install Pillow

   3. Install ffmpeg, on Ubuntu 14.04 you need to add an additional ppa:
      sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get install ffmpeg

   4. Every time you call a command with 'sudo', you will get a warning similar to the following. This is ok. sudo: unable to resolve host MININT-EH2KQSQ

3. Download and clone vatic. Vatic consists of three parts: turkic, pyvision and vatic itself:

   1. Choose a folder to grab vatic library: (for this tutorial, it's C:\src\vatic)
      cd /mnt/c/src
mkdir vatic
cd vatic
git clone https://github.com/cvondrick/turkic.git
git clone https://github.com/cvondrick/pyvision.git
git clone https://github.com/cvondrick/vatic.git

   2. Vatic uses an older version of PIL, so some of the original code in vatic.git needs to be updated:

      1. Download my patch from https://raw.githubusercontent.com/Efreeto/vatic/master/patch.diff
      2. Then apply this patch:
         cd vatic
git apply patch.diff
cd ..

   3. First install turkic:
      cd turkic
sudo python setup.py install
cd ..

   4. Do the same for pyvision:
      cd pyvision
sudo python setup.py install
cd ..

4. Setup Apache server. The client-side (annotation-side) GUI runs on a browser, so it's necessary to host this GUI server.

   1. Now configure the webserver (Apache2). Replace the contents of the following file:
      sudo nano /etc/apache2/sites-enabled/000-default.conf

      WSGIDaemonProcess www-data
      WSGIProcessGroup www-data
      
      <VirtualHost *:80>
          ServerName localhost
          DocumentRoot /mnt/c/src/vatic/vatic/public
      
          WSGIScriptAlias /server /mnt/c/src/vatic/vatic/server.py
          CustomLog /var/log/apache2/access.log combined
      </VirtualHost>
      

   2. Give Apache access to this directory by adding the following to the config file:
      sudo nano /etc/apache2/apache2.conf

      <Directory /mnt/c/src/vatic/vatic/>
          Options Indexes FollowSymLinks
          AllowOverride None
          Require all granted
      </Directory>
      

   3. Restart Apache2:
      sudo cp /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled
sudo apache2ctl graceful

      i. You will get errors similar to the following. This is ok.

      sudo: unable to resolve host MININT-EH2KQSQ
      AH00557: apache2: apr_sockaddr_info_get() failed for MININT-EH2KQSQ
      AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
      httpd not running, trying to start
      [Thu Mar 09 14:14:34.395601 2017] [core:warn] [pid 797:tid 140033705052032] (22)Invalid argument: AH00076: Failed to enable APR_TCP_DEFER_ACCEPT

   4. Test Apache by going to 'localhost' on your browser on Windows

      1. If you see a blank page with the title 'vatic', Apache is running successfully. You can proceed to the next part.
      2. If you see a default Apache page, one of the file paths in your configuration is probably incorrect.
      3. If you don't see any web page, you might have to restart the whole process. :(

   5. (Optional) Open your localhost for other people to annotate on your data

      1. Go to control panel > windows firewall > advance setting (on left sidebar)
      2. Right click on inbound rules -> add new rule -> port -> TCP (Specific port 80 - if your localhost wok on this port) -> Allow the connections -> Give a profile name -> ok

5. Prepare MySQL database for your server. This is only necessary once.

   1. Edit the MySQL database:
      sudo service mysql start
mysql -u root
create database vatic;
exit

       1. You will get errors similar to the following. This is ok.
       sudo: unable to resolve host MININT-EH2KQSQ
       initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
       /lib/lsb/init-functions.d/50-ubuntu-logging: line 5:   131 Aborted                 (core dumped) plymouth --ping > /dev/null 2>&1
        * Starting MySQL database server mysqld                                                                                                                                                                                  No directory, logging in with HOME=/
       /lib/lsb/init-functions.d/50-ubuntu-logging: line 5:   556 Aborted                 (core dumped) plymouth --ping > /dev/null 2>&1
                                                                                                                                                                                                                          [ OK ]
        * Checking for tables which need an upgrade, are corrupt or were
       not closed cleanly.
      

   2. In the vatic/vatic directory:
      cd vatic
cp config.py-example config.py
turkic setup --database
turkic setup --public-symlink

      1. Both turkic commands should run successfully.

   3. Test if vatic is working, by surfing in a browser to localhost, and by testing this in the command line:
      turkic status --verify

      1. Browser should show 'Welcome to vatic' page
      2. On the status command, an error is seen with Mechanical Turk, however, the other features should return ok.

   4. Now give correct permissions for www-data:
      sudo usermod -a -G administrator www-data
sudo mkdir /var/www/.python-eggs/
sudo chown www-data:www-data /var/www/.python-eggs/
sudo apache2ctl graceful

   5. That's it! You can refer to 'Using vatic' in the next section to start using vatic.

   6. If you believe you messed up somewhere and start over, this is how you nuke the whole Ubuntu and start again. From cmd in administrator mode: lxrun /uninstall /full lxrun /install

6. Using vatic (basic commands).

   1. Make sure you are running Apache and MySQL. This is only needed once per session. This step is unnecessary if you already did these above. But you will need these next time you start Ubuntu. sudo apache2ctl graceful
sudo service mysql start

   2. Now, run the rest of the commands in the directory where you have vatic. For this tutorial this is:
      cd /mnt/c/src/vatic/vatic

   3. Convert video to frames:
      export THIS_VIDEO=${THIS_VIDEO:-workshop} turkic extract videos/raw/$THIS_VIDEO.avi videos/extracted/$THIS_VIDEO --no-resize

      Or convert images to vatic format frames: export THIS_VIDEO=${THIS_VIDEO:-building34} mkdir videos/raw/$THIS_VIDEO ffmpeg -i videos/raw/$THIS_VIDEO.mp4 -vf fps=2 videos/raw/$THIS_VIDEO/%d.jpg turkic formatframes videos/raw/$THIS_VIDEO/ videos/extracted/$THIS_VIDEO

   4. Add video for local annotation, identifier gives a unique name, the category is given at the end of the command (Person in this example):
      turkic load identifier videos/extracted/video1 Person --offline --blow-radius=0

   5. Request URL to perform annotation:
      turkic publish --offline

      1. Goto the URL on the output from the above command and annotate! (For example, http://localhost/?id=1&hitId=offline)
      2. If you opened port 80 from the steps above, other people on your network can see your vatic site at your ip address. (For example, http://10.123.456.789/?id=1&hitId=offline)

   6. To export the annotations:
      turkic dump identifier -o output.txt

   7. To visualise tracks (= create image files with the annotations):
      turkic visualize identifier videos/visualized/video1 --merge --labels

   8. To delete a video:
      turkic delete identifier --force

   9. Here is an example script of automatically preparing the annotation, and upon user input to finish annotation, export the annotations to a text file. mkdir outputs mkdir sqls export THIS_VIDEO=${THIS_VIDEO:-building34} turkic load $THIS_VIDEO videos/extracted/$THIS_VIDEO Jackhammer Saws Drill PipeWrench Hammer Person --offline --blow-radius=0 turkic publish --offline read -p "Press Enter after annotation" turkic dump $THIS_VIDEO -o outputs/$THIS_VIDEO.txt mysqldump -u root -p vatic > sqls/$THIS_VIDEO.sql

7. Common errors

   1. turkic.geolocation errors

      1. Make sure you are in the right vatic folder (e.g. C:/src/vatic/vatic)
      2. Make sure "mysql" works and "apache2" is running

   2. no module named PIL sudo pip install PIL sudo pip install Pillow You will need both PIL and Pillow (?) to install PIL correctly. pip version should be >9.0 too (not 1.6. or something). there may be two versions of pip installed, then you will need to uninstall one of them (i.e. sudo apt-get remove python-pip)

   3. Server Error (from Chrome screen) replace 'administrator' to your ubuntu bash id from following

      sudo usermod -a -G administrator www-data sudo mkdir /var/www/.python-eggs/ sudo chown www-data:www-data /var/www/.python-eggs/ sudo apache2ctl graceful

   4. Video tearing

      1. Don't use Edge and Switch to Chrome 2 Try refreshing the page (Don't forget to save your annotations first!)