[GSoC '18] OwnMailbox - Final Report

gsoc-pipilisg-final-report.md

This is the final report/work submission for the OwnMailbox project, for Google Summer of Code 2018.

Student: Georgios Pipilis
Mentor: Pierre Parent

Relevant pull requests

1. Own-Mailbox/proxy#13 (the most important one)
2. Own-Mailbox/debstrap#5
3. Own-Mailbox/cs-com#1
4. Own-Mailbox/debstrap#4
5. Own-Mailbox/debstrap#6

GSoC Final Progress Report

Sections 1 and 2 give you an idea of how the project was before GSoC. Sections 3, 4 and 5 present the results briefly, together with build/run instructions. On section 6 you can find details about the code added and the corresponding commits. Lastly, section 7 describes my personal experience of GSoC and the interaction with my mentor and the community.

Index

1. Overview of OwnMailbox (OMB)
2. OwnMailbox before GSoC
3. Project goals for GSoC
4. OwnMailbox's structure after GSoC
5. How to deploy the OMB proxy server
6. Debugging: Some of the problems encountered (and how they were solved)
7. What did I learn from the project

1 - Overview of OwnMailbox (OMB)

From the OMB project description:

OwnMailbox is a personal email server that is easy to set up and run, with strong privacy protection measures integrated at its core. It provides self-hosted email addresses and allows you to protect the content of your emails from mass surveillance. Own-Mailbox is based on technologies such as debian Gnu/linux, mailpile, postfix, gpg, tor.

The main repositories/parts of the project are:

Debstrap: A collection of scripts that allows you to transform a Debian machine into a full OwnMailbox instance. A script for installation through Vagrant is also available.

Proxy: The server side of OwnMailbox. Handles the subdomains of the users, incoming and outgoing mail.

CS-COM: Scripts that handle communication between OMB client and proxy server.

IHM: A web interface that allows you to install and setup OwnMailbox from localhost. I didn't make any changes to this during the project.

Sni2tor-proxy: Proxies incoming TLS connections to tor hidden services based on the hostname contained in the initial request of the TLS session.

And more repositories like mailpile and smpt2tor-relay that I didn't change directly.

2 - OwnMailbox before GSoC

Before the start of GSoC, the project worked fully with the domain omb.one. A user could simply clone debstrap, edit the config.sh file, run vagrant up and have a fully working OMB container.

The proxy server worked 100% with the omb.one domain but not so well with other/custom domains (however, some commits were already made in order to add this feature). Also, there was support for Docker but it was shaky - a user could create his own proxy server using Docker but it was not functional.

You could deploy the proxy using the scrip install-wo-docker.sh or use the docker-scripts library for Docker.

3 - Project goals for GSoC

Initially, the goals of OwnMailbox focused on three things:

1. Adding support for HSTS to debstrap
2. Fixing the proxy server (make the proxy repository work)
3. Adding support for the MIT GPG key server (publishing GPG key, and searching GPG keys)

At the end of GSoC, we finished all of our goals. This is what was achieved:

1. Added support for HSTS to debstrap
2. Fixed the proxy server & added support for custom domains. Added dovecot to the proxy and automated its configuration.
3. Added full Docker support for the proxy repo (there was some shaky Docker support before GSoC but we made something simpler, fully functional and completely automated)
4. Updated the proxy to Debian Stretch and cs-com to PHP 7
5. Adding support for the MIT GPG key server (publishing GPG key, and searching GPG keys)

In the last few days (and after the testing phase for the proxy server finished), the 5th task (GPG integration) was finished.

4 - OwnMailbox's structure after GSoC

Changes to the client of OMB (debstrap): Changes to the apache configuration, the omb-client script that debstrap uses (these changes affect only the client side, this is why we applied them here and not on the cs-com repository). Also, some hard-coded settings (like IPs and domain names) were replaced with variables, for custom domain support. The structure of the repository, however, remains the same.

Changes to the Proxy server (proxy): The way proxy works has changed a lot after GSoC. Now, the script called automate-docker.sh handles all of the action.

1. This script erases all existing Docker images & containers (OMB proxy is supposed to be running exclusively inside a machine)

2. It runs the docker build command, which uses our Dockerfile to create the container

3. The Dockerfile installs all necessary packages and edits the files we need appropriately

4. Before the Dockerfile exits, it runs the script called dockercfg.sh. This script, in turn, runs the configuration scripts apache2.sh, bind9.sh cs-com.sh, db.sh, dovecot.sh, postfix.sh and sni2torproxy.sh. Basically, it runs all of the configuration files except for get-ssl-cert.sh.

5. The Dockerfile exits and the Docker container is started (always automatically). This causes our init.sh script (that is defined as a CMD in our Dockerfile) to run. This script starts all of the services required by OMB and runs the get-ssl-cert.sh script. The reason we used this approach has to do with Docker's structure.

   By default, Docker is designed to run one service per container. This service is the CMD (or ENTRYPOINT) service specified at the end of the Dockerfile. So, if at the end of the Dockerfile we simply start apache2 or tor, no other services will be started. For this reason (and following Docker standard procedures mentioned in the Docker documentaion) I created init.sh that starts all of the other daemons and then waits forever, acting as a single service.

   Also, we force the Docker container to run using --network host and --privileged. These settings are essential for the functionality of the container and allow us to use the host's network stack, without restrictions and blocked ports.

   The get-ssl-cert.sh is also run here because of this reason. We run it using the command no | ./get-ssl-cert.sh because, otherwise, the script waits for a response to the "e-mail survery" question and the container crashes (see this commit).

6. automate-docker.sh exits and we get shell inside the container. After this step, it takes about 4-5 minutes for all of the daemons to be started properly.

Furthermore, some other files have been added/moved/deleted inside the proxy repository:

1. The docker-scripts integration was completely removed in favor of something simpler for the average user (and more automated).
2. A completely new Dockerfile was created for the needs of the project.
3. The /src/dovecot folder was added. This includes all of the required configuration files for Dovecot. This configuration is handles by a new script called dovecot.sh.
4. The /scripts/bind folder was added. This includes custom configuration files that allow us to print more detailed (and separate for each channel) bind9 logs.
5. A lot of unnecessary settings were removed from settings.sh.

Also, all of the files in the scripts folder were modified heavily for the needs of the project (more details below).

Changes to CS-COM: All of the PHP files inside cs-com/server were updated to PHP 7 and some hard-coded addresses were changed.

5 - How to deploy the OMB proxy server

In order to create a new proxy server and an email address follow these steps:

On the server side:

1. Install Docker
2. Clone the proxy repository and edit the settings.sh file to your liking
3. Be sure that your machine is fully dedicated to being the proxy before running this. Do not install this on a machine that hosts other services. Also make sure that no Apache2, mysql/mariadb or tor is running on this machine.
4. Run bash automate-docker.sh, respond yes to the prompt and wait for the script to finish.
5. You automatically get a shell inside your container. Wait 4-5 minutes until all daemons start (you can check if everything has started by running service --status-all and seeing if postfix, dovecot, apache2 and bind9 are running).

On the client side:

1. Install Vagrant and Docker
2. Edit the config.sh file
3. Run cd Vagrant && vagrant up
4. After the installation finishes, go to localhost on your browser and follow the installation steps on screen

At some point in the installation, an identification link is needed. You can get that (if your proxy server is running) by visiting https://proxy.< your domain >:6565/request-omb/Create_Acounts/.

6 - Debugging: Some of the problems encountered (and how they were solved)

This is section is devoted mostly to the bug-solving I did and not the coding. Usually, it was things like these that slowed me down (and provided me with tons of knowledge).

Keep in mind that, when I started debugging these, I didn't have a lot of knowledge on how the proxy server works. Solving this offered tremendous insights to how the project works.

Problem: Identification links were not created at all.
Solution: Updated all of our PHP scripts in cs-com to PHP 7.
Commits: here and here.

Problem: Normally, an Own-Mailbox should be accessible via its HTTPS address. However, when you created a new container using Docker, this was not happening.
Solution: Used proper cronjobs in order to make sure sni2tor run and started the container daemons in a proper order.
Commits: here and here.

Problem: When a new OMB client container was created (using a custom proxy server and not omb.one) it had an invalid SSL certificate.
Solution: Changed our letsencrypt.cgi script [1], restricted the letsencrypt module to the installation [2], changed our apache.conf [3] and some file permissions.
Commits: [1] [2] [3]

Problem: Incoming mails were not working.
Solution: A lot of things needed to be done here. First of all, add dovecot to the container (many relevant commits, the most important is this [1]). But, most importantly, the file /etc/dovecot/users was not populated properly. This was because of the script needed to run as root from inside PHP. Solved with some visudo changes [2]. Also some file permissions and file locations were affecting this issue.
Commits: [1] [2]

Problem: Outgoing mails were not working.
Solution: Changes to our smtp_tor script, removing some hardcoded IPs from proxy scripts, changes to /etc/postfix/main.cf and fixes to the add-dovecot-user.sh script. This issue had a lot of common fixes with the one above.
Commit: This as an example but many more in total (look at the next section for PRs).

Problem: A lot of incoming requests to the proxy were denied.
Solution: This was due to Docker and changes were made to its configuration. Most notably, it now runs in --network host mode, as explained above.

These are the problems that took the most to solve. There were some others linked to upgrading to Debian Stretch, Postfix and Docker. You can see the relevant commits in the pull requests below.

7 - What did I learn from the project

On the technical side: I coded almost exclusively using bash which made me more familiar with it. I also learned a lot about how Postfix and Dovecot work and I deepened my Docker knowledge. I've never worked with an email server so extensively before so this was also something new for me.

On the "thinking like a programmer" side: I had some problems at debugging code errors and finding the source of "unknown" problems. My mentor, Pierre, helped me a lot with this via some voice calls we had and I thank him deeply for this. He also helped me with my working pace and his advice helped me change how I see coding and facing problems (well, code problems, but some of this advice can be applied to real life 😄) in general.

I believe I learnt a lot about bash and Debian during these months but, most importantly, I learnt how to troubleshoot code properly, work and communicate with other people efficiently and interacted with the amazing Debian community.