Faster Matrix Synapse - Debian - Nginx, Postgres, PyPy, & Workers

stumbling.md

Faster Synapse (v0.33.0)

Quickstart guides get you a slow Synapse.

This is mostly a delta guide, unfortunately - you'll need to read all the other things to get this to work.

If you already have a Synapse server instance running, and you're going to switch from sqlite to postgres... Use Synapse's migration script. If you don't use the migration script Very Bad Things are supposed to happen.

Guide Assumptions

/etc/matrix-synapse/ Contains .yaml, homeserver.yaml, *.log.conf, etc.

/var/lib/matrix-synapse/ Contains .pid files, media, and upload directories.

/var/log/matrix-synapse/ Contains logs.

Deps

Install all the deps recommended for your distro, then hit postgres and pypy;

Debian; sudo apt-get install pypy pypy-dev postgresql postgresql-client

DB

DB Config per guide, then https://github.com/matrix-org/synapse/blob/master/docs/postgres.rst

NGINX Persistent

Nginx per guide, but use this template to enable persistent connections;

upstream matrix_backend {
    server localhost:8008;
    keepalive 16;
}

server {
    listen 443 ssl;
    server_name SERVER;

    ssl_certificate     /etc/letsencrypt/live/SERVER/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/SERVER/privkey.pem;

    ssl_session_cache shared:le_nginx_SSL:1m;
    ssl_session_timeout 1440m;

    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;

    ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:!DSS";

    location /_matrix {
        proxy_pass http://matrix_backend;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
}

Pypy Venv

Set up your virtualenv using pypy;

virtualenv --python=/usr/bin/pypy ~/.synapse as usual;

source ~/.synapse/bin/activate
pip install --upgrade pip
pip install --upgrade setuptools
pip install https://github.com/matrix-org/synapse/tarball/master

then; pip install psycopg2cffi

create the file ~/.synapse/site-packages/psycopg2.py Containing exactly;

from psycopg2cffi import compat
compat.register()

Then, pypy -m synapse.app.homeserver --config-path=[homeserver.yaml]

Reduce Logging Levels

Open up your SERVERNAME.log.config

Change all INFOs to ERRORs (or WARNINGs) https://docs.python.org/2/library/logging.html#levels

In order to make synctl use pypy instead of python, either set python to pypy in virtenv, or edit `which synctl` to replace python w/ pypy.

Workers

Per https://github.com/matrix-org/synapse/blob/master/docs/workers.rst

Create a directory under where your homeserver.yaml is stored called ./workers/.

We'll set up workers for every supported worker, except synapse.app.federation_reader, which requires a SRV record to route through nginx, and synapse.app.event_creator, which simply didn't work for reasons unknown.

under ./workers/ create the following;

homeserver.yaml

worker_app: synapse.app.homeserver
daemonize: true

synchrotron.yaml

worker_app: synapse.app.synchrotron

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8083
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/lib/matrix-synapse/synchrotron.pid
worker_log_config: /etc/matrix-synapse/[NAME].log.config

federation_sender.yaml

worker_app: synapse.app.federation_sender

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8084
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/lib/matrix-synapse/federation_sender.pid

media_repository.yaml

worker_app: synapse.app.media_repository

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8085
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/lib/matrix-synapse/media_repository.pid
worker_log_config: /etc/matrix-synapse/[NAME].log.config

client_reader.yaml

worker_app: synapse.app.client_reader

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8086
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/lib/matrix-synapse/client_reader.pid
worker_log_config: /etc/matrix-synapse/[NAME].log.config

user_dir.yaml

worker_app: synapse.app.user_dir

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8087
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/lib/matrix-synapse/user_dir.pid
worker_log_config: /etc/matrix-synapse/[NAME].log.config

frontend_proxy.yaml

worker_app: synapse.app.frontend_proxy

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8088
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/lib/matrix-synapse/frontend_proxy.pid
worker_log_config: /etc/matrix-synapse/[NAME].log.config

worker_main_http_uri: http://127.0.0.1:8008

../homeserver.yaml

Then add the following lines to the listeners block of the original ./homeserver.yaml

listeners:
..........snip.......
  # The TCP replication port
  - port: 9092
    bind_address: '127.0.0.1'
    type: replication
  # The HTTP replication port
  - port: 9093
    bind_address: '127.0.0.1'
    type: http
    resources:
     - names: [replication]

Then add the following lines to disable functions now carried out by workers;

#Workers
send_federation: False
enable_media_repo: False
update_user_directory: false

Nginx routes

Next up, edit your nginx config to route requests to the right worker instances;

upstream matrix_backend {
    server localhost:8008;
    keepalive 16;
}

upstream synchrotron {
    server localhost:8083;
    keepalive 16;
}

upstream media_repository {
    server localhost:8085;
    keepalive 16;
}

upstream client_reader {
    server localhost:8086;
    keepalive 16;
}

upstream user_dir {
    server localhost:8087;
    keepalive 16;
}
upstream frontend_proxy {
    server localhost:8088;
    keepalive 16;
}
......SNIP......

#Note, location order matters - the _matrix catchall needs to be last.

    ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:!DSS";

    location ~ ^/_matrix/client/(v2_alpha|r0)/sync$ {
        proxy_pass http://synchrotron;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location ~ ^/_matrix/client/(api/v1|v2_alpha|r0)/events$ {
        proxy_pass http://synchrotron;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location ~ ^/_matrix/client/(api/v1|r0)/initialSync$ {
        proxy_pass http://synchrotron;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location ~ ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ {
        proxy_pass http://synchrotron;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location ~ ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ {
        proxy_pass http://client_reader;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location ~ ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$ {
        proxy_pass http://user_dir;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location ~ ^/_matrix/client/(api/v1|r0|unstable)/keys/upload {
        proxy_pass http://frontend_proxy;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }
    
    location /_matrix/media/ {
        proxy_pass http://media_repository;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }

    location /_matrix {
        proxy_pass http://matrix_backend;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
    }

Kill your servers, now start Synapse like so;

(from virtualenv) synctl -a /etc/matrix-synapse/workers start /etc/matrix-synapse/homeserver.yaml

service nginx restart

Ensure synctl uses pypy to launch everything.

just setup my HS with this. problem is, all of the workers have the client resource. as such, media just doesn't work anymore unless the (listener) resource is changed.
considering revising this and using the feature to add files (yes that's a thing in github gists).

Hi @JacksonChen666 - you found my 4 year old guide! I don’t run Matrix servers anymore - so I probably won’t update this Gist unless someone sends me exact suggestions.

-Travis

On Sun, Jul 3, 2022 at 15:23 Jackson Chen ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ just setup my HS with this. problem is, all of the workers have the client resource. as such, media just doesn't work anymore unless the (listener) resource is changed. considering revising this and using the feature to add files (yes that's a thing in github gists). — Reply to this email directly, view it on GitHub <https://gist.github.com/84c43b9709374cfe1e5b0dba635b44f8#gistcomment-4220151>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABNDYADNMJJ5JDBPAV4CYLVSGA2NANCNFSM52QTX4EA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Twitter <https://twitter.com/tbiehn> | LinkedIn <http://www.linkedin.com/in/travisbiehn> | GitHub <http://github.com/tbiehn> | TravisBiehn.com <http://www.travisbiehn.com>

@tbiehn this guide seems pretty old too. there is a page about workers which can be helpful too, i just find reverse proxy examples a bit lacking (or i overlooked the document in quick succession) and that's why i got here.
the changes i found necessary to get things working is this one change (ok it's a list of changes but how am i supposed to permalink to one change?)