tech234a/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Using unmodified third-party Reddit apps with a custom server

The upcoming price increase for the Reddit API has resulted in the developers of many third-party Reddit apps to announce plans to shut down their apps on June 30. Many have encouraged these developers to update their apps to use a different server. Until 2017, the source code for Reddit and its API were open source, and this code is still available as an archive.
This open source version of Reddit lacks some features from the current site, most notably:

new Reddit
built-in image/video uploading
.i/.compact is still available
new Modmail and Modmail APIs
new traffic stats
chat (you can still send messages directly to users, just not in chat format)
some other API scopes/endpoints
anti-spam/anti cheating code

As a proof of concept, here is how to set up a custom Reddit server and use it with unmodified third-party Reddit apps. For this use, we will make it appear to your devices that this server is running at reddit.com; other users will continue to be able to access the real Reddit normally.
Standard disclaimers: Follow the instructions at your own risk. These instructions are a proof of concept and may not be suitable for production use. I have heard that sites that previously used Reddit code ended up finding it difficult to maintain and replaced it with a custom rewrite. Additionally, the license agreement for the code states that you need to have "Powered by reddit" somewhere on the webpages from your server. Also, I'm not interested in hosting a public server myself but hopefully these instructions are helpful for someone else.
Setting up the server


The Reddit source code currently only runs on Ubuntu 14.04. While anyone who would want to host a public server should update/recompile dependencies to get it to run on a newer operating system, this is okay for the proof of concept. Ubuntu 14.04 will receive security updates until April 2024. Note: most newer packages do not support Ubuntu 14.04, and you may have issues with outdated SSL certificates installed on the system. Download and install VirtualBox and also download the Ubuntu 14.04.6 server image. (The desktop image will also work if you'd like a GUI.)
Create a new VM. Select the ISO you downloaded, skip unattended installation, and give it at least 4096MB of memory. I would also recommend giving it multiple CPUs. The default hard disk size is fine.
Install the OS normally, but set your username to reddit. Select "Choose packages manually" at the end.
Once in the system, login and install any security updates: sudo apt update && sudo apt upgrade -y.
Now we'll install Reddit, using the script available in the repo with a minor environment variable tweak:

wget https://raw.github.com/reddit/reddit/master/install-reddit.sh
chmod +x install-reddit.sh
sudo REDDIT_DOMAIN=reddit.com ./install-reddit.sh
Press y when prompted.

Set up port forwarding for your virtual machine. In the menu bar above your VM, go to Machine>Settings>Network>Advanced>Port Forwarding and add the following rule:

Host port: 8443, Guest port: 443
Leave protocol as TCP, leave host IP and guest IP blank. Name doesn't matter. Save the changes.


Now update your configuration. Run cd ~/src/reddit/r2/ && nano development.update and change the file so it looks as follows:

# after editing this file, run "make ini" to
# generate a new development.ini

[DEFAULT]
# global debug flag -- displays pylons stacktrace rather than 500 page on error when true
# WARNING: a pylons stacktrace allows remote code execution. Make sure this is false
# if your server is publicly accessible.
debug = false

disable_ads = true
disable_captcha = true
disable_ratelimit = true
disable_require_admin_otp = true

domain = reddit.com
oauth_domain = oauth.reddit.com
https_endpoint = https://reddit.com

disable_wiki = false
uncompressedJS = true

min_membership_create_community = 0

plugins = 

media_provider = filesystem
media_fs_root = /srv/www/media
media_fs_base_url_http = http://%(domain)s/media/

[server:main]
port = 8001
Save using Ctrl-X then y. Then run make ini followed by sudo reddit-restart.

Some differences in the API between the open source version of Reddit and the current version of Reddit break third-party apps, but can easily be fixed using a reverse-proxy. Users of the app and website connect to the reverse proxy server, which then forwards the request to the actual Reddit instance, making any necessary compatibility adjustments to incoming requests and outgoing responses along the way.
On your host machine, download and install/extract the latest version of OpenResty, which is an nginx-based server with Lua scripting support. Update conf/nginx.conf to read as follows:

worker_processes  1;

events {
    worker_connections  1024;
}


http {
    map_hash_bucket_size 512;

    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;


    # remove unsupported OAuth scopes
    map $request_uri $new_uri {
        /api/v1/authorize.compact?client_id=5JHxEu-4wnFfBA&response_type=code&state=RedditKit&redirect_uri=apollo%3A%2F%2Freddit-oauth&duration=permanent&scope=account%2Ccreddits%2Cedit%2Cflair%2Chistory%2Cidentity%2Clivemanage%2Cmodconfig%2Cmodflair%2Cmodlog%2Cmodothers%2Cmodposts%2Cmodself%2Cmodwiki%2Cmysubreddits%2Cprivatemessages%2Cread%2Creport%2Csave%2Csubmit%2Csubscribe%2Cvote%2Cwikiedit%2Cwikiread%2Cmodcontributors%2Cmodtraffic%2Cmodmail%2Cstructuredstyles /api/v1/authorize.compact?client_id=5JHxEu-4wnFfBA&response_type=code&state=RedditKit&redirect_uri=apollo%3A%2F%2Freddit-oauth&duration=permanent&scope=account%2Ccreddits%2Cedit%2Cflair%2Chistory%2Cidentity%2Cmodconfig%2Cmodflair%2Cmodlog%2Cmodothers%2Cmodposts%2Cmodself%2Cmodwiki%2Cmysubreddits%2Cprivatemessages%2Cread%2Creport%2Csave%2Csubmit%2Csubscribe%2Cvote%2Cwikiedit%2Cwikiread%2Cmodcontributors%2Cmodtraffic;
        /api/v1/authorize?client_id=qnjy_qcqie9-Zg&response_type=code&state=RedditKit&redirect_uri=narwhal://oauth&duration=permanent&scope=account,creddits,edit,flair,history,identity,livemanage,modconfig,modflair,modlog,modothers,modposts,modself,modwiki,mysubreddits,privatemessages,read,report,save,submit,subscribe,vote,wikiedit,wikiread /api/v1/authorize?client_id=qnjy_qcqie9-Zg&response_type=code&state=RedditKit&redirect_uri=narwhal://oauth&duration=permanent&scope=account,creddits,edit,flair,history,identity,modconfig,modflair,modlog,modothers,modposts,modself,modwiki,mysubreddits,privatemessages,read,report,save,submit,subscribe,vote,wikiedit,wikiread;
    }

    # HTTPS server
    
    server {
       listen       443 ssl;
       server_name  localhost;

       ssl_certificate      reddit.com+1.pem;
       ssl_certificate_key  reddit.com+1-key.pem;

       ssl_session_cache    shared:SSL:1m;
       ssl_session_timeout  5m;

       ssl_ciphers  HIGH:!aNULL:!MD5;
       ssl_prefer_server_ciphers  on;

       location / {
            if ($new_uri) {
                return 301 $new_uri;
            }

            access_by_lua_block
            {
                -- replace unsupported sorts with similar options
                local requri = ngx.var.uri
                if requri:find("best.json") then
                    ngx.req.set_uri(ngx.re.gsub(requri, "best.json", "hot.json"), false)
                end
                if requri:find("rising.json") then
                    ngx.req.set_uri(ngx.re.gsub(requri, "rising.json", "new.json"), false)
                end
                -- correct type identifiers for posts/comments
                ngx.req.read_body()
                local req = ngx.req.get_body_data()
                local newreq, n, err = ngx.re.gsub(req, "&thing_id=t3_", "&thing_id=t5_")
                newreq, n, err = ngx.re.gsub(newreq, "&id=t3_", "&id=t5_")
                ngx.req.set_body_data(newreq)
            }

            proxy_ssl_name $host;
            proxy_ssl_server_name on;
            proxy_ssl_verify off;
            proxy_set_header Host $host;
            proxy_set_header Accept-Encoding "";
            proxy_pass https://127.0.0.1:8443;

            # correct type identifiers in responses
            header_filter_by_lua_block { ngx.header.content_length = nil }
            body_filter_by_lua_block {
                local body = ngx.arg[1]
                local contenttype = ngx.header.content_type
                if contenttype == nil then contenttype = "" end
                if body and (contenttype:find("application/json")) then
                    body = ngx.re.gsub(body, '"kind": "t3"', '"kind": "AWARD"')
                    body = ngx.re.gsub(body, '"kind": "t4"', '"kind": "SUBREDDIT"')
                    body = ngx.re.gsub(body, '"kind": "t5"', '"kind": "LINK"')
                    body = ngx.re.gsub(body, '"kind": "t6"', '"kind": "MESSAGE"')

                    body = ngx.re.gsub(body, '"kind": "AWARD"', '"kind": "t6"')
                    body = ngx.re.gsub(body, '"kind": "SUBREDDIT"', '"kind": "t5"')
                    body = ngx.re.gsub(body, '"kind": "LINK"', '"kind": "t3"')
                    body = ngx.re.gsub(body, '"kind": "MESSAGE"', '"kind": "t4"')
                end
                ngx.arg[1] = body
            }
       }
    }

}


Create reddit.com+1.pem and reddit.com+1-key.pem using mkcert. Download mkcert and run mkcert -install followed by running mkcert reddit.com "*.reddit.com" in your conf directory.


On your host, update your HOSTS file or use NextDNS/Pi-hole to point reddit.com and *.reddit.com your host IP address (127.0.0.1 will work to start). (The specific domains that need to be pointed are reddit.com, www.reddit.com, oauth.reddit.com, and ssl.reddit.com). (Revert this change when you want to connect to the real Reddit).


From your server install directory on your host, start the reverse proxy server by running nginx (on Windows it will appear to freeze and nothing will be output, this is normal). (You can stop the reverse proxy server by running nginx -s quit and reload config by running nginx -s reload in a different terminal window).


Now we will create the client IDs for the third party Reddit apps. On the host, go to https://reddit.com/prefs/apps/ and create the admin account. Make an account with the username reddit and any password. If you get a 500 error on signup, just try logging in anyway, it should work. Reddit normally generates client IDs randomly, but we need it to generate a specific client ID in order for third party apps to work unmodified. To do this, you'll need to make a temporary change to the Reddit source code in the VM and be sure to revert it once the app is created. To specify a client ID, you need to change line 58 of ~/src/reddit/r2/r2/models/token.py from kwargs["_id"] = cls._generate_unique_token() to kwargs["_id"] = 'CLIENT_ID_HERE' (replace CLIENT_ID_HERE with the appropriate value), then restore it to what it was when you're done. You can make this edit using nano, and do sudo reddit-restart to make sure the change is applied. Then, back on the https://reddit.com/prefs/apps/ webpage, create the app as an Installed App. The values for a few popular iOS apps are as follows:

Apollo: client ID: 5JHxEu-4wnFfBA, redirect URI: apollo://reddit-oauth
Narwhal: client ID: qnjy_qcqie9-Zg, redirect URI: narwhal://oauth
BaconReader: client ID: up6Arjatrs1nzw, redirect URI: baconreader://redditauth


(I did not omit the client IDs because Reddit does not consider them to be private information; Reddit emails the user this information every time they log into one of these apps.)
You can also find these values in the URL of the Reddit login page for a given app. I found that the app creation worked best from the admin account.
Connecting a mobile device to the custom Reddit instance


I would recommend logging out of any mobile apps before beginning this process.


First, you'll need to install the root certificate authority used to sign your fake reddit.com SSL certificate onto your mobile device. (HTTPS certificates are built on a chain of trust, if you don't install the certificate your device will correctly not trust the server that is pretending to be reddit.com).

For the next step, you'll need your host IP address/hostname. You can find your host IP using ipconfig on Windows, or ifconfig on some other platforms. (Tailscale could make this easier. If you do use Tailscale, be sure to update your Tailscale settings in the web admin console to use the custom NextDNS configuration.) On your host, you can find the certificate by running mkcert -CAROOT.
On iOS, you can install the certificate from Safari by clicking on a link that leads to the .pem file. The easiest way to do this is navigate to the folder containing your certificate on the host, run python -m http.server 8000 and then in Safari, navigate to http://[your host ip]:8000/ and click the link to the .pem file (not the key). You may also alternatively be able to download the certificate as an attachment in the iOS Mail app. After that, open iOS settings, go to General>VPN, DNS, & Device Management, and install the downloaded configuration profile. Finally, go to Settings>General>About>Certificate Trust Settings and turn on trust for the installed certificate.
Instructions will differ for Android.


Set up custom DNS to point your mobile device to the custom Reddit server.

This can most easily be done using NextDNS with a custom configuration, and should also be possible using Pi-hole.
Make an account on https://my.nextdns.io/, create a configuration and go to the Settings tab, and add a rewrite for reddit.com and set the answer to the IP address (or hostname if using Tailscale) of the host that you are running your reverse proxy server on.
If you use Apollo, I also recommend going to the denylist tab and blocking apolloreq.com to prevent the queries being made to the custom server from being counted towards the developer's API usage statistics tracking.


Activate the DNS, either by connecting to Tailscale or using the NextDNS app with the custom configuration specified.


Because the default configuration disallows admins/employees from logging into third-party apps, create a new user with any username to use within the apps.


Open and optionally log into your desired app.


The app should be mostly usable.

Browsing, posting, commenting, upvoting, etc. should mostly work.
In-app Modmail will not work because of missing APIs, but many other moderation features appear to be working.
Search may be possible to configure but I couldn't get it to work; see instructions in the Reddit source archive.
Support for sending emails (confirmation, password reset, newsletter, etc.) is not configured.
Because best sort and rising sort did not exist in the open source version of Reddit, the reverse proxy replaces best sort with hot sort and rising sort with new sort.
The reverse proxy also corrects some type IDs in the API to better match what the apps are expecting; for this proof of concept this is essentially implemented as a find and replace which could possibly mess up content in a few rare scenarios, though this seems unlikely.
Some apps will not immediately be able to log in and will present an error on the authorization page because they request an invalid scope that was not present in the open source version of Reddit. You can fix this by adding an entry in the map in nginx.conf for the reverse proxy that redirects the login URL used by the app to one that does not include the invalid scopes, and then reloading the server. See the map section of nginx.conf which provides examples for Apollo and Narwhal; BaconReader did not request any of the invalid scopes and thus did not require an entry. The scopes that would need to be removed (if present) are structuredstyles, modmail, livemanage, and modnote.


To connect to real Reddit instead, simply log out and close your app then switch off NextDNS/Pi-hole/Tailscale. (You can leave the certificate installed for convenience. On iOS, you can turn off trust if you'd like.)


A request for third-party app developers

Before shutting down your apps, please consider pushing a simple app update to enable specifying a custom client ID and Reddit API domain, as this would greatly simplify setup for allowing users to connect to real Reddit with a custom client ID or to connect to a custom Reddit server.