Debugging HTTP traffic with mitmproxy

README.md

This will let you see the request and response headers for traffic going through.

We're going to run this as a reverse proxy, rather than a usual proxy, so you don't get completely flooded with traffic.

Start the proxy

1. create a new VM
2. expose port 8080 to the public internet
3. SSH to the VM
4. make sure you have at least python 3.6
5. install mitmproxy with pip

   sudo -H pip3 install mitmproxy

6. create a python script that will be run for each request. Use the script.py file in this gist.
7. define an environmental variable with the endpoint you wish to proxy

   export THE_ENDPOINT=https://example.com:80 # change me

8. find out your public network interface address

   ifconfig

9. create an env var with the public IP:

   export THE_INTERFACE=11.22.33.44 # change me

10. start mitmproxy

    mitmdump \
      --listen-host $THE_INTERFACE \
      -p 8080 \
      --mode reverse:$THE_ENDPOINT \
      --set block_global=false \
      -s /path/to/script.py

Configure your client

Now you need to configure your client. Remember, this is not a proxy server, it's a reverse proxy so just change your code/config to use the IP and port of the VM instead of your endpoint.

Then you trigger your client and you should see see the traffic in the console on the VM as mitmproxy is running in the foreground.

Example output

==================================================
GET /preheat HTTP/1.1
------------------------- request headers -------------------------
ACCEPT-ENCODING               : identity
X-REQUESTED-WITH              : XMLHttpRequest
ACCEPT                        : application/json
USER-AGENT                    : Appcelerator Titanium/7.2.0 (Nexus 5; Android API Level: 27; en-AU;)
HOST                          : example.com
CONNECTION                    : Keep-Alive
------------------------- response headers -------------------------
CONTENT-TYPE                  : application/json; charset=utf-8
CONTENT-LENGTH                : 27
CONNECTION                    : keep-alive
DATE                          : Wed, 30 Jan 2018 03:05:42 GMT
VARY                          : Accept-Encoding
X-POWERED-BY                  : up
ETAG                          : W/"1b-SWSBjMAAv99BCA8zPdG9jaVeqEs"
------------------------- body (first 100 bytes) -------------------------
some body stuff...
118.210.255.140:47127: GET https://example.com/preheat
                    << 200 OK 27b

script.py

def response(flow):
    print("")
    print("="*50)
    print(flow.request.method + " " + flow.request.path + " " + flow.request.http_version)

    print("-"*25 + " request headers " + "-"*25)
    for k, v in flow.request.headers.items():
        print("%-30s: %s" % (k.upper(), v))

    print("-"*25 + " response headers " + "-"*25)
    for k, v in flow.response.headers.items():
        print("%-30s: %s" % (k.upper(), v))

    print("-"*25 + " body (first 100 bytes) " + "-"*25)
    print(flow.request.content[0:100])