Project: Improving support for DNS and HTTP3 in mitmproxy
Contributor: Gaurav Jain
Organization: The Honeynet Project
Mentor: Maximilian Hils, Emanuele Micheletti
mitmproxy
is a fantastic tool to intercept, inspect, modify and replay a wide range of network traffic including SSL/TLS-protected protocols. It can be used for debugging, testing, privacy measurements, and penetration testing. The tool is mostly based on python, rust and typescript. The goals of this project was to debug and fix many missing/broken bits in DNS and HTTP3 modes.
- Strip DNS HTTPS records to prevent clients from sending Encrypted ClientHello (ECH) messages
- First step - Block HTTPS records entirely: mitmproxy/mitmproxy#6876
- Parse HTTPS records: mitmproxy/mitmproxy#6884
- Update addon to parse HTTPS records and strip the ECH value alone: mitmproxy/mitmproxy#6905
- Use existing API for parsing domain names in HTTPS records in an effort to reduce code reuse: https://github.com/mitmproxy/mitmproxy/pull/6949/files
- Add support for DNS over TCP flows
- Make mitmproxy listen on TCP ports for DNS mode: mitmproxy/mitmproxy#6912
- Parse DNS over TCP messages: mitmproxy/mitmproxy#6935
- Support proxying all query types in DNS mode
- Use rust's
hickory_resolver
for lookups and extracting system's name server: mitmproxy/mitmproxy_rs#148, mitmproxy/mitmproxy_rs#158 - Use the rust API for lookups that require checking the hosts file, forward other requests to the server specified in the options, thus supporting all DNS query types: mitmproxy/mitmproxy#6975
- Fix a crash on startup when mitmproxy is unable to determine the OS' DNS servers: https://github.com/mitmproxy/mitmproxy/pull/7066/files
- Use rust's
- Fix bug in DNS message parsing issue where compression pointers were not handled properly
- Debug and document existing issues related to HTTP3: mitmproxy/mitmproxy#7025
- Handle fragmented QUIC client hello: mitmproxy/mitmproxy#7067
- Addon to update the target host in
alt-svc
HTTP headers in reverse proxy mode: mitmproxy/mitmproxy#7093 - Add HTTP3 support in HTTPS reverse-proxy mode: mitmproxy/mitmproxy#7114
- Add
HttpConnectedHook
andHttpConnectErrorHook
: mitmproxy/mitmproxy#6930 - Allow
typing.Sequence[str]
to be an editable option: mitmproxy/mitmproxy#7001 - Handle errors when remote host has closed UDP socket: mitmproxy/mitmproxy_rs#161
- Turned out to be an issue with rust's stdlib: rust-lang/rust#128072
- Set
IPV6_V6ONLY
flag for UDP sockets: mitmproxy/mitmproxy_rs#167, mitmproxy/mitmproxy#7084
- Autofix generated JS files and do not patch them in tests: mitmproxy/mitmproxy#6910
- Optimize
response/request_body_buf
by using list of byte chunks to avoid concatenation overhead: mitmproxy/mitmproxy#6952 - Guard
server_event
against reentrancy: mitmproxy/mitmproxy#7031 - Update
next_layer
to detect HTTP3 flows: mitmproxy/mitmproxy#7037 - Update
next_layer
to detect SSH flows as non-HTTP: mitmproxy/mitmproxy#7041 - Improve UX when users specify invalid certs: mitmproxy/mitmproxy#7073
- QUIC: Support
STOP_SENDING
frames: mitmproxy/mitmproxy#7119
- Speedup tests for
dns_resolver
: mitmproxy/mitmproxy#7115
mitmproxy
now supports
- proxying all query types in DNS flows
- proxying DNS over TCP flows
- stripping
ech
value from HTTPS type DNS records - proxying compressed DNS records
- HTTP3 based reverse proxy and local mode albeit a few issues
- listening on IPv4 and IPv6 for UDP based servers
- Server QUIC handshake takes 10x more time on windows as compared to linux: mitmproxy/mitmproxy#7033
- HTTP3 over local mode: Unable to access certain websites using firefox via HTTP3: mitmproxy/mitmproxy#7117
- Proxying raw QUIC flows: mitmproxy/mitmproxy#7038
- Handle exposing
HttpConnectErrorHook
: mitmproxy/mitmproxy#7083
This entire journey wouldn't be as rewarding if it weren't for all the frustrating challenges that I've faced on the way. For me, the most difficult and time-consuming part of the project was debugging issues. This usually involved a lot of aimless googling and falling deep into rabbit holes. There'd be times when even after spending countless hours, no real progress was made. I want to thank my mentor here for being patient enough and guiding me in the right direction. This process, I feel, has greatly helped me develop a "never give up" and "try harder" attitude for fixing bugs.
Understanding and writing asynchronous rust code with python bindings has been another difficulty that I had faced during the project. But on the brighter side of things, I'm definitely a bit more confident with rust now and learnt a LOT of new stuff - from understanding protocols to writing asynchronous code. I also had some difficulty managing my time once my college started in August. Juggling these different pots really started to take a toll on my sleep schedule but on the flip side all those sleepless nights were definitely worth it :P
Thanks a ton to GSoC and The Honeynet Project for this wonderful opportunity, these past 3 months will be something that I'll remember for quite a long time. This experience has no doubt greatly helped me develop my technical and communication skills and made me fall in love with open source a bit more. I'd also like to thank my mentor Maximilian Hils for being extremely friendly and for making sure that I had a smooth and enjoyable experience throughout. I hope to continue to staying active and contributing to this exciting world of open source in the future as well.