- Install fiddler
winget install -e --id Telerik.Fiddler
- Open fiddler and go to
Tools -> Options -> HTTPS
- Enable
Decrypt HTTPS traffic
- Click the
Actions
button and selectExport root certificate to desktop
- Right click the
FiddlerRoot.cer
file on the desktop and clickOpen with -> Crypto Shell Extensions
- In the Certificate window that opens up go to
Details -> Copy to File
- Click
Next
then selectBase-64 encoded X.509 (.CER)
then specify the file name (E.G. FiddlerRootBase64.cer) - Click
Next
to create the new file - Open the file and copy the contents to the clipboard
- Find out where python is storing its certificate authority certs. For newer version of the python you can run the following commands
pip install certifi
python -c "import certifi; print(certifi.where())"
- Open the file where the CA certs are and paste the fiddler base64 certificate contents to the end of the file
The easiest way to do this is to have a block of code in the entry point of your application that sets some environment variables that the python requests
module will use
import os
os.environ['HTTP_PROXY'] = 'http://127.0.0.1:8888'
os.environ['http_proxy'] = 'http://127.0.0.1:8888'
os.environ['HTTPS_PROXY'] = 'https://127.0.0.1:8888'
os.environ['https_proxy'] = 'https://127.0.0.1:8888'
You can also configure the proxy via code in the requests module if preferred
If you get any check_hostname requires server_hostname
when using the proxy try updating changing the version of the urllib3
python module you're using https://stackoverflow.com/questions/66642705/why-requests-raise-this-exception-check-hostname-requires-server-hostname
I was looking to sniff python requests in Fiddler. A fair bit of experience in python, but new to Fiddler. Intermediary applications like Fiddler are useful for inspecting request headers/cookies without having to rely on the server response; for instance, https://httpbin.org/headers will capitalize headers according to RFC outline, but what if you want to see the exact headers you sent to a website? What if some anti-bot is specifically looking for header order or case sensitivity?
I wanted to test if a specific HTTP Adapter in requests was ordering headers properly and sending with case sensitivity. Later, I wanted to test the same in Scrapy.
Initially, I was able to get this working by following a documentation guide here.
The problem was I kept having to add
verify=False
to every request I made, even if I wasn't using Fiddler proxy.Using the steps outlined above, I was able to make requests without having
verify=False
. I wasn't able to get this to work on Ubuntu on Windows (WSL), but I was on macOS with no additional trouble shooting required (M1 chip, Apple Silicon).Thank you!