mitmproxy upstream proxy: inspect your scraper's traffic on its way through the provider

mitmproxy is an open-source intercepting proxy you run on your own machine. In upstream mode it decrypts what your scraper sends, shows it to you, then forwards it to a second proxy, so the request still leaves from your provider's IP. Start it with mitmdump --mode upstream:http://HOST:PORT --upstream-auth USERNAME:PASSWORD, point your client at http://127.0.0.1:8080, and make the client trust mitmproxy's CA certificate.

Every command and error below came from mitmproxy 12.1.2 (the official mitmproxy/mitmproxy Docker image) chained to a local HTTP proxy that requires a username and password. mitmproxy is MIT licensed.

#Start mitmproxy in upstream mode

mitmdump --mode upstream:http://HOST:PORT --upstream-auth USERNAME:PASSWORD

Or in Docker, keeping the CA in your home directory so clients trust it across restarts:

docker run --rm -it -p 127.0.0.1:8080:8080 \
  -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy \
  mitmproxy/mitmproxy:12.1.2 \
  mitmdump --mode upstream:http://HOST:PORT --upstream-auth USERNAME:PASSWORD

Use mitmproxy instead of mitmdump for the interactive console, or mitmweb for a browser UI. Two constraints:

• The upstream must be an HTTP proxy. --mode upstream:socks5://... stops at startup with Invalid server scheme: socks5.
• --upstream-auth is Basic authentication, added to every request and CONNECT mitmproxy sends upstream.

On a rotating residential pool, every request can leave from a different exit, which makes before-and-after comparisons noisy. Name a sticky session in the username (the dashboard shows the format for your account) and the exit holds while you debug.

#Trust the mitmproxy CA

mitmproxy creates its CA on first start, in ~/.mitmproxy/. Without it, clients refuse the certificates it issues: curl fails with curl: (60) SSL certificate problem: unable to get local issuer certificate, Python requests with CERTIFICATE_VERIFY_FAILED. Trust it per client rather than system-wide:

curl -x http://127.0.0.1:8080 --cacert ~/.mitmproxy/mitmproxy-ca-cert.pem \
  "https://api.ipify.org?format=json"

HTTPS_PROXY=http://127.0.0.1:8080 \
REQUESTS_CA_BUNDLE=~/.mitmproxy/mitmproxy-ca-cert.pem python scraper.py

NODE_EXTRA_CA_CERTS=~/.mitmproxy/mitmproxy-ca-cert.pem node scraper.mjs

We ran the Python line with requests 2.34 and the Node line with Node 22 and undici's ProxyAgent. A browser set to use the proxy can fetch the certificate from http://mitm.it instead.

The JSON from ipify should show the IP of the upstream exit, not yours. In curl's verbose output the issuer reads CN=mitmproxy; O=mitmproxy, which is how you know decryption is on.

#Log only the requests that fail

A scraper sends thousands of requests and you care about the few that get blocked. Save this as blocked.py:

import logging

from mitmproxy import http

HIDDEN = {"cookie", "authorization", "proxy-authorization"}


def response(flow: http.HTTPFlow) -> None:
    if flow.response.status_code in (403, 407, 429):
        sent = {k: v for k, v in flow.request.headers.items() if k.lower() not in HIDDEN}
        logging.warning("%s %s sent=%s", flow.response.status_code, flow.request.pretty_url, sent)

Run it with mitmdump --flow-detail 0 -s blocked.py plus the upstream flags. Each blocked request prints with the headers your client sent. Skip -q: it silenced the addon's warnings in our test. Filter proxy-authorization as shown, because in upstream mode plain-HTTP flows carry the upstream credentials as a header and would otherwise land in your logs.

To leave a host undecrypted, add --ignore-hosts with a regular expression; matching connections are tunnelled straight through to the upstream.

#What decryption changes

mitmproxy terminates TLS and opens a new connection to the site, so the target talks to mitmproxy's TLS stack:

• The TLS fingerprint changes. Against a test server, curl alone offered one list of key-exchange groups; through mitmproxy the server saw a different list, mitmproxy's own. A target that scores TLS fingerprints can answer differently with mitmproxy in the path, so a block that appears only while debugging may be about mitmproxy. TLS fingerprinting with curl_cffi covers why that matters.
• Pinned clients fail. curl --pinnedpubkey with the site's real key failed through mitmproxy with curl: (90) SSL: public key does not match pinned public key, and passed through the plain upstream proxy. Apps that pin certificates behave the same way.
• Headers are left alone. mitmproxy added no request headers in our test; anything you see added came from the upstream or from an addon.

#Troubleshooting

• 502 on HTTPS, Upstream proxy HOST:PORT refused HTTP CONNECT request in the log. The upstream rejected the tunnel; the status after it says why, a 407 for wrong credentials.
• The upstream's 407 or 401 page on plain HTTP. Same cause, shown directly.
• --ssl-insecure turns off upstream certificate checks. Use it only against your own test servers.

#With ProxyHive

Debug on the pool your scraper runs on. Residential proxies authenticate with username and password and offer HTTP, which is what upstream mode needs, and the free 1 GB on residential covers a debugging session. The HTTP debugging proxies comparison weighs it against Charles, Fiddler, Burp and ZAP.