Every HTTP client announces itself. A browser sends a long string naming its engine, version and operating system; libraries send something short and honest, such as python-requests/2.32.3 or curl/8.5.0. Sites that filter automation start with this header, because a library default is the cheapest signal there is.
See what you send
curl -s https://httpbin.org/user-agent
curl -s -A "$UA" https://httpbin.org/user-agent
The first prints curl's default; the second sends whatever string you put in $UA. Copy a current one from a real browser (it is shown in the request headers in developer tools) rather than from an old list. In Python requests, set it in the session headers so every request carries it.
Why a browser string is not enough
Changing the User-Agent only changes a claim. The site can check it against the TLS fingerprint, the HTTP/2 fingerprint, the other headers and their order, and, in Chromium browsers, the Sec-CH-UA client hints. A Chrome User-Agent over Python's TLS handshake is more suspicious than an honest library string, because nothing about it adds up.
Practical rules
- Match the rest of the request to the claim, or use a client that impersonates the browser fully.
- Keep one User-Agent per session. Changing it on every request, while the cookies stay the same, looks like one client lying.
- Keep it current. A browser version from years ago stands out.
- For a crawler that wants to be identified, send an honest name with a contact URL; some sites allow those.
Common confusion
A proxy does not change your User-Agent. For HTTPS sites, the header travels inside the encrypted tunnel and reaches the site exactly as your client wrote it. How to avoid getting blocked covers headers alongside the other layers.