Web scraping
Proxies for web scraping
Most crawls stall on blocks and per-IP rate limits long before the parser has trouble. Spread requests over a rotating residential pool, keep a few static IPs for the sources that want a fixed origin, and pay for the traffic you use.
Where it breaks
What goes wrong first
Targets count requests per address. Send a crawl from a short list of IPs and every rate limiter on the other end sees the same few callers asking for page after page.
- Per-IP rate limits that stall a crawl halfway through a catalogue
- Ranges already flagged as automated, refused before the first page loads
- Pages that change with visitor location, so one origin records one version
- Retries that burn traffic on addresses the target has stopped answering
How the hive handles it
Built for web scraping
Rotation across consumer connections
Residential traffic rotates across household connections, so a long crawl is spread over many origins instead of queuing behind a handful. You pay per gigabyte, and what you leave unspent stays on the account.
Static IPs for the stubborn sources
Some sources answer only an allowlisted address, or break when the origin changes mid-session. ISP and datacenter IPs are yours for the term you pick, placed in the country you choose at checkout.
Priced for bursty work
Crawls come in waves. Traffic you buy never expires, so a quiet month does not eat the balance, and the rate you bought at holds for 12 months.
The rules come first
Public data collection is on the allowed list, published before you pay. Keep within the target’s rate limits and you are inside the policy.
What we would buy
The line to start on, and the one to fall back to
Prices come from the catalogue checkout reads. No geo surcharge and nothing added at the payment step.
First choice
from $1.75 / GB
Residential
Rotating residential IPs, priced per gigabyte, no expiry.
Rotation across consumer connections keeps a broad crawl moving when the target throttles per address.
- One unit at checkout
- $5.50 / GB
- Location at checkout
- Rotating, no location picker
Fallback
from $1.50 / IP
Datacenter
Your own datacenter IPs, rented by the IP for a fixed term.
For sources that tolerate datacenter ranges, a static IP priced per address moves the same pages without a gigabyte meter running.
- One unit at checkout
- $3.20 / IP
- Location at checkout
- By country, region and city
Coming soon: Scraper API. The Scraper API will take a URL and hand back the HTML, billed per thousand requests, for teams that would rather not run rotation themselves. It is not on sale yet and carries no published price.
A typical setup
From signup to steady state
Step 1
Point the crawler at the gateway with your residential credentials
Step 2
Let listing pages rotate, and move fixed-origin sources onto a static IP
Step 3
Track block and retry rates per target, and switch lines when a source starts refusing
Step 4
Top up when the balance runs low; whatever is left carries over
Rules before purchase
Allowed
Public web data collection
Prices, product listings, reviews, availability, job postings, public company data: anything a logged-out visitor can see. Respect the target’s rate limits and do not degrade their service.
The whole allowed-use policy12-month rate lock
Your rate holds from the day you buy, and follows the list price down.
Traffic never expires
No monthly reset. What you buy waits until you spend it.
Questions
Web scraping, answered
Can I pick the country residential requests come from?
No. Residential is sold as rotating coverage across the network, with no country picker at checkout. When a source needs a particular country, order an ISP or datacenter IP and choose the country as you buy.
What happens to traffic I do not use?
It stays on your account. Residential traffic has no expiry date and no monthly reset.
Is scraping public pages allowed?
Collecting what a logged-out visitor can see is on the allowed list, as long as you respect the target’s rate limits and do not degrade its service. Personal data still needs a lawful basis, and paywalled content is out of bounds. Read the allowed-use policy before you scope a job.
When is datacenter the better buy for scraping?
When the target does not care what kind of network a request comes from. Many public APIs, documentation sites and smaller catalogues fall in that group. Test a sample first, and move the source to residential if the block rate climbs.
Is there a minimum order?
No minimum order. The smallest top-up is $10, and a new account starts with a 1 GB trial.
Start free
Start web scraping on Residential
Sign up, configure Residential, and pay what the configurator showed. The smallest top-up is $10 and there is no minimum order.