Building a Web Scraper That Survives: Anti-Bot Evasion Tactics

At 9:42 PM on a Tuesday, DoxDock hit production for the first time. By 9:47 PM, 47 pages in, Cloudflare had slammed the door. The request logs showed the pattern clearly: 200, 200, 200, 200, then a 403, then a challenge page that pretended to be a 503 while running a JavaScript proof-of-work that never resolved. Five minutes. Four hundred and seventy records. Zero survivors.

The infuriating part was that the HTTP stack was perfect. Retry logic with exponential backoff. A rotating proxy pool with session stickiness. Robust parsing on the response side. Every line of network code was sound, and none of it mattered. The block never fired at the HTTP layer. It fired at the browser layer — a combined fingerprint score that told Cloudflare, on the first page load, that the session was automation. The browser was flagging itself before it even asked for a single page.

That failure launched a three-month rebuilding process that produced the current version of DoxDock. This article is the post-mortem: the anti-bot tactics that survived production contact, the ones that wasted our time, and the architecture we landed on. The core lesson is simple, but it changed everything: anti-bot systems are not gatekeepers to be beaten. They are statistical classifiers to be satisfied. Your scraper lives or dies based on how far its behavior deviates from a human's.

The Anti-Bot Arms Race Is a UX Problem

Every serious anti-bot product — Cloudflare Turnstile, Akamai Bot Manager, DataDome, Human Security — is doing the same thing under the hood: computing the probability that the session is human, based on tens of thousands of behavioral and environmental signals. The scores change constantly. The detection surface changes with every Chrome release. But the architecture is consistent: it's a classifier, not a rule set.

This reframes everything. A human is a distribution of behaviors, not a fixed profile. A human scrolls unevenly, hesitates before clicks, gets interrupted, leaves the tab half-read. A bot is a delta function: deterministic timing, zero variance, identical every time. The classifier's job is to compute how improbable your session is. If the session sits in a low-probability region of the "human" distribution, the challenge fires.

Two implications fall out of this. First: you don't win against anti-bot systems by being cleverer at HTTP. You win by being boring. An identity that quietly passes 5,000 requests over three days is worth more than a burst that yanks 50,000 pages in an hour, even if the burst is technically "undetected." Second: no single tactic matters. Every field — the TLS fingerprint, the proxy IP, the browser object, the scrolling pattern — is one feature in a giant probability model. Patch one and the model still has 9,000 other dimensions to catch you.

Fingerprinting: The First Wall You Never See

Long before the first real request, the target has already sent you a page. That page runs a script that reads everything: navigator.webdriver, navigator.plugins, canvas rendering, WebGL vendor, audio context behavior, screen orientation, device pixel ratio, timezone. Playwright leaks on nearly every one of these by default. The most damning leak is navigator.webdriver, which is true unless you explicitly remove it.

The fix is to inject your patches before any page script runs. Playwright calls this an init script, and it executes in the page's main world with timing that beats anything the target can attach.

from playwright.sync_api import sync_playwright

def install_stealth(context):
    """Patch the highest-signal automation leaks before the page loads."""
    context.add_init_script("""
        // The single most important flag: automation is exposed as true
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });

        // Real plugins are sparse; the Chrome PDF plugin is always present
        Object.defineProperty(navigator, 'plugins', {
            get: () => [
                {name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer'},
                {name: 'Chrome PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai'}
            ]
        });

        // Humans report the languages their OS was configured with
        Object.defineProperty(navigator, 'languages', {
            get: () => ['en-US', 'en']
        });

        // Mimic Chrome's automation-controlled flag being disabled
        Object.defineProperty(navigator, 'maxTouchPoints', {
            get: () => Math.random() < 0.3 ? 5 : 0
        });
    """)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    install_stealth(context)
    page = context.new_page()
    page.goto("https://target.example/data")

The temptation is to patch a hundred more properties. Don't. Real browsers are inconsistent with each other, and over-patching creates a fingerprint that no real Chrome has. The winning move is to patch the known leaks, then let the real browser supply the rest of its authentic fingerprint. If you fake the WebGL vendor, you have to fake it consistently across every context. One place it isn't — a WebGL call that returns different output than the UA string claims — and you've created a unique, permanent bot signature.

Playwright Stealth: The 80/20 of Not Looking Like a Bot

The npm package playwright-stealth is a good starting point and a terrible production dependency. It goes stale within a month of every Chrome release, and its aggressive patches collide with real browser behavior. We replaced it with a list of five small init scripts, one per browser context, and we maintain the list ourselves by reading the failure logs.

The 80/20 split came out like this: patching navigator.webdriver, plugins, and languages gets you past basic checks. Surviving hard targets like DataDome requires getting the rendering pipeline right — which is why we stopped fighting headless mode entirely.

Headful under Xvfb

We run headful Chromium under Xvfb on production machines. It costs about 300 MB of RAM per identity and we run seven identities per box. It is more expensive than headless, but it buys us one decisive property: no rendering-path difference. Headless mode — even the new --headless=new — renders through a different pipeline, and sophisticated fingerprinting can detect the discrepancy in canvas and WebGL output.

The desktop comparison

| Strategy | Setup cost | Detection risk | Throughput | Notes | |---|---|---|---|---| | Headless (default) | Free | High on strict sites | Highest | Distinct rendering pipeline; canvas and WebGL diverge from real Chrome | | Headful (local display) | Low | Low | Good | Requires a display server; fine for laptops, bad for servers | | Headful (Xvfb) | Medium | Low | Good | Our production choice; ~300 MB per context | | Disguised headless (--headless=new) | Free | Medium | High | Passes naive checks, leaks in JS heap and GPU behavior |

One more lesson: do not run in --single-process mode or with --disable-gpu. Both flags are known automation tells, and they also push the browser's rendering behavior further from a normal user's Chrome, which raises the classifier's suspicion on every subsequent signal.

Rotating Identities, Not Just IPs

Every anti-bot vendor computes geo-consistency. If the IP resolves to Dallas, the timezone had better be America/Chicago, and the language list had better include English. An IP that changes every request while the browser claims an English-only user agent from Berlin is the kind of impossible combination that flags instantly.

IP rotation is table stakes. Identity rotation is the actual defense. DoxDock creates a full browser identity per profile: user agent, viewport, timezone, language list, hardware concurrency, and a proxy in the same geography. The identity is bound to a persistent browser profile and an entry in a SQLite ledger that tracks every state change.

CREATE TABLE identity_pool (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    profile_path TEXT NOT NULL,
    user_agent   TEXT NOT NULL,
    viewport     TEXT NOT NULL,          -- '1440x900'
    timezone     TEXT NOT NULL,          -- 'America/New_York'
    language     TEXT NOT NULL,          -- 'en-US,en'
    proxy_host   TEXT NOT NULL,
    proxy_port   INTEGER NOT NULL,
    status       TEXT DEFAULT 'cold',    -- cold | warming | hot | banned | cooldown
    captcha_count INTEGER DEFAULT 0,
    successful_scrapes INTEGER DEFAULT 0,
    last_used_at  DATETIME,
    cooldown_until DATETIME
);

CREATE INDEX idx_status ON identity_pool(status);
CREATE INDEX idx_cooldown ON identity_pool(cooldown_until);

The ledger is not optional decoration. Without it, the scraper finds itself reusing a banned profile, or putting a hot proxy on a cold identity and triggering a challenge-rate spike. We chose SQLite because the query pattern is simple, writes are rare, and there is no network dependency to fail.

Human Behavior Modeling: Timing, Jitter, and Scroll Patterns

Deterministic behavior is the second pillar of detection, right after the fingerprint. A scraper that clicks and types with perfect metronomic timing stands out even when the fingerprint is genuinely human-adjacent. We built a small behavior module for DoxDock that models three human behaviors: typing, scrolling, and reading.

import random
import time


class Humanizer:
    """Inject human timing into keyboard, mouse, and scroll behavior."""

    def type_text(self, page, selector: str, text: str) -> None:
        page.click(selector)
        for ch in text:
            page.keyboard.type(ch)
            # Typists vary: 30-120ms per key with a longer pause ~4% of the time
            delay = random.uniform(0.03, 0.12)
            if random.random() < 0.04:
                delay = random.uniform(0.30, 0.80)
            time.sleep(delay)

    def scroll_to(self, page, selector: str) -> None:
        box = page.locator(selector).bounding_box()
        # A wheel scroll has momentum; never jump directly to the node
        for _ in range(random.randint(8, 20)):
            page.mouse.wheel(0, random.uniform(120, 420))
            time.sleep(random.uniform(0.06, 0.22))
        page.mouse.click(box["x"] + box["width"] / 2, box["y"] + box["height"] / 2)

    def reading_pause(self, seconds: float = 4.0) -> None:
        # Humans read; bots rush. Pause like a person skimming a paragraph.
        time.sleep(max(0.4, random.gauss(seconds, seconds * 0.25)))

The scroll function is the most important piece. Calling scrollIntoView() is an instant tell because no human produces a single jump event. A wheel-event series with approximate durations is the behavioral approximation of a thumb on a trackpad — and, crucially, it costs nothing to implement.

We added one global rule: never issue two autonomous navigation events within the same second. And we deliberately leave some pages half-read. Perfect, exhaustive behavior is itself a bot signal, because real users abandon pages constantly.

The CAPTCHA Threshold: When to Fight, When to Retreat

We tried CAPTCHA-solving services in week two. The economics looked good on paper: two to four dollars per 1,000 solves, a fraction of what residential bandwidth costs. The economics forgot one thing: every solve trains the vendor's classifier that the IP-plus-browser combination is resolvable. Solve a CAPTCHA from a flagged identity and you just taught the system that the identity responds to challenges. The next CAPTCHA arrives sooner, from a higher-confidence model.

In production, we treat CAPTCHAs as a circuit breaker, not a puzzle. Each identity has a threshold: two CAPTCHAs in an hour puts it into cooldown for six hours. A third failure in 24 hours retires the identity for static analysis.

from dataclasses import dataclass, field
from datetime import datetime, timedelta


@dataclass
class IdentityState:
    id: str
    captcha_events: list[datetime] = field(default_factory=list)

    def observe_captcha(self, now: datetime) -> bool:
        """Returns True if the identity should be cooled down."""
        self.captcha_events = [t for t in self.captcha_events if t > now - timedelta(hours=1)]
        self.captcha_events.append(now)
        return len(self.captcha_events) >= 2

    def can_retry(self, now: datetime) -> bool:
        last = max(self.captcha_events, default=now - timedelta(hours=1))
        return now - last > timedelta(hours=6)

state = IdentityState(id="profile-07")
if state.observe_captcha(datetime.utcnow()):
    print("cooling down profile-07 for 6 hours")

The threshold numbers came from measurement. We watched the challenge-rate distribution across forty identities for a week and found that a single CAPTCHA was common noise, but two in an hour predicted a ban within three hours with near certainty. Six hours of cooldown is enough for the risk score to decay on most vendors; retrying sooner just adds more evidence to the ban decision.

Proxy Architecture That Doesn't Fall Apart

The naive approach — rotate the proxy on every request — makes detection worse. Each new IP resets DNS caches and connection pools, producing a burst of fresh TLS connections. That pattern is itself a bot signature. We use sticky sessions: an identity keeps the same proxy for the duration of its life, and we only rotate when the proxy dies or the identity is banned.

We use residential proxies in production. Datacenter proxies are cheap and we still use them for development, but they burn out within minutes on any strict site. The table reflects what we measured across two months of scraping a set of moderately protected targets:

| Proxy type | Cost / GB | IP churn | Relative challenge rate | Best use | |---|---|---|---|---| | Datacenter | $0.50-1.00 | Very high | High | Dev, staging, unprotected targets | | Residential (rotating) | $5-15 | High | Moderate | General production, high page volume | | Residential (sticky) | $8-20 | Low | Low | Heavy fingerprinting, login-gated sessions | | Mobile | $15-30 | Low | Very low | Geo-specific content, app stores |

Warming a cold IP

A fresh IP from a residential pool is immediately suspicious if it opens with a burst of traffic. We warm every new proxy: five requests over ten minutes on a harmless endpoint, then a slow ramp to full rate. Warming costs time but saves lives — identity status goes cold → warming → hot, and the hot state is the only one allowed to drive real volume. The cold column in the identity table is not decorative; it gates all traffic.

Logging, Metrics, and the Post-Mortem Habit

The most valuable code in DoxDock is not the scraper. It is the logging layer. Every response — good or bad — is recorded with identity id, proxy, status code, response headers, and a boolean indicating whether the page contained a challenge. On any non-200 or challenge, we also capture a screenshot, a DOM snapshot, and the HAR.

That data is the raw material of every improvement. We plot the challenge rate per identity and per proxy ISP on a three-day moving average. Nothing changes in the scraper based on a single 403; changes come only from moving-average trends. When the rate climbs across all identities, the vendor shipped a new fingerprint rule and we inspect the next failure's DOM snapshot for the new leak. When it climbs on one ISP, we drop that proxy tier.

Maintaining this habit turns the anti-bot fight from a guessing game into an iterative engineering loop. The classifier is a moving target; the only way to hit it is to observe it continuously and adapt in small increments.

Conclusion

The anti-bot arms race does not end. Every tactic in this article is a snapshot of a moving target, and the vendors will keep shipping new classifiers, new browser checks, and new behavioral models. That is fine. Our goal was never to win the war; it was to keep a scraper alive long enough for the data to be worth more than the cost of collecting it.

The architecture that emerged from the rebuild is boring by design. Patch a handful of known leaks. Run headful Chromium under Xvfb. Keep each identity's geo-consistent story intact. Jitter the behavior. Respect the CAPTCHA circuit breaker. Warm the proxies. Log everything. None of these steps is exotic, and that is exactly the point — a scraper that survives is one that gives the classifier nothing unusual to work with.

If I had to compress the whole learning into one line, it would be this: do not build a scraper that hides. Build a session that belongs. The moment your browser context feels like a neighbor's slow, messy, half-ignored browsing session, the walls come down on their own.

DoxDock is the living artifact of that lesson, and it is still running in production today. View the project on GitHub.