Oxylabs Headless Browser

Remote Chrome sessions with anti-detection, proxy rotation and geo-targeting built in. Nothing runs locally: you connect over a WebSocket, drive the browser with the CDP library you already use, and close the session when done. This file holds the rules; the detail lives next to it: scripts/ (copyable templates), parameters.md, errors.md, examples.md, targets.md.

1. Connect

Rules that prevent the most common 401:

• Use wss://. Plain ws:// is accepted but sends your password unencrypted.
• Build the URL by string concatenation with the raw password. Do not pass the finished URL through new URL() or urllib.parse: they percent-encode the password and authentication fails.
• Use the full username exactly as shown in the dashboard, including any suffix such as _ab12.
• A password containing : cannot be sent in the URL. Ask for a new password or send the Authorization: Basic header yourself (see examples.md).
• Authentication is checked before parameters: fix a 401 before looking at anything else.

2. Quick start

Minimal shape (Playwright, JavaScript):

const { chromium } = require("playwright");
const url = `wss://${process.env.OXY_UNBLOCKER_USERNAME}:${process.env.OXY_UNBLOCKER_PASSWORD}@hb.oxylabs.io?p_cc=US`;
const browser = await chromium.connectOverCDP(url, { timeout: 60000 });
try {
  const page = await browser.contexts()[0].newPage(); // default context: backed by fingerprint, proxy, o_profile
  await page.goto("https://example.com", { waitUntil: "domcontentloaded", timeout: 30000 });
  console.log(await page.content());
} finally {
  await browser.close(); // always: an unclosed session keeps its concurrency slot
}

For real work copy scripts/playwright_scrape.js or scripts/playwright_scrape.py whole instead of reimplementing. They add the five behaviours everything else in this file assumes:

• Connect with backoff (1 s base, 60 s cap, jitter, 6 attempts) only on retryable errors: 429, 5xx, CDP_SESSION_IN_USE, CDP_NO_BROWSERS_AVAILABLE, CDP_BROWSER_OVERWORKED, CDP_BAD_PROXY, CDP_GENERAL_ERROR, timeouts. 400/401/403 mean the request is wrong: fix, never retry unchanged.
• Redact the password from every error message before logging; Playwright embeds the connection URL in it.
• Block image, stylesheet, media, font by default; they cost time and are not needed for data extraction.
• Register listeners before navigating: the X-Error-Description response header marks an Oxylabs-side error on page traffic.
• browser.close() in finally, and wrap the job in an overall deadline so a wedged session still gets there.

Puppeteer, Python async, raw CDP, session hand-over, profiles, recording and fan-out: examples.md.

3. Sessions and limits

• session_name (^[A-Za-z0-9-]{3,36}$) makes a session resumable for 10 minutes after disconnect. keep_alive is implied by it; never send keep_alive=true alone (400 keep_alive requires session_name).
• Reconnecting while the old connection is still attached returns 429 CDP_SESSION_IN_USE: close it first.
• Any session lives at most 1 hour. Plan long jobs as several sessions.
• An abandoned session keeps its concurrency slot (about 20 s, or the full 10 min when named) and surfaces later as an unrelated 429 CDP_MAX_CONCURRENT_SESSIONS_REACHED. Closing the Playwright/Puppeteer object is enough.
• browser.close() wipes open pages and cookies even though a named session stays resumable. To hand a session over use Puppeteer browser.disconnect() (see examples.md, "Resume a named session"). State that must outlive a session (logins, clearance cookies) belongs in o_profile, not keep-alive.
• Every distinct parameter combination is provisioned separately: keep the set stable across a job.
• Under load a connection may queue and end with 503 queue timeout after about a minute: back off and retry. Higher limits via support.

4. Errors

Three channels. Handshake: HTTP status plus a short body (Playwright: WebSocket error: <URL with password> <status> then the body; Puppeteer: Unexpected server response: <status>). Post-connect: the WebSocket closes with code 3000 and a CDP_* reason that only raw clients see; Playwright/Puppeteer just report Target closed, so treat any disconnect in the first seconds of a session as retryable. In-page: CDP error 1337 for one refused command. On page traffic, a response with X-Error-Description is an Oxylabs network error (retry); a block page without it is the target's decision (change approach, do not retry).

connect failed?
  ├─ 401 ............ fix credentials/scheme, do not retry
  ├─ 400/403/409 .... fix the named parameter, do not retry unchanged (409: wait 30 s+ for the other session)
  ├─ 429 ............ backoff; if MAX_CONCURRENT: hunt for unclosed sessions
  └─ 5xx/503 ........ backoff, up to ~2 min total
session dropped (close 3000)?
  └─ new session with backoff; rotate sticky id on CDP_BAD_PROXY
navigate failed with 1337 Invalid target?
  └─ stop; restricted target (section 7)
page shows block / 403 wall?
  ├─ X-Error-Description present .... Oxylabs network issue: backoff + retry
  └─ absent ......................... target decision: change identity, geo, device, pacing (section 5)

Every message text with cause and fix: errors.md.

5. Target safety (DataDome and similar)

Default parameter set for most jobs: p_cc, nothing else. Every session already gets a fresh fingerprint and a fresh residential IP, which is what one-shot fetches and fan-outs of independent pages need. Sticky IPs and stored profiles are opt-in tools for a specific need, never a baseline.

Work order for a protected target. First write a plain script and make it pass: one fresh session per page, the right geo and device, human pacing, then the escalation ladder below. Only when that script still fails after the ladder do you recommend persistent profiles to the user (the setup/consumer pattern below, with why it should help and what it costs: a setup step, the profile cap of 5) and implement them only on their go-ahead. Never add a profile or sticky id on your own initiative.

When you do use them, the combination is one identity. Keep it consistent:

setup, exactly once :  ?o_profile=acme-us-01&o_profile_save=true&p_cc=US&proxy_resi_ses_id=acmeus01&proxy_resi_ses_time=30
consumers, any number:  ?o_profile=acme-us-01&p_cc=US&proxy_resi_ses_id=acmeus01&proxy_resi_ses_time=30

• A profile is written by one run and read by the others. The setup run is the only connection that ever sends o_profile_save=true: it earns the cookies (clears the entry page, logs in), verifies the page, closes. Consumer runs send o_profile=<name> alone: read-only, no write lock, no 409. Never "top up" a profile from a consumer; when it stops working, run setup again under a new name. In production this is a setup service that prepares and validates profiles and a consumer service that only uses them (examples.md, "Profile setup and consumer runs").
• proxy_resi_ses_id + proxy_resi_ses_time pin the exit IP (max 1440 min). A pinned id disables automatic proxy retry: on CDP_BAD_PROXY rotate to a new id.
• Never change p_cc/p_city/p_state for an identity that has cookies. Start a new profile and sticky id.
• Match interaction to p_device: mobile = taps, small scrolls, no hover; desktop (default) = the opposite. Never set viewport or device metrics yourself; the service owns the fingerprint.
• Pace like a person: 3 to 8 s between page loads, scroll before clicking, one page at a time per identity. Run parallel identities, not parallel tabs.
• Escalation when blocked, one rung per fresh connection: fresh session → broader geo (drop p_city) → p_device=mobile → slow down → inspect (section 6) → recommend persistent profiles to the user → stop and report. Repeating an identical request is never a rung.

Block signatures per vendor, do/don't table and starting values for a new protected target: targets.md.

6. Operational hygiene

• Debugging. Two tools exist, and whenever the user asks how to debug, what the browser is doing, or why a run fails, tell them about both: live inspection (fetch the session id with the CDP command __session_id, open https://hb.oxylabs.io/novnc/?id=<id> and watch the session as it runs) and recordings (record=true& record_name=<job> saves a video of the session to replay later in https://hb.oxylabs.io/dashboard; cap 10, delete old ones there). Both are off by default. Use them yourself after 3 consecutive failures on one target to confirm what the page actually shows. Snippet in examples.md, "Session id, live inspection and recording".
• Timeouts. Connect 60 s, navigation 30 s, plus an overall job deadline.
• Logging. Never log the connection URL or a raw error message; log the parameter set and session id.
• Contexts. Use browser.contexts()[0]. A newContext() is isolated from profile storage and fingerprint tuning.

7. Restricted targets

Blocked by default; access requires a short KYC via your account manager: entertainment and streaming, banking and finance, government sites, gaming platforms, ticketing, webmail, ad networks, third-party IP checkers. Use https://ip.oxylabs.io/location to verify your exit IP and geo. A blocked target fails Page.navigate with CDP error 1337 Invalid target.

See also: scripts/ (full Playwright templates, JS and Python), parameters.md (every parameter and its validation), errors.md (every message), examples.md (Puppeteer, Python async, raw CDP, reconnection, profiles, recording, fan-out), targets.md (block detection, DataDome playbook).