Start simple, escalate when blocked
The decision framework for the whole series: the escalation ladder, the two questions that pick a browser surface, and a safety rule for unattended jobs.
You need to get data off a website, or have an agent do something on one for you. There are many tools for that, and it's rarely obvious which one your situation calls for.
This series is the map. There's an order to these tools, cheapest first. What follows is the setup I actually run, generalized so you can use it without copying my machine.
Guideline: reach for the lightest tool that gets the data, and escalate only when you actually hit a wall.
The ladder
Cheapest and simplest at the bottom. You climb only when the rung you're on fails.
- Plain fetch and parse. Download the raw HTML, pull out what you want. No browser at all. This works for static pages, server-rendered pages, and (the one people miss) any site with a hidden JSON API you can call directly. Before you do anything fancier, check the network tab: a lot of "JavaScript-heavy" sites are just calling a clean JSON endpoint you can hit yourself.
- A clean-text reader. A hosted service loads the page, runs its JavaScript, and hands back tidy markdown or structured data. This is the right tool when you want to feed a page to an AI ("read this and summarize it") and you don't need to click anything. It costs a little per page, so it's for research and one-offs, not high volume.
- A real browser you drive. Now you're rendering JavaScript and interacting: clicking, typing, waiting, scrolling. This is the workhorse tier, and it splits two ways depending on whether you let an agent explore live or run a script you've already built. That's Part 3.
- A real browser you're already logged into. Instead of a fresh, empty browser, you point your automation at a Chrome that already has your sessions. Now it can reach anything you can reach, and it looks like what it is: a genuine, warmed-up human session. That's Part 2.
- Acting inside your everyday browser, while you watch. An agent operates in the exact window you're using, with all your real logins, human in the loop. Great for "just do this for me, right now." That's Part 4.
- Stealth tooling. Some sites are built to block automation outright. Purpose-made stealth browsers get through where a normal one can't. This is heavy, high-maintenance, and changes every few months, so it's a last resort, not a starting point. That's Part 6.
Two more rungs exist that I don't personally use but you should know: managed browser clouds, for running hundreds of sessions at once without owning servers, and multi-identity browsers, for keeping many separate accounts cleanly apart. Both solve scale and isolation problems that a solo builder reusing one real session doesn't have. I'll say more about why I skip them in Part 6.
Don't memorize the ladder. Just start at the bottom. Every time I've been stuck, the fix was usually one rung down, not one rung up.
The two questions that pick a browser surface
Once you're in browser territory (rungs 3 through 6), the surfaces differ along two axes. Get these two answers and the choice mostly makes itself.
One: does the task need a login that outlives a single run? If yes, you want a persistent, logged-in browser you can reuse (Part 2). If no, a disposable browser that starts clean every time is simpler and safer (Part 5). Logins are the whole reason the heavier surfaces exist. When there's no login, most of the machinery falls away.
Two: are you still figuring the site out, or do you already know the steps? When you're exploring, you want an agent that can see the page and react, so it isn't guessing blindly at where the buttons are. When the steps are known and you'll run them again, you want a script: one command, no exploration cost. Explore once with the agent, then bake the working flow into a script. Part 3 is about that handoff.
That's it. Persistent or disposable, exploring or repeating. Almost every "which tool should I use" question resolves into those two.
A safety rule for unattended jobs
A logged-in browser you reuse is convenient, but it runs in a visible window, and activating a tab in it pulls focus to that window. That's fine when you're sitting there. It's a problem when a job runs on a schedule or in the background while you're working: the window jumps to the front and takes over the screen mid-task. I've had a scheduled job do that, twice.
The rule: if a job could run unattended, or while you're working, it must not raise or focus a visible window. Use a headless surface for anything scheduled or daytime, a browser with no window to steal, and keep the reusable logged-in browser for when you're present and driving. "Reuse my session" and "never grab my focus" pull against each other, and headless is how you resolve it when you can't be sure you're watching.
The decision table
Find your row, go to the part.
| I want to… | Reach for | Covered in |
|---|---|---|
| Just get a page's text or data, no clicking | Plain fetch, or a clean-text reader | This part |
| Render a JavaScript page and click through it | A real browser you drive | Part 3 |
| Reuse a session I'm logged into (my dashboards, X, LinkedIn) | A persistent logged-in browser | Part 2 |
| Keep a hostile or fragile job isolated from everything else | A fresh, dedicated browser per run | Part 2 |
| Figure out an unfamiliar site | An agent driving the browser live | Part 3 |
| Run the same flow for the hundredth time | A script I built and tested | Part 3 |
| Do something in my own browser, right now, while I watch | An extension agent in my daily browser | Part 4 |
| Render or screenshot a public page in a test | A throwaway browser | Part 5 |
| Scrape a site that blocks normal automation | Stealth tooling, after ruling out a network-level block | Part 6 |
| Run this unattended or during my workday | A headless surface, never a visible window | This part (the safety rule) |
Three takeaways
- Start at the bottom of the ladder. A plain fetch handles more than you'd expect, and a hidden JSON API handles even more. Climb only when the current rung actually fails, not when you assume it will.
- Two questions pick your browser surface. Does the task need a persistent login, and are you exploring or repeating? Answer those and the tool is nearly chosen for you.
- Unattended means headless. A reused logged-in browser runs in a visible window that grabs focus when it activates a tab. If a job could run unattended, give it no window to raise.
Next up, Part 2: driving a real browser you're already logged into, and when to share one session versus spin up a fresh one.
About this series
This is the setup I actually run, generalized so you can use it without copying my machine. The layering is mine, worked out from running these surfaces daily and learning where each one breaks. A lot of the hard-won detail in the later parts comes from the wider community, especially the r/webscraping and r/antidetectbrowser regulars, whose recurring lesson (that a block is usually the network layer, not the browser) saved me more than once. Tool names in that space change every few months, so treat them as current, not permanent.
I used to have a standalone guide comparing Playwright's two modes for driving a browser. It was right, but narrow. Part 3 absorbs it and sets it next to the other surfaces, so that's the place to go now.
There's a companion guide on how the skill behind this series is built: the deterministic folder pattern, where a skill keeps the judgment and hands every mechanical step to a script. This series is the what and the when; that guide is the how.
A sanitized version of the skill I use to pick and drive these surfaces is free to download from the series landing page. Free to read, free to use. If you find something wrong, or advice that has aged, tell me.
Sources
- The explore-then-script handoff, and the token-cost case for scripts, traces back to a conversation with a developer I trust.
- The "check the network layer before blaming the browser" instinct in the ladder comes from the r/webscraping community; the detail is in Part 6.