Building an SEO crawler that borrows my own Chrome

The first time bot protection cost me an audit, I did not notice for two days.

The crawl finished. It exported cleanly. The numbers looked plausible, and I had already sent the deck before someone asked why a section of the site with four hundred pages in it showed up with eleven. Cloudflare had been serving an interstitial to my crawler for most of the run, and the crawler had recorded each of those as a perfectly ordinary page.

That is the part nobody warns you about. A blocked crawl does not fail. It succeeds, with a hole in it, and the hole is shaped exactly like the pages you most wanted to look at.

Everything I tried had a half-life

The obvious first move is to make the headless browser look less headless. There is a whole genre of plugins for this, and they work — for a while. A headless Chrome gives itself away in a dozen places, you patch the ones a given check looks at, and the next release checks somewhere else. The fingerprint you spent a week on is stale by the time you have finished building around it.

The second move is to stop pretending and export your cookies. Copy a real session out of the browser, feed it to the scraper, and you get a crawler that is genuinely signed in. Until the session rotates. Or the IP changes. Or the TLS fingerprint stops matching the one the session was minted against, and the whole thing quietly turns back into a bot.

The third is to relaunch Chrome with --remote-debugging-port and drive the real thing. This works properly, and I used it for a while, and it is a terrible idea. That port has no concept of ownership: whatever connects to it gets your entire profile. Every tab, including your mail, including the client dashboard you left open in the background. I was solving a crawling problem by opening a hole in the browser I do my banking in.

All three share a shape. They are attempts to build a browser that resembles mine closely enough to pass.

So use mine

The inversion is short enough to say in a sentence: instead of imitating my browser, the crawler drives it.

There is nothing left to fake, because the profile doing the crawling is the one already signed in — same cookies, same IP, same TLS fingerprint, same everything, because it is the same browser. That is Krawlify. It is self-hosted, MIT licensed, and it comes in four pieces you can adopt one at a time.

The catch is that you cannot simply connect to a running Chrome. That is the whole reason --remote-debugging-port exists, and it is the thing I was trying to avoid.

Why there is an extension, and why there is a relay

chrome.debugger is the only API that speaks the DevTools Protocol to a Chrome that was started normally. It belongs to extensions. So an extension it is.

Which immediately creates a second problem: a Manifest V3 extension cannot listen on a port. It has no way to accept a connection. It can only dial out.

Meanwhile your crawler wants to connect in — that is what puppeteer.connect does. Two halves, both wanting to be the client, neither able to be the server.

The relay is what sits between them. Your crawler connects in, your browser dials out, and the relay matches the two by the Bearer token in the request. It is one Node process with one dependency, bound to 127.0.0.1 unless you tell it otherwise, and it holds nothing — tab ownership lives in memory and is thrown away when the connection ends.

The token is the interesting part. The relay has no token of its own and keeps no list of valid ones. Your browser generates its own, you paste it into the crawl, and the match is what pairs them. Which is why one relay can serve many browsers without ever holding any of them, and why a wrong token and no token look identical from outside.

const browser = await puppeteer.connect({  browserWSEndpoint: 'ws://127.0.0.1:9333/devtools/browser/<guid>',  headers: { authorization: 'Bearer <token>' },});const browser = await chromium.connectOverCDP('http://127.0.0.1:9333', {  headers: { authorization: 'Bearer <token>' },});

No fork, no patched client. The relay presents an ordinary DevTools endpoint, so if your code talks to a headless Chrome today it talks to your real profile tomorrow. You change the URL and add a header.

What the crawler can see, and what it cannot

This is the part I would want answered before installing anything that attaches a debugger to my browser.

The extension attaches to tabs the automation created, and only those. They appear grouped under a purple Krawlify label so you can watch it work, and they close themselves five seconds after the last client disconnects.

Your own tabs are not filtered out of a list somewhere. They are not addressable in the protocol at all — there is no request the crawler can make that reaches them. Nor history, bookmarks, downloads, saved passwords, chrome:// pages, other extensions or the Web Store, all of which are outside what any extension may attach to.

It stores three values: the relay URL, the token the browser generated, and whether you last pressed Connect or Disconnect. No content scripts, no telemetry, no crash reporting. Cookies are an optional permission you grant on the options page and can revoke whenever.

It is also not on the Chrome Web Store, and will not be. A Manifest V3 reviewer cannot determine what the extension does without access to the relay you point it at, and pointing your browser at infrastructure you operate is the entire feature. It ships as a signed CRX installed by enterprise policy, or a ZIP you load unpacked.

The crawler on top

All of that is plumbing for the thing I actually wanted, which is an audit.

The app is a self-hosted crawler that lands a run in thirteen tabs — internal and external links, response codes, titles, meta descriptions, headings, directives, images, broken assets, redirects, sitemaps and dates. Titles and meta carry both character and pixel widths, because the character count is the one everybody quotes and the pixel width is the one that decides whether it gets truncated.

The Dates tab is the one I built for myself. It pulls datePublished and dateModified out of schema.org markup and lines both up against <lastmod> from the sitemap, per URL. When those three disagree, something in the publishing pipeline is lying, and it is usually lying to Google too.

Results stream in over Server-Sent Events while the crawl runs, so the dashboard fills up as pages land instead of sitting on a spinner. Every setting is stored on the crawl itself rather than read from the environment, which means Redo crawl genuinely reruns it — same depth, same limit, same user agent, six months later.

And after the audit I opened this post with, one rule: if the remote browser drops mid-crawl, the run fails and says so. It does not finish with a hole in it.

Three ways in

The UI stopped being the only client fairly quickly.

There is a REST API on keys you mint in Settings, with in-app docs and a request runner that prints the equivalent curl. And there is an MCP server — streamable HTTP, stateless, same keys, seven tools — which is enough for an assistant to start a crawl, watch it, and read the tabs it needs.

claude mcp add --transport http krawlify \  http://localhost:3000/api/mcp \  --header "Authorization: Bearer YOUR_KEY"

There is a fourth piece for the case where you already know which URLs you care about: a Claude Code skill with a CLI underneath, published on npm as krawlify-agent. Hand it a list, get back one Excel workbook. No discovery, no link following. It can run against the relay, or launch your installed Chrome on your own profile with --launch, which needs no extension and no relay at all.

What it does not do

Driving a real browser through an extension buys a lot and costs a few things.

Chrome allows one debugger client per tab, and DevTools wins — open it on a crawler's tab and Krawlify detaches from that tab. Target.createBrowserContext is unsupported, so browser.newContext() and incognito are out; that follows from using one real profile rather than being an oversight. Browser.setDownloadBehavior is accepted and then ignored, so Playwright's download API will not work through the relay.

And none of this is a guarantee against bot protection. It waits up to sixty seconds for an interstitial to clear and then continues through the same cookie jar, which works often enough to have changed how I audit sites. It still only applies to sites you are allowed to crawl.

Running it

# Postgres, Redis, migrations, the app and the crawl workerdocker compose up --build# → http://localhost:3000 — register, then start a crawl

Start there. The app on its own crawls most sites fine, and the browser path is worth adding the first time something blocks you — a second container for the relay, the extension in Chrome, and a token pasted into the crawl form.

Everything is on GitHub across krawlify-app, krawlify-relay — which holds both the extension and the relay — and krawlify-skill. There is a live instance at app.krawlify.com if you would rather click around before installing anything; it is a shared public sandbox on one admin account, so put nothing confidential in it.

No accounts, no tiers, nothing sent anywhere. The crawl runs on your machine, through your browser, and the results stay there.