Krawlify — SEO Crawler & Audit Platform

Overview

Every SEO crawler I had used made the same bet: that the site would let a headless browser in. Increasingly they do not, and the failure is quiet — you get a crawl that finishes, exports cleanly, and is missing the third of the site that sat behind a bot check.

Krawlify is the crawler I wanted instead. It runs on my own machine, it can borrow the Chrome I am already signed in to when a site fights back, and it hands the same results to me, to my code, and to an assistant.

What it produces

A crawl lands in thirteen tabs, close to what Screaming Frog gives you and laid out the same way, because that is the shape SEO people already read. Titles and meta descriptions carry both character and pixel widths. Headings are broken out H1 to H6. Images are listed with their alt text, missing or too long. Broken assets and redirect chains are separated from the ordinary internal report rather than mixed into it.

The Dates tab is the one I use most and had never found elsewhere: it pulls datePublished and dateModified out of schema.org markup and lines them up against <lastmod> in the sitemap, per URL. Disagreements between those three are usually the first sign that a site's publishing pipeline is lying to search engines.

Anything on screen exports as CSV or Excel with your filters applied, and a master workbook writes every tab to its own sheet. A crawl.completed webhook fires on finish.

Built to be run by something other than a person

The UI came first, but it stopped being the only client fairly quickly.

The REST API is key-authenticated, with keys minted in Settings and shown once. There are in-app docs at /docs/api and a request runner at /playground that prints the equivalent curl for whatever you just ran.

The MCP server is streamable HTTP at /api/mcp, stateless, on the same keys. Seven tools — list, get, results, tabs, start, stop, delete — which is enough for an assistant to run an audit end to end and read the parts it needs.

claude mcp add --transport http krawlify \  http://localhost:3000/api/mcp \  --header "Authorization: Bearer YOUR_KEY"

Multi-user without the usual holes

Members and admins, per-user crawls and per-user API keys, and guards that refuse to let you disable the last active admin. Small thing, but it is the failure everyone hits once and nobody tests for.

Running it

Postgres, Redis, the migrations, the Next.js app and the crawl worker all come up together. Images are published for linux/amd64 and linux/arm64, so it runs unmodified on Apple silicon and on an ARM server.

# Pull the published image instead of buildingdocker pull kamenarov/krawlify-app:0.1.0# Postgres, Redis, migrations, web and workerIMAGE=kamenarov/krawlify-app:0.1.0 docker compose up -d# → http://localhost:3000 — register, then start a crawl

Defaults are depth 10, a 10,000 URL cap, concurrency 8, a 15-second request timeout and robots.txt respected. Politeness delay, custom user agent and an optional proxy are all on the crawl form.

Where it lives

Source is at kamenarov/krawlify-app, images at kamenarov/krawlify-app on Docker Hub. MIT licensed, no accounts, no tiers, published from tagged releases by CI rather than from my laptop.

There is a live instance at app.krawlify.com with published sign-in details. It is a shared public sandbox on a single admin account, so treat everything in it as public and expect it to be reset.