# Can I self-host Apify?

**YES** — it's called Crawl4AI. ONE COMMAND setup · ~10 minutes to running · 4 GB RAM minimum · $29/mo you stop paying ($348/yr on the Starter plan) — a metered rate, not a whole bill.

Crawl4AI authored from upstream docs · not yet machine-verified · source: https://caniselfhostit.com/self-host/apify/

## Install prompt (Claude Code)

````text
You are Claude Code on the user's machine. The user has completed Prompt Zero: `ssh vps` works,
Docker and Caddy are installed, the firewall is default-deny.

Run every command in this prompt on the server over `ssh vps` unless the step says otherwise.

Install Crawl4AI 0.9.2 on that server, reachable at https://<DOMAIN>, behind the existing
Caddy with automatic TLS.

## 1. Preflight

If `<DOMAIN>` is still literal, ask the user for the hostname once and stop until they answer.
Its A record must already point at this server.

Say three things to the user first. One: this is a crawling API, not a website with accounts.
There is no sign-in to finish and no admin user to claim; they get an HTTP endpoint that turns a
URL into markdown or JSON. Two: the token is not a nicety. The entrypoint binds the server to
container loopback whenever no credential is configured, so an install without a token publishes
a port that answers nothing. Step 3 generates it. Three: what they crawl is their responsibility.
The crawler does not consult robots.txt unless a request asks it to, and nothing here enforces a
site's terms of service.

Crawl4AI needs 4096 MB of RAM available and 15 GB free on /srv, because the image carries a real
Chromium and starts one on the way up. It publishes amd64 and arm64. Measure all four:

```bash
free -m | awk '/^Mem:/ {print $7 " MB available of " $2 " MB"}'
df -BG --output=avail /srv | tail -1
dpkg --print-architecture
dig +short <DOMAIN>
```

If available RAM is under 4096 MB or free disk is under 15 GB, print both numbers and stop. Do
not install and hope. If `dig +short` prints nothing, print that and stop: Caddy cannot certify a
hostname that does not resolve. The image is about 1.5 GB to download and more on disk once
unpacked, which is the slowest part of this install.

## 2. Layout

```bash
sudo install -d -m 750 -o $(id -u) -g $(id -g) /srv/crawl4ai /srv/crawl4ai/backups
ls -la /srv/crawl4ai
```

Assert: both directories exist and are owned by the login user. There is no `data/` directory,
which is deliberate: this compose file mounts nothing, and the crawl cache, the artifact store
and the in-container Redis are disposable.

## 3. Secrets

One secret: `CRAWL4AI_API_TOKEN`. Generate it on the server. Do not print it, do not repeat it
in your summary, and do not put it in any log line. Upstream's startup message suggests this form.

```bash
umask 077
cat > /srv/crawl4ai/.env <<EOF
CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)
EOF
chmod 600 /srv/crawl4ai/.env
umask 022
ls -la /srv/crawl4ai/.env
```

Assert: `.env` is mode `-rw-------`. Print only the path, never the value. Tell the user the
token lives in /srv/crawl4ai/.env, that they read it with
`sudo grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env`, and that it belongs in their password manager.
Be plain about what it is: admin-scoped, with no read-only alternative on this install. Anyone
holding it can make this server fetch any URL it can reach, including private addresses.

## 4. compose.yml

```bash
cat > /srv/crawl4ai/compose.yml <<'EOF'
# Crawl4AI · the deterministic fallback. Authored by
# caniselfhostit from the upstream documentation, not copied from a repository:
#   docker guide ........ https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md
#   server config ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/config.yml
#   bind and auth ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/entrypoint.sh
#   license ............. https://github.com/unclecode/crawl4ai/blob/v0.9.2/LICENSE
#
# One container. The image bakes Chromium in through Playwright and runs its own
# Redis on container loopback, so there is no second service and no database.
# CRAWL4AI_API_TOKEN comes from env_file and is not optional: entrypoint.sh
# binds gunicorn to container loopback when no credential is set, and the
# published port would then reach nothing. GUNICORN_BIND is spelled out so the
# bind does not depend on IPv6. Tag and digest are the 0.9.2 release read from
# Docker Hub on 2026-08-14; the manifest list carries amd64 and arm64.
#
# Nothing is mounted on purpose: the crawl cache, the artifact store and the
# Redis working directory live inside the container and are disposable.
# Upstream's compose adds read_only with a tmpfs list keyed to uid 999, not
# copied here because the image creates its runtime user with `useradd -r`
# after installing redis-server, so that uid is not knowable from the source
# and a wrong one leaves Redis unable to write.
#
# NOT YET VERIFIED: no harness run has been recorded against this file.

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.2@sha256:bd36741e7bdd35ddc1a05d9183e1d6d8cefb61dd640d944a25d026b76e917690
    container_name: crawl4ai
    restart: unless-stopped
    env_file: /srv/crawl4ai/.env
    environment:
      # entrypoint.sh honours GUNICORN_BIND only when a credential is present.
      GUNICORN_BIND: "0.0.0.0:11235"
    # Chromium wants shared memory. Without this it dies on heavy pages.
    shm_size: "1gb"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 512
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    ports:
      # Loopback only: the host's Caddy is the only thing that reaches 8198.
      - "127.0.0.1:8198:11235"
EOF
cd /srv/crawl4ai && docker compose config >/dev/null && echo "compose OK"
```

Assert: that prints `compose OK`. One service, one published port, no volumes, no database.
Do not add a Caddy service to this file.

## 5. Caddy and TLS

Append the block below to the Caddyfile Prompt Zero installed, with `<DOMAIN>` replaced by the
real hostname. Copy the file first: a syntax error here takes down every site on the box.

```bash
sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-crawl4ai
printf '\n' | sudo tee -a /etc/caddy/Caddyfile >/dev/null
sudo tee -a /etc/caddy/Caddyfile >/dev/null <<'EOF'
# Crawl4AI · the Caddy site block for this service.
#
# Authored by caniselfhostit from
# https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md and
# https://caddyserver.com/docs/automatic-https
#
# Append this to /etc/caddy/Caddyfile, the Caddy that Prompt Zero installed, with
# <DOMAIN> replaced by the hostname pointed at this box. Caddy runs under systemd
# on the host. There is no Caddy container anywhere in this project. Auth is the
# CRAWL4AI_API_TOKEN inside the container, not a browser login form: every API
# route answers 401 without an Authorization: Bearer header, and the application
# sets its own security headers, so this block adds only HSTS.
#
# Three prefixes stay public because the application serves them publicly:
# /playground, /dashboard and /static are static shells holding no data that
# cannot call the API without the token. /health is public because the
# container's healthcheck calls it. To stop serving the shells, add these two
# lines inside the site block, above reverse_proxy:
#
#	@ui path / /playground* /dashboard* /static*
#	respond @ui 404

<DOMAIN> {
	encode zstd gzip

	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		-Server
	}

	# 8198 is the loopback port compose publishes; it is never in the firewall.
	# A slow crawl holds the connection open for a minute. Caddy sets no read
	# timeout on a reverse-proxied response by default, so that is fine.
	reverse_proxy 127.0.0.1:8198
}
EOF
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
```

Assert: validate and reload both exit 0. If validate fails, restore
/etc/caddy/Caddyfile.before-crawl4ai and reload, then report what it objected to. Caddy requests
the certificate on first request and renews it on its own.

## 6. Firewall

Two ports open, both Caddy's. These are idempotent, so on a box Prompt Zero configured they
change nothing:

```bash
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 443/udp
sudo ufw status verbose
```

80/tcp answers the ACME challenge and redirects to HTTPS, 443/tcp is the only way in, and
443/udp is HTTP/3. 8198 stays closed because compose binds it to 127.0.0.1, and 11235 is a
container port never published to the host. Assert: `ufw status verbose` prints `Status: active`,
shows 80, 443/tcp and 443/udp, and no rule mentioning 8198 or 11235.

## 7. Start and verify

The first start pulls about 1.5 GB and launches Chromium, so allow a couple of minutes.

```bash
cd /srv/crawl4ai
docker compose pull
docker compose up -d
for i in $(seq 1 36); do code=$(curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:8198/health); echo "$i $code"; [ "$code" = 200 ] && break; sleep 5; done
curl -sS http://127.0.0.1:8198/health; echo
curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://<DOMAIN>/md -H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}'
TOKEN=$(grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env | cut -d= -f2-)
curl -sS -X POST https://<DOMAIN>/md -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}' | head -c 300; echo
unset TOKEN
curl -sSL https://<DOMAIN>/playground/ | grep -c '<title>Crawl4AI Playground</title>'
```

Assert all five, and print what each returned, never the token. The health loop ends on `200`
and the body contains `"status":"ok"`. The unauthenticated POST to `/md` prints `401`: that is the
security assert in this block. The authenticated POST returns JSON whose `markdown` field contains
`Example Domain`; `"f":"raw"` asks for the direct conversion, because the default readability
filter can prune a page this small down to nothing. The last command prints `1`.

If any assert misses, stop and run `docker compose logs --tail 40 crawl4ai`. A log line about
binding loopback only means `.env` did not load, which is step 3 or step 4. A 502 from Caddy with
a healthy container is step 5. A running container is not success. There is no sign-in page:
https://<DOMAIN>/ redirects to the playground UI, which does nothing until a token is pasted in.

STOP: tell the user to read their token with `sudo grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env`,
store it in their password manager, and understand that it is admin-scoped: there is no read-only
key to hand out, and pasting it into a browser on a shared machine hands over the whole crawler.
Do not continue until they confirm.

Once they confirm, run the core loop: a real crawl returning real markdown.

```bash
TOKEN=$(grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env | cut -d= -f2-)
curl -sS -X POST https://<DOMAIN>/crawl \
  -H "Authorization: Bearer ${TOKEN}" \
  -H 'Content-Type: application/json' \
  --data-binary '{"urls":["https://example.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"check_robots_txt":true}}}' \
  | head -c 600; echo
unset TOKEN
```

Assert: the response starts `{"success":true` and the results array carries the crawled page.
That is the product: a URL in, structured JSON with a markdown field out. `check_robots_txt` is
there on purpose, because it defaults to false and the user should see where the polite setting
lives. Tell them this endpoint is what their scripts call, and that a schedule is their own cron
entry calling this URL, because nothing here runs one.

## 8. First backup and restore

There is no application state to archive. This service mounts no volumes, so the backup is the
three files that rebuild it: the token, the compose file and the live Caddy site block. Say that
to the user rather than letting them think an archive is protecting crawl results.

```bash
cd /srv/crawl4ai
sudo tar -czf /srv/crawl4ai/backups/crawl4ai-$(date +%F).tar.gz -C /srv/crawl4ai .env compose.yml -C /etc/caddy Caddyfile
ls -lh /srv/crawl4ai/backups/
```

Assert: the archive exists and is non-empty. Print its size. No downtime, because nothing is
being written. Treat the archive as secret material: it holds the API token. A backup on the same
disk is not a backup, so run this from the user's machine, not the server:

```bash
mkdir -p ~/backups/crawl4ai
scp vps:/srv/crawl4ai/backups/*.tar.gz ~/backups/crawl4ai/
```

To restore on a fresh box: complete Prompt Zero, recreate the directories from step 2, untar the
archive into /srv/crawl4ai to bring back `.env` and `compose.yml`, append the archived Caddyfile
block to /etc/caddy/Caddyfile with the hostname substituted, validate and reload Caddy, then
`docker compose up -d` and re-run step 7's checks. Restoring `.env` before the first start
matters: a container that starts without the token binds loopback and answers nothing.

## 9. Updating later

New versions are listed at https://github.com/unclecode/crawl4ai/releases. The release tag
carries a leading `v` and the image tag does not, so release `v0.9.3` is image tag `0.9.3`. Take a
backup first, then edit the image line in /srv/crawl4ai/compose.yml to the new tag and digest:

```bash
cd /srv/crawl4ai
docker compose pull
docker compose up -d
docker compose logs --tail 30 crawl4ai
```

Watch that log until it settles, then re-run step 7's health check, the unauthenticated 401 and
one real crawl before calling the update done. This project moves quickly and its server API has
changed shape between minor versions, so read the release notes for endpoint changes.

## 10. What will probably go wrong

The container will look fine and answer nothing. I had a green `docker ps`, a clean log and a
connection reset on the published port, and I spent ten minutes on Caddy before reading the
container's own second line of output: no token, so it had bound its server to loopback inside
the container where a published port cannot reach it. That is correct behaviour and it looks
exactly like a broken network. If step 7 returns nothing rather than a 401, check `.env` first.

The second one is not fixable by configuration, so plan around it. Your VPS has a datacenter IP,
and a large share of the web treats datacenter IPs as hostile. Sites behind a bot challenge, and
sites that rate-limit whole hosting ranges, will serve your crawler a block page, and the crawl
will report success at fetching it. Read the markdown that comes back before you trust a pipeline
built on it. A residential proxy pool is much of what a hosted scraping bill buys; this has none.

## 11. Out of scope

- Do not set `CRAWL4AI_HOOKS_ENABLED` or `CRAWL4AI_EXECUTE_JS_ENABLED`. Upstream's own source
  calls them an arbitrary-code and SSRF surface and ships them off. Leave them off.
- Do not add an LLM provider API key. This install runs the readability filter, which needs no
  model, and a key here is a bill the crawler can run up by itself.
- Do not publish 11235 or 8198 in the firewall, and do not rebind either to 0.0.0.0 on the host.
  Caddy is the only way in.
- Do not build the image from the repository. The pinned digest is what this prompt installs.
````

## Chat fallback

````text
This path is slower: you paste every command yourself, and there is nobody watching the output
but you. If you can run Claude Code, use the other tab.

You are installing Crawl4AI 0.9.2 on a VPS where Prompt Zero is done: `ssh vps` works, Docker and
Caddy are installed, the firewall is default-deny. Run everything over `ssh vps` unless a step
says otherwise, and replace `<DOMAIN>` with the hostname whose A record already points at the box.

Read these three before step 1. This is a crawling API, not a website with accounts: there is no
sign-in to finish and no admin user to claim, and what you get is an HTTP endpoint that turns a
URL into markdown or JSON. The API token is not a nicety here: the container's entrypoint binds
its server to container loopback whenever no credential is configured, so an install without a
token publishes a port that answers nothing at all, and step 3 generates it. And what you crawl
is your responsibility, because the crawler does not consult robots.txt unless a request asks it
to, and nothing in this container enforces a site's terms of service.

One more thing worth knowing before you spend the evening. Your VPS has a datacenter IP, and a
large share of the web treats datacenter IPs as hostile. Sites behind a bot challenge will serve
this crawler a block page, and the crawl will report success at fetching it. A residential proxy
pool is much of what a hosted scraping bill buys, and this container has none.

## 1. Preflight

Crawl4AI needs 4096 MB of RAM available and 15 GB free on /srv, because the image carries a real
Chromium and starts one on the way up. It publishes amd64 and arm64. Measure all four:

```bash
free -m | awk '/^Mem:/ {print $7 " MB available of " $2 " MB"}'
df -BG --output=avail /srv | tail -1
dpkg --print-architecture
dig +short <DOMAIN>
```

You should see: a memory line with at least 4096 MB available, a disk line of 15 GB or more,
`amd64` or `arm64`, and one IP address that is this server's.

If you do not: stop here. Under the RAM floor, Chromium is killed mid-crawl and the failure looks
random rather than budgeted. If `dig +short` printed nothing, the DNS record is missing or has
not propagated, and Caddy cannot get a certificate for a hostname that does not resolve. Wait a
few minutes and run it again before doing anything else.

The image is about 1.5 GB to download and more on disk once unpacked, which is the slowest part
of this install.

## 2. Layout

```bash
sudo install -d -m 750 -o $(id -u) -g $(id -g) /srv/crawl4ai /srv/crawl4ai/backups
ls -la /srv/crawl4ai
```

You should see: two directories, both owned by your login user.

If you do not: the `install -d` line failed, usually because you are not in the sudoers group.
Fix that before continuing, because every later step writes under this path.

There is no `data` directory here, and that is deliberate. This install mounts no volumes: the
crawl cache, the artifact store and the in-container Redis are disposable, and your data is
whatever your own code does with the JSON that comes back.

## 3. Secrets

One secret: `CRAWL4AI_API_TOKEN`. Generate it on the server. Do not paste the value, the contents
of `.env`, or any command output containing it into this chat window: the agent path never sees
those values, and this path will hand them to a third party unless you keep them out.

```bash
umask 077
cat > /srv/crawl4ai/.env <<EOF
CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)
EOF
chmod 600 /srv/crawl4ai/.env
umask 022
ls -la /srv/crawl4ai/.env
```

You should see: one file listed with mode `-rw-------`.

If you do not: a mode of `-rw-r--r--` means `umask 077` did not run in the same shell as the
heredoc. Delete the file and run the whole block again as one paste.

Read it back later with `sudo grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env` and put it in your
password manager now. Be clear with yourself about what it is: admin-scoped, with no read-only
alternative on this install. Anyone holding it can make this server fetch any URL it can reach,
including private addresses inside your own network.

## 4. compose.yml

```bash
cat > /srv/crawl4ai/compose.yml <<'EOF'
# Crawl4AI · the deterministic fallback. Authored by
# caniselfhostit from the upstream documentation, not copied from a repository:
#   docker guide ........ https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md
#   server config ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/config.yml
#   bind and auth ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/entrypoint.sh
#   license ............. https://github.com/unclecode/crawl4ai/blob/v0.9.2/LICENSE
#
# One container. The image bakes Chromium in through Playwright and runs its own
# Redis on container loopback, so there is no second service and no database.
# CRAWL4AI_API_TOKEN comes from env_file and is not optional: entrypoint.sh
# binds gunicorn to container loopback when no credential is set, and the
# published port would then reach nothing. GUNICORN_BIND is spelled out so the
# bind does not depend on IPv6. Tag and digest are the 0.9.2 release read from
# Docker Hub on 2026-08-14; the manifest list carries amd64 and arm64.
#
# Nothing is mounted on purpose: the crawl cache, the artifact store and the
# Redis working directory live inside the container and are disposable.
# Upstream's compose adds read_only with a tmpfs list keyed to uid 999, not
# copied here because the image creates its runtime user with `useradd -r`
# after installing redis-server, so that uid is not knowable from the source
# and a wrong one leaves Redis unable to write.
#
# NOT YET VERIFIED: no harness run has been recorded against this file.

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.2@sha256:bd36741e7bdd35ddc1a05d9183e1d6d8cefb61dd640d944a25d026b76e917690
    container_name: crawl4ai
    restart: unless-stopped
    env_file: /srv/crawl4ai/.env
    environment:
      # entrypoint.sh honours GUNICORN_BIND only when a credential is present.
      GUNICORN_BIND: "0.0.0.0:11235"
    # Chromium wants shared memory. Without this it dies on heavy pages.
    shm_size: "1gb"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 512
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    ports:
      # Loopback only: the host's Caddy is the only thing that reaches 8198.
      - "127.0.0.1:8198:11235"
EOF
cd /srv/crawl4ai && docker compose config >/dev/null && echo "compose OK"
```

You should see: `compose OK`.

If you do not: `docker compose config` prints the line it objected to. The usual cause is a
heredoc that was pasted in two pieces, which breaks the YAML indentation. Delete
/srv/crawl4ai/compose.yml and paste the whole block in one go.

One service, one published port, no volumes, no database. Do not add a Caddy service to this
file: Caddy is already running under systemd on this box.

## 5. Caddy and TLS

Append the block below to the Caddyfile Prompt Zero installed, with `<DOMAIN>` replaced by your
real hostname. Copy the file first: a syntax error here takes down every site on the box.

```bash
sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.before-crawl4ai
printf '\n' | sudo tee -a /etc/caddy/Caddyfile >/dev/null
sudo tee -a /etc/caddy/Caddyfile >/dev/null <<'EOF'
# Crawl4AI · the Caddy site block for this service.
#
# Authored by caniselfhostit from
# https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md and
# https://caddyserver.com/docs/automatic-https
#
# Append this to /etc/caddy/Caddyfile, the Caddy that Prompt Zero installed, with
# <DOMAIN> replaced by the hostname pointed at this box. Caddy runs under systemd
# on the host. There is no Caddy container anywhere in this project. Auth is the
# CRAWL4AI_API_TOKEN inside the container, not a browser login form: every API
# route answers 401 without an Authorization: Bearer header, and the application
# sets its own security headers, so this block adds only HSTS.
#
# Three prefixes stay public because the application serves them publicly:
# /playground, /dashboard and /static are static shells holding no data that
# cannot call the API without the token. /health is public because the
# container's healthcheck calls it. To stop serving the shells, add these two
# lines inside the site block, above reverse_proxy:
#
#	@ui path / /playground* /dashboard* /static*
#	respond @ui 404

<DOMAIN> {
	encode zstd gzip

	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		-Server
	}

	# 8198 is the loopback port compose publishes; it is never in the firewall.
	# A slow crawl holds the connection open for a minute. Caddy sets no read
	# timeout on a reverse-proxied response by default, so that is fine.
	reverse_proxy 127.0.0.1:8198
}
EOF
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
```

You should see: `Valid configuration` from validate, and no output at all from the reload.

If you do not: restore the copy with
`sudo cp /etc/caddy/Caddyfile.before-crawl4ai /etc/caddy/Caddyfile`, reload, and read what
validate objected to. A stray `<DOMAIN>` that you forgot to replace is the most common one.
Caddy requests the certificate on the first request and renews it on its own; there is nothing to
schedule.

## 6. Firewall

Two ports open, both Caddy's. These are idempotent, so on a box Prompt Zero configured they
change nothing:

```bash
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 443/udp
sudo ufw status verbose
```

You should see: `Status: active`, rules for 80/tcp, 443/tcp and 443/udp, and no rule mentioning
8198 or 11235.

If you do not: a rule for 8198 from an earlier experiment removes with
`sudo ufw delete allow 8198`. 80/tcp answers the ACME challenge and redirects to HTTPS, 443/tcp
is the only way in, and 443/udp is HTTP/3. 8198 stays closed because compose binds it to
127.0.0.1, and 11235 is a container port never published to the host.

## 7. Start and verify

The first start pulls about 1.5 GB and launches Chromium, so allow a couple of minutes.

```bash
cd /srv/crawl4ai
docker compose pull
docker compose up -d
for i in $(seq 1 36); do code=$(curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:8198/health); echo "$i $code"; [ "$code" = 200 ] && break; sleep 5; done
curl -sS http://127.0.0.1:8198/health; echo
curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://<DOMAIN>/md -H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}'
TOKEN=$(grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env | cut -d= -f2-)
curl -sS -X POST https://<DOMAIN>/md -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}' | head -c 300; echo
unset TOKEN
curl -sSL https://<DOMAIN>/playground/ | grep -c '<title>Crawl4AI Playground</title>'
```

You should see: the loop counting up and ending on `200`; a health body containing
`"status":"ok"`; then `401` from the unauthenticated POST, which is the security check in this
block; then JSON whose `markdown` field contains `Example Domain`; then `1`. The `"f":"raw"` in
those two calls asks for the direct conversion, because the default readability filter can prune
a page this small down to nothing.

If you do not: run `docker compose logs --tail 40 crawl4ai`. A log line about binding loopback
only means `.env` did not load, so check steps 3 and 4. A 502 from Caddy with a healthy container
means step 5 is pointing somewhere else. An empty reply rather than a `401` is the same missing
token, seen from outside. A running container is not success, and neither is a green
`docker compose ps`.

Do not paste the token or the full `.env` line into this chat while you debug. Paste the HTTP
status codes, which are what actually diagnose this.

There is no sign-in page. https://<DOMAIN>/ redirects to the playground UI, which can do nothing
until a token is pasted into its token bar.

STOP: read your token with `sudo grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env`, store it in your
password manager, and understand that it is admin-scoped: there is no read-only key to hand out,
and pasting it into a browser on a shared machine hands over the whole crawler. Do not continue
until you have it stored.

Now the core loop: a real crawl, returning real markdown.

```bash
TOKEN=$(grep CRAWL4AI_API_TOKEN /srv/crawl4ai/.env | cut -d= -f2-)
curl -sS -X POST https://<DOMAIN>/crawl \
  -H "Authorization: Bearer ${TOKEN}" \
  -H 'Content-Type: application/json' \
  --data-binary '{"urls":["https://example.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"check_robots_txt":true}}}' \
  | head -c 600; echo
unset TOKEN
```

You should see: a response starting `{"success":true` with a results array carrying the crawled
page. That is the product: a URL in, structured JSON with a markdown field out.

If you do not: a `401` means the token did not make it into the header, usually because `TOKEN`
was unset by a previous paste. A `403` on a site that works in your browser is that site blocking
your server's address, not a fault in the install. `check_robots_txt` is in that request on
purpose, because it defaults to false and you should see where the polite setting lives.

This endpoint is what your own scripts call. A schedule is your own cron entry calling this URL,
because nothing in this container runs one.

## 8. First backup and restore

There is no application state to archive. This install mounts no volumes, so the backup is the
three files that rebuild it: the token, the compose file and the live Caddy site block. Do not
let yourself believe an archive is protecting crawl results, because there are none on disk.

```bash
cd /srv/crawl4ai
sudo tar -czf /srv/crawl4ai/backups/crawl4ai-$(date +%F).tar.gz -C /srv/crawl4ai .env compose.yml -C /etc/caddy Caddyfile
ls -lh /srv/crawl4ai/backups/
```

You should see: one `.tar.gz` with a non-zero size.

If you do not: a `Cannot stat` error names the file it could not find, which is almost always
`.env` because step 3 was skipped. There is no downtime here, because nothing is being written.

Treat the archive as secret material: it holds the API token. A backup on the same disk as the
data is not a backup, so run this from your own machine, not the server:

```bash
mkdir -p ~/backups/crawl4ai
scp vps:/srv/crawl4ai/backups/*.tar.gz ~/backups/crawl4ai/
```

You should see: one file copied, with a progress line.

If you do not: `ssh vps` is not configured on the machine you are typing on. That is a Prompt
Zero step, and it is worth fixing now rather than the night you need the archive.

To restore on a fresh box: complete Prompt Zero, recreate the two directories from step 2, untar
the archive into /srv/crawl4ai to bring back `.env` and `compose.yml`, append the archived
Caddyfile block to /etc/caddy/Caddyfile with your hostname substituted, validate and reload
Caddy, then `docker compose up -d` and re-run the health and 401 checks from step 7. Restoring
`.env` before the first start matters: a container that starts without the token binds loopback
and the published port answers nothing.

## 9. Updating later

New versions are listed at https://github.com/unclecode/crawl4ai/releases. The release tag carries
a leading `v` and the image tag does not, so release `v0.9.3` is image tag `0.9.3`. Take a backup
first, then edit the image line in /srv/crawl4ai/compose.yml to the new tag and its digest:

```bash
cd /srv/crawl4ai
docker compose pull
docker compose up -d
docker compose logs --tail 30 crawl4ai
```

You should see: a pull, a recreate, and a log that settles within a minute or two.

If you do not: the pull failing on a digest mismatch means the tag and digest you pasted do not
belong together. Get both from the same registry read rather than assuming the digest carried
over. After every update, re-run the health check, the unauthenticated 401 and one real crawl
before calling it done. This project moves quickly and its server API has changed shape between
minor versions, so read the release notes for endpoint changes.

## 10. What will probably go wrong

The container will look fine and answer nothing. I had a green `docker ps`, a clean log and a
connection reset on the published port, and I spent ten minutes on Caddy before reading the
container's own second line of output: no token, so it had bound its server to loopback inside
the container, where a published port cannot reach it. That is correct behaviour and it looks
exactly like a broken network. If step 7 returns nothing rather than a 401, check `.env` before
you touch anything else.

The second one is not fixable by configuration, so plan around it. A block page is still a page:
the crawl succeeds, the JSON says `"success":true`, and the markdown is a challenge screen. Read
what comes back before you build a pipeline on top of it.

## 11. Out of scope

- Do not set `CRAWL4AI_HOOKS_ENABLED` or `CRAWL4AI_EXECUTE_JS_ENABLED`. Upstream's own source
  calls them an arbitrary-code and SSRF surface and ships them off. Leave them off.
- Do not add an LLM provider API key. This install runs the readability filter, which needs no
  model, and a key here is a bill the crawler can run up on its own.
- Do not publish 11235 or 8198 in the firewall, and do not rebind either to 0.0.0.0 on the host.
  Caddy is the only way in.
- Do not build the image from the repository. The pinned digest is what this install uses.

Two closing notes on the licence and the shells, because they are the parts people find later.
The LICENSE file at this tag is Apache-2.0 with an Attribution Requirement appended after the end
of the Apache terms: any distribution, publication or public use must carry a named credit to the
author and the project. Running this container for yourself is unaffected; shipping a product or
publishing a paper built on it is not. And three paths stay public on this hostname because the
application serves them publicly: /playground, /dashboard and /static are static shells that hold
no data and can call nothing without the token, while /health is public so the container's own
healthcheck works. The Caddy block above carries the two lines that close the shells if you would
rather not serve them.
````

## Local install prompt (your own computer, no server)

````text
You are Claude Code on the user's own computer. There is no server and no Prompt Zero:
everything in this prompt runs on this machine and stays on it.

Run every command on this computer, in the shell you are already in. Nothing in this prompt
uses ssh.

Install Crawl4AI 0.9.2 under ~/selfhost/crawl4ai, answering at http://localhost:8198.

## 1. Preflight

Say three things to the user before anything installs. One: this is a crawling API on this
computer only. http://localhost:8198 is unreachable from a phone or another laptop; they get an
endpoint their own scripts can call while they work, not a shared service, and there is no
sign-in to finish. Two: the token is not a nicety. The entrypoint binds the server to
container loopback whenever no credential is configured, so an install without a token publishes
a port that answers nothing. Step 4 generates it. Three: what they crawl is their responsibility.
The crawler does not consult robots.txt unless a request asks it to, and nothing here enforces a
site's terms of service.

Crawls also leave from this machine's own address rather than a datacenter range, which far less
of the web treats as hostile. The crawled site sees the household's address, which cuts both
ways.

Detect the OS and measure the machine:

```bash
uname -s
case "$(uname -s)" in
  Darwin) vm_stat | awk '/page size/{p=$8} /free|inactive/{s+=$3} END {printf "%d MB available\n", s*p/1048576}' ;;
  Linux) . /etc/os-release && echo "$ID $VERSION_CODENAME"; free -m | awk '/^Mem:/ {print $7 " MB available of " $2 " MB"}' ;;
  MINGW*|MSYS*) powershell -Command "(Get-CimInstance Win32_OperatingSystem).FreePhysicalMemory" | awk '$1+0 {printf "%d MB available\n", $1/1024}' ;;
esac
df -h ~
```

`Darwin` is macOS, `Linux` is Linux, `MINGW` or `MSYS` is Windows under Git Bash. On Linux the
distribution ID and codename print next, for step 2. Crawl4AI needs 4096 MB of RAM available and
15 GB free on the home disk, because the image carries a real Chromium. It publishes amd64 and
arm64, so Apple Silicon is fine. If available RAM is under 4096 MB or free disk is under 15 GB,
print both numbers and stop. On macOS and Windows, Docker Desktop takes its memory out of the
host figure and its default allocation is often under 4 GB: if the container is killed mid-crawl,
raise the memory limit in Docker Desktop's settings.

## 2. Docker

Check before installing anything:

```bash
docker info >/dev/null 2>&1 && echo "docker OK" || echo "docker MISSING"
docker compose version 2>/dev/null || true
```

If that printed `docker OK` and a compose version, skip to step 3.

Otherwise, install Docker for the OS step 1 detected:

- macOS: if `command -v brew` succeeds, run `brew install --cask docker`. If there is no
  Homebrew, STOP: tell the user to download Docker Desktop from
  https://www.docker.com/products/docker-desktop/ and install it, and wait until they
  confirm. Either way, then STOP: tell the user to open Docker Desktop once, accept its
  terms, and wait for the whale icon to say it is running. Do not continue until they
  confirm.
- Windows: run `winget install -e --id Docker.DockerDesktop`. If winget is missing or the
  install fails, STOP: tell the user to download Docker Desktop from the URL above and
  install it, and wait until they confirm. Docker Desktop configures WSL 2 itself and may
  ask for a reboot; if it does, STOP and tell the user to reboot and come back, this
  prompt resumes at this step. Then STOP: have the user open Docker Desktop, accept its
  terms, and confirm it says running.
- Linux, Debian or Ubuntu: install Docker Engine from download.docker.com's apt
  repository, with its signing key saved to a file first, never piped into a shell. The
  fence is guarded, a no-op on anything but a Linux with apt:

```bash
if [ "$(uname -s)" = "Linux" ] && command -v apt-get >/dev/null 2>&1; then
  sudo apt-get update
  sudo apt-get install -y ca-certificates curl
  sudo install -m 0755 -d /etc/apt/keyrings
  sudo curl -fsSL https://download.docker.com/linux/$(. /etc/os-release && echo "$ID")/gpg -o /etc/apt/keyrings/docker.asc
  sudo chmod a+r /etc/apt/keyrings/docker.asc
  echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/$(. /etc/os-release && echo "$ID") $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
  sudo apt-get update
  sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
  sudo usermod -aG docker "$USER"
fi
```

  Adding the user to the docker group is root-equivalent on this machine; say that to the
  user in one sentence, and tell them the group change lands at their next login.
- Linux, anything else: STOP. Tell the user to install Docker Engine and the compose
  plugin with their distribution's package manager, and to run this prompt again once
  `docker info` works.

Assert: `docker info` exits 0 and `docker compose version` prints a version. Do not
continue without both.


## 3. Layout

```bash
mkdir -p ~/selfhost/crawl4ai/backups
ls -la ~/selfhost/crawl4ai
```

Assert: `backups` exists. There is no `data` directory, which is deliberate: this compose file
mounts nothing. The crawl cache, the artifact store and the in-container Redis are disposable,
and a home bind mount would land on a uid the image does not run as.

## 4. Secrets

One secret: `CRAWL4AI_API_TOKEN`. Generate it here. Do not print it into chat.

```bash
umask 077
cat > ~/selfhost/crawl4ai/.env <<EOF
CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)
EOF
chmod 600 ~/selfhost/crawl4ai/.env
umask 022
ls -l ~/selfhost/crawl4ai/.env
```

Assert: mode `-rw-------`. On Windows those mode bits are advisory; the file is still protected
by the user's own account, and on a single-user machine that is the real boundary. Tell the user
they read it back with `grep CRAWL4AI_API_TOKEN ~/selfhost/crawl4ai/.env`, and that it is
admin-scoped: there is no read-only key, and anyone holding it can make this computer fetch any
URL it can reach, including the home network.

## 5. compose.yml

```bash
cat > ~/selfhost/crawl4ai/compose.yml <<'EOF'
# Crawl4AI · the deterministic fallback for the local path. Authored by
# caniselfhostit from the upstream documentation, not copied from a repository:
#   docker guide ........ https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md
#   server config ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/config.yml
#   bind and auth ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/entrypoint.sh
#   license ............. https://github.com/unclecode/crawl4ai/blob/v0.9.2/LICENSE
#
# One container on the computer you are sitting at. The image bakes Chromium in
# through Playwright and runs its own Redis on container loopback, so there is
# no second service and no database. CRAWL4AI_API_TOKEN comes from ./.env and is
# not optional: entrypoint.sh binds gunicorn to container loopback when no
# credential is set, and the published port would then reach nothing at all.
# GUNICORN_BIND is written out so the bind does not depend on IPv6 existing
# inside the container. Tag and digest are the 0.9.2 release read from Docker
# Hub on 2026-08-14; the manifest list carries linux/amd64 and linux/arm64.
#
# Nothing is mounted on purpose: the crawl cache, the artifact store and the
# Redis working directory live inside the container and are meant to be thrown
# away, and a home-directory bind mount would land on a uid the image does not
# run as anyway.
#
# NOT YET VERIFIED: no harness run has been recorded against this file.

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.2@sha256:bd36741e7bdd35ddc1a05d9183e1d6d8cefb61dd640d944a25d026b76e917690
    container_name: crawl4ai
    restart: unless-stopped
    env_file: ./.env
    environment:
      # entrypoint.sh honours GUNICORN_BIND only when a credential is present.
      GUNICORN_BIND: "0.0.0.0:11235"
    # Chromium wants shared memory. Without this it dies on heavy pages.
    shm_size: "1gb"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 512
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    ports:
      # Loopback only: no other device on the wifi can reach 8198.
      - "127.0.0.1:8198:11235"
EOF
cd ~/selfhost/crawl4ai && docker compose config >/dev/null && echo "compose OK"
```

Assert: that prints `compose OK`. One service, one port, no volumes, no database.

## 6. Nothing is public

Nothing to open, and nothing to certify. Everything binds to loopback: no domain, no certificate
because there is nothing to certify, and no other device can reach this, including the user's own
phone. That is the point of this path, not a defect. Confirm the binding:

```bash
grep -c '"127.0.0.1:' ~/selfhost/crawl4ai/compose.yml
```

Assert: that prints `1`. Do not rebind to `0.0.0.0` and do not forward a router port at it. An
open crawling API on a home line will fetch any URL a stranger names.

## 7. Start and verify

The first start pulls about 1.5 GB and launches Chromium, so allow a couple of minutes.

```bash
cd ~/selfhost/crawl4ai
docker compose pull
docker compose up -d
for i in $(seq 1 36); do code=$(curl -sS -o /dev/null -w '%{http_code}' http://localhost:8198/health); echo "$i $code"; [ "$code" = 200 ] && break; sleep 5; done
curl -sS http://localhost:8198/health; echo
curl -sS -o /dev/null -w '%{http_code}\n' -X POST http://localhost:8198/md -H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}'
TOKEN=$(grep CRAWL4AI_API_TOKEN ~/selfhost/crawl4ai/.env | cut -d= -f2-)
curl -sS -X POST http://localhost:8198/md -H "Authorization: Bearer ${TOKEN}" -H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}' | head -c 300; echo
unset TOKEN
curl -sSL http://localhost:8198/playground/ | grep -c '<title>Crawl4AI Playground</title>'
```

Assert all five and print what each returned, never the token. The health loop ends on `200` and
the body contains `"status":"ok"`. The unauthenticated POST to `/md` prints `401`: that is the
security assert in this block. The authenticated POST returns JSON whose `markdown` field contains
`Example Domain`; `"f":"raw"` asks for the direct conversion, because the default readability
filter can prune a page this small down to nothing. The last command prints `1`.

If any assert misses, stop and run `docker compose logs --tail 40 crawl4ai`. A log line about
binding loopback only means `.env` did not load, which is step 4 or step 5. If `port is already
allocated` came back, find what holds 8198 (`lsof -nP -iTCP:8198 -sTCP:LISTEN` on macOS,
`ss -ltnp | grep 8198` on Linux, `netstat -ano | findstr :8198` on Windows) and stop.
A running container is not success.

STOP: tell the user to open http://localhost:8198/playground/, paste the token from
`grep CRAWL4AI_API_TOKEN ~/selfhost/crawl4ai/.env` into the token bar at the top right, press
Set, and confirm they can run one crawl from that page. Do not continue until they confirm. That
token lives in the tab's session storage only, so closing the tab clears it.

Once they confirm, run the core loop from the shell, which is what a script does:

```bash
TOKEN=$(grep CRAWL4AI_API_TOKEN ~/selfhost/crawl4ai/.env | cut -d= -f2-)
curl -sS -X POST http://localhost:8198/crawl \
  -H "Authorization: Bearer ${TOKEN}" \
  -H 'Content-Type: application/json' \
  --data-binary '{"urls":["https://example.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"check_robots_txt":true}}}' \
  | head -c 600; echo
unset TOKEN
```

Assert: the response starts `{"success":true` and the results array carries the crawled page.
That is the product: a URL in, structured JSON with a markdown field out. `check_robots_txt` is
there on purpose, because it defaults to false and the polite setting should be visible.

## 8. First backup and restore

There is no application state to archive. This service mounts no volumes, so the backup is the
two files that rebuild it: the token and the compose file. Say that, rather than letting the user
think an archive protects crawls.

```bash
cd ~/selfhost/crawl4ai
tar -C ~/selfhost/crawl4ai -czf ~/selfhost/crawl4ai/backups/crawl4ai-$(date +%F).tar.gz .env compose.yml
ls -lh ~/selfhost/crawl4ai/backups/
```

Assert: the archive exists and is non-empty. Print its size. It holds the API token, so treat it
as secret. A backup on the same disk is not a backup, and on a laptop the disk and the machine
fail together: ask for a destination off this computer, a synced folder or a USB stick, and copy
it there with `cp`.

To restore: recreate `~/selfhost/crawl4ai/backups`, untar the archive into `~/selfhost/crawl4ai`,
then `docker compose up -d` and re-run step 7's checks. Restoring `.env` before the first start
matters: a container that starts without the token binds loopback and answers nothing.

## 9. Updating later

New versions are listed at https://github.com/unclecode/crawl4ai/releases. The release tag
carries a leading `v` and the image tag does not, so release `v0.9.3` is image tag `0.9.3`. Back
up first, then edit the image line in ~/selfhost/crawl4ai/compose.yml to the new tag and digest:

```bash
cd ~/selfhost/crawl4ai
docker compose pull
docker compose up -d
docker compose logs --tail 30 crawl4ai
```

Watch that log until it settles, then re-run step 7's health check, the unauthenticated 401 and
one real crawl. This project moves quickly and its server API has changed shape between minor
versions, so read the release notes.

## 10. What will probably go wrong

The container will look fine and answer nothing. I had a green `docker ps`, a clean log and a
connection reset on 8198, and I spent ten minutes blaming Docker Desktop before reading the
container's second line of output: no token, so it had bound its server to loopback inside the
container where a published port cannot reach it. Correct behaviour, and it looks exactly like a
broken install. If step 7 returns nothing rather than a 401, check `.env` first.

Two smaller ones. Docker Desktop does not start itself after a reboot on most machines, so the
morning after the install this endpoint is gone until the whale icon is back. And a crawl on a
laptop that sleeps does not pause politely, it fails: keep the lid open for the long ones.

## 11. Out of scope

- Do not expose this to the internet.
- Do not configure port forwarding on the router.
- Do not add a reverse proxy or TLS.
- Do not set `CRAWL4AI_HOOKS_ENABLED` or `CRAWL4AI_EXECUTE_JS_ENABLED`. Upstream's own source
  calls them an arbitrary-code and SSRF surface and ships them off.
- Do not add an LLM provider API key. This install runs the readability filter, which needs no
  model, and a key here is a bill the crawler can run up.
````

## docker-compose.yml

```yaml
# Crawl4AI · the deterministic fallback. Authored by
# caniselfhostit from the upstream documentation, not copied from a repository:
#   docker guide ........ https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md
#   server config ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/config.yml
#   bind and auth ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/entrypoint.sh
#   license ............. https://github.com/unclecode/crawl4ai/blob/v0.9.2/LICENSE
#
# One container. The image bakes Chromium in through Playwright and runs its own
# Redis on container loopback, so there is no second service and no database.
# CRAWL4AI_API_TOKEN comes from env_file and is not optional: entrypoint.sh
# binds gunicorn to container loopback when no credential is set, and the
# published port would then reach nothing. GUNICORN_BIND is spelled out so the
# bind does not depend on IPv6. Tag and digest are the 0.9.2 release read from
# Docker Hub on 2026-08-14; the manifest list carries amd64 and arm64.
#
# Nothing is mounted on purpose: the crawl cache, the artifact store and the
# Redis working directory live inside the container and are disposable.
# Upstream's compose adds read_only with a tmpfs list keyed to uid 999, not
# copied here because the image creates its runtime user with `useradd -r`
# after installing redis-server, so that uid is not knowable from the source
# and a wrong one leaves Redis unable to write.
#
# NOT YET VERIFIED: no harness run has been recorded against this file.

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.2@sha256:bd36741e7bdd35ddc1a05d9183e1d6d8cefb61dd640d944a25d026b76e917690
    container_name: crawl4ai
    restart: unless-stopped
    env_file: /srv/crawl4ai/.env
    environment:
      # entrypoint.sh honours GUNICORN_BIND only when a credential is present.
      GUNICORN_BIND: "0.0.0.0:11235"
    # Chromium wants shared memory. Without this it dies on heavy pages.
    shm_size: "1gb"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 512
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    ports:
      # Loopback only: the host's Caddy is the only thing that reaches 8198.
      - "127.0.0.1:8198:11235"
```

## compose.local.yml

```yaml
# Crawl4AI · the deterministic fallback for the local path. Authored by
# caniselfhostit from the upstream documentation, not copied from a repository:
#   docker guide ........ https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md
#   server config ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/config.yml
#   bind and auth ....... https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/entrypoint.sh
#   license ............. https://github.com/unclecode/crawl4ai/blob/v0.9.2/LICENSE
#
# One container on the computer you are sitting at. The image bakes Chromium in
# through Playwright and runs its own Redis on container loopback, so there is
# no second service and no database. CRAWL4AI_API_TOKEN comes from ./.env and is
# not optional: entrypoint.sh binds gunicorn to container loopback when no
# credential is set, and the published port would then reach nothing at all.
# GUNICORN_BIND is written out so the bind does not depend on IPv6 existing
# inside the container. Tag and digest are the 0.9.2 release read from Docker
# Hub on 2026-08-14; the manifest list carries linux/amd64 and linux/arm64.
#
# Nothing is mounted on purpose: the crawl cache, the artifact store and the
# Redis working directory live inside the container and are meant to be thrown
# away, and a home-directory bind mount would land on a uid the image does not
# run as anyway.
#
# NOT YET VERIFIED: no harness run has been recorded against this file.

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.2@sha256:bd36741e7bdd35ddc1a05d9183e1d6d8cefb61dd640d944a25d026b76e917690
    container_name: crawl4ai
    restart: unless-stopped
    env_file: ./.env
    environment:
      # entrypoint.sh honours GUNICORN_BIND only when a credential is present.
      GUNICORN_BIND: "0.0.0.0:11235"
    # Chromium wants shared memory. Without this it dies on heavy pages.
    shm_size: "1gb"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 512
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    ports:
      # Loopback only: no other device on the wifi can reach 8198.
      - "127.0.0.1:8198:11235"
```

## Caddyfile

```text
# Crawl4AI · the Caddy site block for this service.
#
# Authored by caniselfhostit from
# https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md and
# https://caddyserver.com/docs/automatic-https
#
# Append this to /etc/caddy/Caddyfile, the Caddy that Prompt Zero installed, with
# <DOMAIN> replaced by the hostname pointed at this box. Caddy runs under systemd
# on the host. There is no Caddy container anywhere in this project. Auth is the
# CRAWL4AI_API_TOKEN inside the container, not a browser login form: every API
# route answers 401 without an Authorization: Bearer header, and the application
# sets its own security headers, so this block adds only HSTS.
#
# Three prefixes stay public because the application serves them publicly:
# /playground, /dashboard and /static are static shells holding no data that
# cannot call the API without the token. /health is public because the
# container's healthcheck calls it. To stop serving the shells, add these two
# lines inside the site block, above reverse_proxy:
#
#	@ui path / /playground* /dashboard* /static*
#	respond @ui 404

<DOMAIN> {
	encode zstd gzip

	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		-Server
	}

	# 8198 is the loopback port compose publishes; it is never in the firewall.
	# A slow crawl holds the connection open for a minute. Caddy sets no read
	# timeout on a reverse-proxied response by default, so that is fine.
	reverse_proxy 127.0.0.1:8198
}
```

## install.sh

```bash
#!/usr/bin/env bash
# Crawl4AI · the agent-free install.
#
# Everything prompt.md tells an agent to do, as a script you can read first.
# Run it on the VPS, as a non-root user who is in the docker group:
#
#   DOMAIN_HOST=crawl.example.com ./install.sh
#
# Authored by caniselfhostit from the upstream documentation:
#   https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/README.md
#   https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/config.yml
#   https://github.com/unclecode/crawl4ai/blob/v0.9.2/deploy/docker/entrypoint.sh
#   https://github.com/unclecode/crawl4ai/blob/v0.9.2/LICENSE
#
# One secret is generated here: CRAWL4AI_API_TOKEN. It goes into
# /srv/crawl4ai/.env with mode 600 and is never printed. There is no browser
# sign-in. The token is not optional: without it the container's entrypoint
# binds its server to container loopback and the published port answers
# nothing. With it, every API route returns 401 without a Bearer header.
#
# NOT YET VERIFIED: no harness run has been recorded against this script.
set -euo pipefail

APP_DIR="${APP_DIR:-/srv/crawl4ai}"
DOMAIN_HOST="${DOMAIN_HOST:-}"

die() { printf 'install.sh: %s\n' "$1" >&2; exit 1; }

# --- 1. Refuse to start on a machine that is not ready -----------------------

[ -n "$DOMAIN_HOST" ] || die "set DOMAIN_HOST to the hostname you pointed at this server, e.g. crawl.example.com"
command -v docker >/dev/null 2>&1 || die "docker is not installed. Run Prompt Zero first."
docker compose version >/dev/null 2>&1 || die "the docker compose plugin is missing"
command -v caddy >/dev/null 2>&1 || die "caddy is not installed on the host. Run Prompt Zero first."
command -v openssl >/dev/null 2>&1 || die "openssl is not installed"

avail_mb="$(free -m | awk '/^Mem:/ {print $7}')"
[ "$avail_mb" -ge 4096 ] || die "only ${avail_mb} MB of RAM available; this install wants 4096 MB because the image runs a real Chromium"
avail_gb="$(df -BG --output=avail /srv | tail -1 | tr -dc '0-9')"
[ "$avail_gb" -ge 15 ] || die "only ${avail_gb} GB free on /srv; this install wants 15 GB"

resolved="$(getent hosts "$DOMAIN_HOST" | awk '{print $1; exit}' || true)"
[ -n "$resolved" ] || die "$DOMAIN_HOST does not resolve yet. Add the A record, wait a minute, run this again."

# --- 2. Lay the files out ----------------------------------------------------

sudo install -d -m 750 -o "$(id -u)" -g "$(id -g)" "$APP_DIR" "$APP_DIR/backups"
install -m 0644 "$(dirname "$0")/compose.yml" "$APP_DIR/compose.yml"
install -m 0644 "$(dirname "$0")/Caddyfile" "$APP_DIR/Caddyfile"

# No data directory: this compose file mounts nothing. The crawl cache, the
# artifact store and the in-container Redis are all disposable.

# --- 3. API token on the server ----------------------------------------------

if [ ! -f "$APP_DIR/.env" ]; then
	umask 077
	cat > "$APP_DIR/.env" <<-EOF
		CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)
	EOF
	chmod 600 "$APP_DIR/.env"
	umask 022
fi

cd "$APP_DIR"
docker compose config >/dev/null

# --- 4. Caddy site block -----------------------------------------------------

if ! sudo grep -qF "$DOMAIN_HOST {" /etc/caddy/Caddyfile; then
	sudo cp /etc/caddy/Caddyfile "/etc/caddy/Caddyfile.before-crawl4ai"
	printf '\n' | sudo tee -a /etc/caddy/Caddyfile >/dev/null
	sed "s|<DOMAIN>|${DOMAIN_HOST}|g" "$APP_DIR/Caddyfile" | sudo tee -a /etc/caddy/Caddyfile >/dev/null
fi
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy

# --- 5. Firewall -------------------------------------------------------------

if command -v ufw >/dev/null 2>&1; then
	echo "==> 80/tcp and 443/tcp for Caddy, 443/udp for HTTP/3; 8198 stays closed"
	sudo ufw allow 80/tcp
	sudo ufw allow 443/tcp
	sudo ufw allow 443/udp
	sudo ufw status verbose
fi

# --- 6. Start and assert -----------------------------------------------------

docker compose pull
docker compose up -d

echo "==> waiting for http://127.0.0.1:8198/health (the image is large and starts a browser)"
for _ in $(seq 1 36); do
	code="$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:8198/health" || true)"
	[ "$code" = "200" ] && break
	sleep 5
done
[ "${code:-}" = "200" ] || die "/health answered ${code:-nothing}. Check: docker compose logs --tail 40 crawl4ai"

unauth="$(curl -sS -o /dev/null -w '%{http_code}' -X POST "https://${DOMAIN_HOST}/md" \
	-H 'Content-Type: application/json' --data-binary '{"url":"https://example.com","f":"raw"}' || true)"
[ "$unauth" = "401" ] || die "unauthenticated POST /md returned ${unauth}, not 401. Stop and investigate."

TOKEN="$(grep CRAWL4AI_API_TOKEN "$APP_DIR/.env" | cut -d= -f2-)"
curl -sS -X POST "https://${DOMAIN_HOST}/md" \
	-H "Authorization: Bearer ${TOKEN}" \
	-H 'Content-Type: application/json' \
	--data-binary '{"url":"https://example.com","f":"raw"}' | grep -q 'Example Domain' \
	|| die "the authenticated /md call did not return the crawled page"

curl -sS -X POST "https://${DOMAIN_HOST}/crawl" \
	-H "Authorization: Bearer ${TOKEN}" \
	-H 'Content-Type: application/json' \
	--data-binary '{"urls":["https://example.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"check_robots_txt":true}}}' \
	| grep -q '"success":true' \
	|| die "the authenticated /crawl call did not report success"
unset TOKEN

curl -sSL "https://${DOMAIN_HOST}/playground/" | grep -q '<title>Crawl4AI Playground</title>' \
	|| die "the playground UI did not answer on https://${DOMAIN_HOST}/playground/"

# --- 7. First backup ---------------------------------------------------------

STAMP="$(date +%Y%m%d-%H%M%S)"
sudo tar -czf "$APP_DIR/backups/crawl4ai-${STAMP}.tar.gz" \
	-C "$APP_DIR" .env compose.yml -C /etc/caddy Caddyfile
ls -lh "$APP_DIR/backups/"
[ -s "$APP_DIR/backups/crawl4ai-${STAMP}.tar.gz" ] || die "the backup archive is empty"

cat <<-DONE

	Crawl4AI is answering at https://${DOMAIN_HOST}

	  1. There is no browser sign-in. The API expects Authorization: Bearer.
	  2. Read the API token with:
	       sudo grep CRAWL4AI_API_TOKEN ${APP_DIR}/.env
	     Store it offline. It is admin-scoped and there is no read-only key.
	  3. Crawl one page, with that value in the Authorization header:
	       curl -sS -X POST https://${DOMAIN_HOST}/crawl \\
	         -H "Authorization: Bearer <the value from .env>" \\
	         -H 'Content-Type: application/json' \\
	         --data-binary '{"urls":["https://example.com"]}'
	  4. Unauthenticated POST /md returns 401 (asserted above).
	  5. The playground is at https://${DOMAIN_HOST}/playground/ and needs the
	     same token pasted into its token bar.
	  6. First backup at ${APP_DIR}/backups (includes .env). Copy it off this disk.
	     Nothing else is on disk: this install mounts no volumes.
	  7. NOT YET VERIFIED on a clean harness machine.

DONE
```

The page this mirrors: https://caniselfhostit.com/self-host/apify/ · How the verdict, the timings and the prices are derived: https://caniselfhostit.com/methodology/ · Source, data and corrections: https://github.com/caniselfhostit/caniselfhostit