Site Punchlist

For site administrators

About SitePunchlistBot

If you have arrived here from your server logs, this page tells you what our crawler is, how it behaves, and how to allow or block it.

How to identify it

It sends this user-agent string:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/131.0.0.0 Safari/537.36 SitePunchlistBot/1.0 (+https://sitepunchlist.com/bot)

It really is Chromium — the pages have to render for the test to mean anything — so the Chrome portion of that string is accurate rather than a disguise. Match on SitePunchlistBot.

How it behaves

  • It reads /robots.txt before anything else and obeys it, including Crawl-delay. A group naming SitePunchlistBot takes precedence over *.
  • It requests at most three pages at a time, and only one at a time if you have set a crawl delay.
  • It stays on one site and does not follow links off it.
  • It requests HTML only, and skips images, media, archives and documents.
  • It does not submit forms, log in, or click anything. It loads a page, reads the rendered result, and leaves.
  • It runs when somebody asks it to — either a free single-page test, or a crawl somebody has paid for. It does not roam.
  • If your firewall challenges it, it stops. We do not attempt to defeat bot protection, and we tell the customer their site blocked the test.

To block it

User-agent: SitePunchlistBot
Disallow: /

That is honoured on the next request; nothing is cached between runs.

To allow it through a firewall

Add a rule allowing requests whose user-agent contains SitePunchlistBot. On Cloudflare that is a WAF custom rule with http.user_agent contains "SitePunchlistBot" set to Skip. We cannot give you a fixed IP range — the crawler runs on shared infrastructure — so the user-agent string is the reliable signal.

Something wrong?

If the crawler has behaved badly on your site, email support@sitepunchlist.com with a log excerpt and we will look at it the same day.