DocumentationMCP ToolsAPI ReferenceGuidesResources
Documentation/Integrations/Rampify Crawler

Rampify Crawler

When you scan a site, Rampify fetches its pages with its own crawler: the sitemap first, then each page to record its title, meta tags, links, structured data and status. If your host's bot protection decides those requests look automated, it can rate-limit or challenge them, and Rampify reports the scan as blocked instead of filling your report with error pages.

How it identifies itself#

Every request the crawler makes carries this user agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 RampifyBot/1.0 (+https://www.rampify.dev/docs/integrations/crawler)

To match it in a firewall rule, check that the user agent contains RampifyBot. The crawler runs on cloud infrastructure without a fixed set of IP addresses, so match on the user agent rather than an IP.

How fast it crawls#

The crawler checks pages a few at a time with a short, randomized pause between groups, so a small site is scanned in a minute or two. If a scan comes back blocked by status codes alone (for example every page returning 429 Too Many Requests or 403), Rampify waits several minutes and retries once at a slower pace before it reports the block. Many hosts rate-limit a burst of requests and then let the same requests through a few minutes later, so this retry clears a lot of blocks on its own.

What a blocked scan looks like#

In the dashboard and to your AI tool, a blocked scan shows a message like "Crawl blocked: example.com served an anti-bot gate for most URLs", with the status codes Rampify saw. Rampify keeps your last good scan and doesn't file findings from the blocked one, so nothing in your report is overwritten with error pages.

Allowlist it on Vercel#

If your site is hosted on Vercel with Firewall or Attack Challenge Mode turned on:

  1. Open your project in the Vercel dashboard and go to Firewall.
  2. Add a custom rule: if User Agent contains RampifyBot, then Bypass.
  3. Save and publish the rule, then run the scan again (from your AI tool with crawl_site, or from the Website page in Rampify).
Attack Challenge Mode

While Attack Challenge Mode is on, Vercel challenges every visitor that can't run its browser check, including crawlers. Turn it off once the attack has passed, or keep the bypass rule above in place.

Allowlist it on Cloudflare#

If your domain is proxied through Cloudflare:

  1. In the Cloudflare dashboard, go to Security and open WAF custom rules.
  2. Create a rule where User Agent contains RampifyBot, with the action Skip.
  3. Choose what to skip: the remaining custom rules, rate limiting rules, managed rules, and Super Bot Fight Mode if your plan has it.
  4. Deploy the rule and run the scan again.
Free plan Bot Fight Mode

Cloudflare's free-plan Bot Fight Mode can't be bypassed with a rule. If it's on and your scans are blocked, turning it off is the only way to let the crawler through on the free plan.

Other hosts#

Most firewalls and bot managers (Netlify, Fastly, Akamai, Sucuri, Wordfence and others) let you allow traffic by user agent. Add a rule that allows requests whose user agent contains RampifyBot, then run the scan again. If your host can't do that, contact us and we'll help you find another way.