Whitelisting Cromojo's crawler in Cloudflare

Written By David

Last updated 22 days ago

If your site is protected by Cloudflare, its security rules can block Cromojo from reaching your sitemap. When that happens, your pages won't be picked up for indexing and your dashboard may show 0 indexed / 0 in progress even though you've added URLs.

The fix is to create a Cloudflare rule that lets Cromojo's crawler IPs through.

Cromojo's crawler IPs

Whitelist both of these:

  • 178.104.46.97 — sitemap access (indexing)

  • 178.104.136.223 — page monitoring

Adding both ensures indexing and page monitoring work without interruption.

How to whitelist them in Cloudflare

  1. Log in to your Cloudflare dashboard and select the website you're working on.

  2. Go to Security → Security rules → Custom rules, then click Create rule.

  3. Give the rule a name, e.g. Allow Cromojo crawler.

  4. Under When incoming requests match…, set:

    • Field: IP Source Address

    • Operator: equals

    • Value: 178.104.46.97

  5. Click Or to add a second row for the other IP:

    • Field: IP Source Address

    • Operator: equals

    • Value: 178.104.136.223

    Your Expression Preview should read:

 (ip.src eq 178.104.46.97) or (ip.src eq 178.104.136.223)

  1. Under Then take action…, open Choose action and select Skip. This tells Cloudflare to let our crawler through instead of challenging or blocking it. If Cloudflare shows options for what to skip, enable skipping the remaining custom rules and managed rules so nothing else blocks us.

    ⚠️ Don't choose Block or Managed Challenge — those will keep our crawler locked out. Skip is the one that allows it in.

  2. When you choose Skip, a WAF components to skip panel appears. Tick all of these boxes:

    • ☑ All remaining custom rules

    • ☑ All rate limiting rules

    • ☑ All managed rules

    • ☑ All Super Bot Fight Mode Rules

    (Super Bot Fight Mode is the setting most likely to be blocking our crawler, so this one matters most.) Because the rule only matches our two crawler IPs, these skips apply only to Cromojo's crawler — never to your real visitors — so it's safe to check them all. You can leave More components to skip alone.

  3. Click Deploy.

After whitelisting

Re-run your indexing (or give it a little time) and your pages should start processing normally. If they still don't appear after a while, reach out to support and we can take a closer look.

Prefer not to touch Cloudflare?

No problem — contact support and we can import your URLs manually on our side as a workaround so your pages still get indexed.