Whitelisting Cromojo's crawler on SiteGround

If your site is hosted on SiteGround, its security system can block Cromojo's crawler even when your sitemaps and files are valid. This article explains how to spot the problem and fix it.

Symptoms

You may see one or more of these:

• "The URL does not contain valid sitemap XML" on every sitemap you submit, even though the sitemaps open fine in your browser and pass other validators.

• IndexNow verification fails with a message saying the key file doesn't exist, even though you can open it yourself.

• No new pages being submitted for indexing.

These errors can continue even after you whitelist Cromojo in Cloudflare or turn off the Cloudflare proxy. That's because the block happens on SiteGround's own servers.

Why it happens

SiteGround runs an Anti-Bot AI system on all its servers. When it decides a request looks like a bot, it sends back a challenge page (HTTP 202 with an sg-captcha: challenge header) instead of your actual file.

Cromojo's crawler receives that challenge page instead of your sitemap or IndexNow key file, so the check fails. The challenge doesn't trigger on every request. That's why SiteGround's logs may show some successful requests from Cromojo.

How to fix it

Only SiteGround support can whitelist an IP in their Anti-Bot AI. Adding the IP to the IP block list in Site Tools won't help, because that list doesn't control the Anti-Bot system.

1. Contact SiteGround support through your SiteGround dashboard.

2. Ask them to whitelist Cromojo's IP address in their Anti-Bot AI / SG-Captcha. You can copy and paste this message:

Please whitelist IP 178.104.46.97 in your Anti-Bot AI / SG-Captcha for [yourdomain.com]. It's the crawler for Cromojo, an indexing tool I use, and it's receiving HTTP 202 responses with the "sg-captcha: challenge" header instead of my sitemap files and IndexNow key file. Please whitelist it permanently.

3. If you also use Page Performance scans, ask them to whitelist 178.104.136.223 as well.

4. Once SiteGround confirms, go back to Cromojo and:

   • recheck your sitemaps,

   • click Verify on IndexNow, if you use it.

If SiteGround replies that they can't whitelist IPs, mention the Anti-Bot AI by name. General firewall requests are sometimes handled differently from Anti-Bot requests.

Cromojo's crawler IPs

Using Cloudflare too?

If your SiteGround site is also behind Cloudflare, you need to whitelist Cromojo in both places. See Whitelisting Cromojo's crawler in Cloudflare.

Still not working?

Contact us at support@cromojo.com with your domain and the sitemap URL that's failing. We'll check our crawler logs to see exactly what your server is sending back.