Extra Tools
Issues with Crawling a Domain

Issues with Crawling a Domain

Whitelist SplitSignalBot:

To whitelist the bot, contact your webmaster or hosting provider and ask them to whitelist SplitSignalBot.

The bot’s IP subnetworks are: and

The bot is using standard 80 HTTP and 443 HTTPS ports to connect.

If you use any plugins (Wordpress, for example) or CDNs (content delivery networks) to manage your site, you will have to whitelist the bot IP within those as well.

For whitelisting on Wordpress, contact Wordpress support.

Common CDNs that block our crawler include:

  • Cloudflare - read how to whitelist here.
  • Incapsula - read how to whitelist here (add Semrush as a “Good bot”).
  • ModSecurity - read how to whitelist here.
  • Sucuri - read how to whitelist here.

Please note: If you have shared hosting, it is possible that your hosting provider may not allow you to whitelist any bots or edit the Robots.txt file.

Hosting Providers

Below is a list of some of the most popular hosting providers on the web and how to whitelist a bot on each or reach their support team for assistance: 

* Please note: these instructions work for HostGator and Bluehost if you have a website on VPS or Dedicated Hosting.

Visualizza di più
Visualizza di più