Indexify SEO Platform
About our crawler
If you have seen our crawler in your website logs, this page explains what it is and what it does.
What it does
- It only visits websites whose owners have signed up to Indexify SEO Platform, to read their pages and suggest improvements.
- It reads at most 30 pages per visit, and visits again every few days to check whether suggested changes are live.
- It does not crawl competitors' websites. Information about competitors comes from search data, not from visiting their sites.
- It does not fill in forms, log in or buy anything.
How to recognise it
It identifies itself with this user agent:
Mozilla/5.0 (compatible; IndexifySEOPlatformBot/1.0; +https://app.indexify.co.uk/bot)
It respects robots.txt
The crawler reads your robots.txt file before visiting and follows its rules. To block it from your whole site, add this to robots.txt:
User-agent: IndexifySEOPlatformBot Disallow: /
If your site is blocked, we cannot check your pages, so the suggestions for that site will be limited.
Contact
If the crawler is causing a problem, or you did not expect it on your site, please email info@indexify.co and we will look into it.