GeoItAuditBot

GeoItAuditBot is the web crawler used by GeoIt to run site audits.

What it does

It visits the pages of a website to check technical and content issues (for example status codes, robots rules, canonical tags, structured data and page text) and produces an audit report. It does not log in, submit forms, or bypass paywalls or access controls.

When it visits

Only when an audit of a site has been ordered. Each crawl is started by a person and covers only the sites named in the order, within its page limit. Site audits read publicly available pages only.

How it identifies itself

User-agent: GeoItAuditBot/0.1.0 (+https://geoitauditbot.dealflowagent.vc/)

How it behaves

• It reads and follows robots.txt: the rules for GeoItAuditBot, or the rules for all crawlers (*) when there is no GeoItAuditBot group.
• It sends one request at a time to a site, at least 2 seconds apart, and waits longer if robots.txt sets a Crawl-delay (up to 60 seconds).
• If a server answers 429 or 503, it waits (following Retry-After, up to 2 minutes), and it stops crawling that site after three such answers in a row.
• It fetches a limited number of pages per audit (at most 200 for a site audit).

How to block it

Add this to your robots.txt:

User-agent: GeoItAuditBot
Disallow: /

Contact

Questions or problems with our crawler: support@mekomiai.com