About the crawler

CodoSEObot

CodoSEObot is the crawler behind CodoSEO audits. Someone asked us to check your site, usually its owner or an SEO working for them. It reads public pages to find SEO problems such as broken links, missing titles and noindex tags. It never submits forms, logs in or changes anything.

How it identifies itself

Every request sends this user agent:

CodoSEObot/0.1 (+https://codoseo.com/bot)

How politely it crawls

• At most 5 requests per second and 2 connections at a time, per site.
• It follows robots.txt, including Crawl-delay (up to 10 seconds) and Disallow rules for CodoSEObot. A site that blocks all crawlers is left alone.
• On a 429 or 503 it waits for Retry-After and halves its speed. After 20 failures in a row it stops.
• It reads pages up to 5 MB, never stores a page's HTML, and a free audit stops after 100 pages.

Allow or block it

To keep CodoSEObot out of part of your site, or all of it, add this to robots.txt:

User-agent: CodoSEObot
Disallow: /private/

Use Disallow: / to block it completely. If your firewall or bot protection stops it, allow the user agent above (and the address, if listed) to let audits through.

Questions

Open an issue on GitHub and mention your domain. CodoSEO is open source, so you can read exactly what the crawler does.