SiteTellsBot

SiteTellsBot is the client behind SiteTells, a tool that shows what a website is built with: its platform, theme, plugins and hosting. It visits a site when someone asks SiteTells about it, and when SiteTells surveys the web to count how widely each technology is used.

What it requests

• robots.txt, before anything else.
• The home page, and up to two more public pages linked from it.
• On WordPress sites: the public REST API index and its list of content types, the feed, and the active theme’s style.css.
• Sometimes a few of the stylesheets and scripts the page already loads, to identify them by their content.
• At most 10 page requests per visit, all plain GET requests. It does not log in, submit forms, or look for files that are not linked.

How to recognise it

Requests carry this user agent:

Mozilla/5.0 (compatible; SiteTellsBot/0.1; +https://sitetells.com/bot)

How to switch it off

Add this to your robots.txt. SiteTellsBot reads the file first and will not fetch anything else from your site, whether the visit is part of a survey or someone asked about your site directly:

User-agent: SiteTellsBot
Disallow: /

When it surveys the web, SiteTellsBot also follows the rules you set for all crawlers (User-agent: *). You can block the user agent in your firewall as well.

Contact

Questions, or a request to leave your site out: bot@sitetells.com.