AgencyFix Research Crawler

Last updated: October 2026

AgencyFix Research operates an automated web crawler called AgencyFixResearchBot. If you found this page in your server logs, this is what that visitor is, why it came, and how to ask us not to come back.

What it does

The crawler supports aggregate research and analysis of publicly accessible business websites — for example, how local service businesses present their contact options, published hours and services online. Results are reported only as statistics across many businesses. We do not publish rankings of individual businesses from this research, and we do not name individual small businesses negatively.

How it identifies itself

Every request carries this user-agent string:

AgencyFixResearchBot/1.0 (+https://agencyfix.com/research-bot/)

The crawler identifies itself openly. It does not disguise itself as a web browser or as another company’s crawler. Some pages are rendered through a third-party rendering service acting on our behalf; those requests carry the same user-agent.

What it collects — and what it doesn’t

• Only publicly accessible pages: typically ten or fewer per website, such as the home, contact, service and about pages.
• It never logs in, never submits forms, never books appointments and never bypasses paywalls or access controls.
• It does not attempt to collect personal information. Research results are published in aggregate only.
• It requests pages slowly and visits a site infrequently, so it should have no noticeable effect on your server.

How to opt out

Option 1 — robots.txt (automatic). The crawler reads and obeys your robots.txt before fetching any page. To exclude your whole site, add:

User-agent: AgencyFixResearchBot
Disallow: /

You can also disallow specific paths in the same way. If you have no rule for AgencyFixResearchBot, the crawler follows your rules for all user-agents (User-agent: *).

Option 2 — ask us. Send a request through our Contact page with the subject “Research crawler opt-out” and the website domain. We add the domain to a permanent exclusion list that the crawler checks before any request, and it will be excluded from all future AgencyFix research crawling.

Questions

Questions about the crawler or our research? Reach us through our Contact page — AgencyFix / Brave Digital Ventures LLC, Santa Clarita, California.