About LinkoriaBot
If LinkoriaBot appeared in your server logs, this page explains what it is and what to do about it.
How to identify it
Our requests carry this user agent, both when we crawl the pages of a site and when we check where a link on it leads:
LinkoriaBot/1.0 (+https://linkoria.io/crawler)The URL in the user agent points to this page, so anyone reading your logs can find it.
Every LinkoriaBot request is also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421), so your server or CDN can verify that it really comes from us. Our public keys are published here:
https://app.linkoria.io/.well-known/http-message-signatures-directoryA small share of requests carry a browser user agent instead β see βBrowser fallbackβ below.
What it does
- Reads robots.txt before anything else, and honours it.
- Reads sitemaps to find which pages to check.
- Requests those pages and extracts the links on them.
- Follows each outbound link through its redirects to find where it ends up. Before each request to the link's destination, it reads that site's robots.txt and honours it, including crawl-delay.
Why it is on your site
Someone using Linkoria asked us to check the links on a site. Either that site is yours, or a link on another site points to yours β in which case we checked your robots.txt and requested a single URL to see whether it responds, and nothing more.
Browser fallback
Some sites refuse every automated client, including verified ones. If such a site refuses LinkoriaBot, the rest of that scan's checks on the site, and the ones it refused, are sent with a standard browser profile, so that the person checking can still tell whether their links work. We never do this on sites protected by Cloudflare or Akamai, where a refusal stops all requests to the site for that scan. A robots.txt that blocks LinkoriaBot from your whole site, as shown below, rules it out as well. The switch lasts for one scan only.
How it behaves
- Requests are rate limited and spread out over time.
- It obeys robots.txt, including crawl-delay.
- It does not submit forms, log in, or execute JavaScript.
- It does not collect personal data from your pages.
How to block it
Add this to your robots.txt and LinkoriaBot will stop requesting your pages from the next scan on, both to crawl your site and to check links that point to it:
User-agent: LinkoriaBot
Disallow: /This also rules out the browser fallback. To have crawling stopped immediately, or your domain excluded in some other way, email us.
Contact
For anything about the crawler, including takedown or exclusion requests: abuse@linkoria.io