Kortaga's crawler

If you run a website, this page tells you whether we visit it, what we take, and how to stop us.

We do not crawl your site. Kortaga's index is built from pages we publish ourselves and from public web archives that other people have already collected. Your server does not receive requests from us.

Where the pages come from

Our own sites. Everything under luyml.com and kortaga.com, which we run.
Public archives, principally Common Crawl, which crawls the web openly and publishes the result. We read their archive rather than knock on your door a second time for a page somebody already fetched.

This is a deliberate choice, not a stage we have not reached yet. Reading an archive costs your server nothing at all, which is a better answer than crawling politely.

If we ever do crawl

It would identify itself as KortagaBot with a link to this page, read robots.txt first and obey it including Crawl-delay, take one request at a time, and never sign in or fill a form. This page would say so before the first request rather than after. Right now it does not apply, because we do not crawl.

Keeping us out anyway

If you would rather not appear, this works today and would work then:

User-agent: KortagaBot
Disallow: /

Or narrow it to the parts you want left alone:

User-agent: KortagaBot
Disallow: /private/
Crawl-delay: 10

Something wrong?

Write to [email protected] with your domain. If a page of yours is in our index and you want it out, say so and it goes. If you are seeing traffic claiming to be us, tell us, because it is not.