Crawler notice
LOGITBot
The collector LOGIT uses to read the public feeds and pages of its sources.
User-Agent
LOGITBot/1.0 (+https://www.synapseent.com/logit/bot)
What it takes
It reads only public RSS, Atom, APIs and public HTML listings. It stores the body of official announcements, government documents, paper abstracts and GitHub release notes. For news articles it stores only the title, feed summary, URL and publication time.
How it asks
It caches robots.txt per domain for 24 hours and stops collecting on Disallow.
It uses If-None-Match and If-Modified-Since to fetch only what changed. Feeds are polled every 45 minutes by default, HTML every 3 hours.
On 403, 429 or 401 it does not work around the block. It follows Retry-After or backs off, and turns itself off after 5 failures in a row.
Downloads are limited to 2 MB/s overall and 2 concurrent requests.
How to stop it
Add a Disallow rule for LOGITBot to your robots.txt and it stops within 24 hours. Takedown requests go through the page below.