Skip to content
a college counselor for all four years

LoreBot

LoreBot is the crawler Lore uses to check college figures against colleges’ own websites. If you found it in your logs, this page says what it reads, how often, and how to reach us or turn it away.

Who we are

Lore is a college counselor for families, for all four years of high school. Next to every cost, scholarship and deadline figure we show, we link the college’s own page and the date we last checked it. LoreBot is how that check happens. Lore is run by its founder, as an early-stage project that is not incorporated yet. Contact: hello@loreeducation.com.

What it reads

  • Public pages that a person on our team has opened and confirmed as a college’s own page for one of four things: the cost of attendance, merit scholarships, application deadlines, and supplemental essay prompts. And each site’s robots.txt.
  • When we add a college, a few more pages, to find candidates for that list: its home page, its admissions and financial-aid pages, and scholarship pages we found by search. A person on our team still opens and confirms each candidate before LoreBot reads it again.
  • Nothing else. It doesn’t crawl your site, sign in, or submit forms. It follows a redirect only to another page on the college’s own site.
  • Web pages only, at most 3 MB each. It stops reading a page after 45 seconds and tries again at its next check.

How often

  • Each page about once a week from August through January, when deadlines and scholarship terms change, and about once a month the rest of the year. That’s a handful of pages per college.
  • Each run waits at least 1.5 seconds between requests to the same site.

robots.txt

LoreBot reads your robots.txt before anything else on your site and follows it. It answers to the token LoreBot, and otherwise to the rules for *. If robots.txt can’t be read (a server error, no answer, or a 401, 403 or 429), LoreBot reads nothing on the site that run.

To turn it away from your whole site:

User-agent: LoreBot
Disallow: /

Its full user-agent string:

LoreBot/1.0 (+https://loreeducation.com/bot; college data verification)

What happens to what it reads

  • When a page changes, an AI model proposes the new figures, quoting your page word for word, and our code checks every figure against the page itself. A person on our team reviews a college’s figures before any of them is shown to families as verified.
  • We keep a text copy of each page we read, as our record of what it said and when. We don’t sell or republish your pages: families see the figures with a link to your page.
  • If LoreBot is turned away, we can’t re-check your figures, so we stop showing them as verified and point families to your page instead.

Reach us

Email hello@loreeducation.com with your site and what you’d like: fewer visits, different pages, or none at all. If a figure we show about your college doesn’t match your page, tell us the same way. Our terms say what our figures are and aren’t.