Building ScholarOS: A Local-First Research Assistant, In Progress

This one's a builder's log, not a case study, because the thing I'm describing doesn't exist yet as a finished product. I want to write about it anyway, because the way I'm approaching it is the actual lesson, not the tool itself.

I have a few thousand files' worth of MBA coursework and notes sitting in a private vault. Useful, searchable in theory, mostly dead weight in practice because finding the right note at the right moment takes longer than just re-deriving the answer. The obvious fix is a private AI research assistant that can answer questions against that corpus with citations. So I started scoping one.

Here's the part I want to flag, because I think it's the part people skip: before writing a line of product code, I wrote a spec, and before building any interface, I built a fixed set of evaluation questions, twenty-five of them, with known correct answers, that the system has to actually answer well before it earns a UI. Not "does it feel smart when I poke at it." Does it get a fixed, checkable set of questions right, consistently, with real citations back to source material.

That ordering matters more than it sounds like it should. It's tempting to build the chat interface first, because that's the fun, visible part. But a slick interface over an assistant that hallucinates half its citations is worse than no product at all, because it's actively unreliable while looking finished. Evaluation first means you find out the calibration problem while it's still cheap to fix.

Where it stands now: source manifest built, eval questions written, no interface yet. That's deliberately the order. I'll write the next post in this thread when there's something worth showing, not before.