Fair AI scoring: no more “the answer is correct, but it doesn’t sound right”
At LearnSpace, we found an AI-first bug: the model was under-scoring correct but unconventional answers. We added calibration, human-in-the-loop review, and step-by-step error analysis.
In LearnSpace, we ran into an unpleasant AI-first-era bug: the model would lower scores when a student answered correctly but in an unconventional way. In other words, the meaning was fine, but the wording didn’t match the template — and the score dropped. Not exactly fair play.
What we did: we added scoring calibration and human-in-the-loop review. Now edge cases go to a teacher for review instead of being decided by black box magic. On top of that, the AI tutor no longer just throws out a dry “incorrect”: it gives a step-by-step breakdown of mistakes and takes the context of previous attempts into account — in other words, it explains why the answer is off.
Against the backdrop of the hype around AI safety and fairness, this is no longer a nice-to-have — it’s basic infrastructure. And yes, as in real life, sometimes you need not just a smart algorithm, but also a colleague who says, “wait, the student is actually right here.”