An AI interviewer is only as good as the questions you hand it. The good news: the properties that make a question scorable by AI are the same ones that make it a good question, period — it targets one criterion, invites specific evidence, and leaves room to probe. The machine just makes bad questions fail faster and more visibly.
Here's the method we recommend for building an async AI interview set — one piece of running an AI-first hiring stack — and the patterns that quietly ruin scoring.
Start from the rubric, not from a question bank
Every question exists to generate evidence for a criterion in your hiring rubric. So the sequence is: rubric first, then for each criterion, one or two questions designed to surface its evidence line. A question that doesn't map to a criterion produces answers nobody can score — the AI will dutifully evaluate it against the rubric and return weak signal, because you asked for something the rubric doesn't measure.
A practical budget for a 20–30 minute interview: one strong question per criterion for a 5–8 criterion rubric, weighted toward the Must-Haves (give your most important criteria the questions with the most room to go deep — in Rubrily, per-question priority tells the AI exactly that).
The anatomy of a scorable question
Ask for the past, not the hypothetical. "Describe a time you shipped a system under a hard deadline — what did you cut and why?" beats "How would you handle a tight deadline?" Past-tense questions force specifics: names, tradeoffs, outcomes. Hypotheticals invite well-rehearsed theory that scores identically for candidates who have and haven't done the thing.
One criterion per question. "Tell me about a technically complex project and how you communicated it to stakeholders" splits the answer's evidence between two criteria, shortchanging both. Ask two questions.
Make the evidence request explicit. End with the specific thing you want: "...and walk me through the tradeoff you personally made." The phrase tells strong candidates exactly how to show their strength — and its absence in an answer is itself signal.
Leave follow-up room. The AI cross-questions adaptively, and its follow-ups are where padded answers collapse and real experience shines. Questions with a single factual answer ("What does REST stand for?") leave nothing to probe; questions about decisions and tradeoffs give the follow-up somewhere to go.
What makes a question hard for AI to score?
The same things that make it hard for humans to score, amplified: vague prompts ("tell me about yourself") that produce unanchored answers; double-barreled questions that split evidence across criteria; hypotheticals that reward eloquence over experience; and trivia with binary answers that generate no reasoning to evaluate. If two interviewers would disagree about what a good answer looks like, the question needs rewriting before any interview — human or AI.
Patterns to avoid
The warm-up that isn't: "walk me through your CV" burns minutes on information the CV evaluation already scored. The culture-fit proxy: "would you say you're a team player?" — every candidate says yes; ask for the conflict story instead. The trick question: gotchas measure puzzle exposure, not the criterion, and they poison candidate experience. The kitchen sink: a five-part question produces four forgotten parts; the AI scores what was answered, and what was answered is a fraction of what you needed.
A worked example
Criterion: Problem-solving under ambiguity (Very Important). Evidence line: structures an unfamiliar problem aloud; asks clarifying questions before answering.
Weak question: "Are you comfortable with ambiguity?" (Yes. Everyone is comfortable with ambiguity in interviews.)
Strong question: "Tell me about a time you were handed a problem nobody could fully define. How did you figure out what to actually build, and what did you get wrong at first?" — past-tense, single-criterion, explicit evidence request ("what did you get wrong" invites honesty and self-awareness), and endless follow-up room ("what would you clarify first today?").
Score the answer 0–10 against the evidence line, justify it in a sentence, and you have a data point that survives comparison across two hundred candidates. That's the whole game — and it's exactly what the AI does with every answer, justification included.
Calibrate after the first cohort
After the first ten submissions, read the transcripts against the scores. Questions where everyone scores 5–6 aren't discriminating — sharpen the evidence request. Questions where answers wander off-criterion are mis-mapped — move or rewrite them. Interview sets are reusable assets in Rubrily, so a calibration pass improves every future role that borrows the set.
FAQ
How many questions should an AI interview have? Enough to cover the rubric once with depth: typically 6–10 scripted questions for a 20–30 minute interview, with adaptive follow-ups filling the gaps. More questions with less depth each is a worse trade — follow-ups on strong questions produce better evidence than breadth ever does.
Should I use the same questions for every candidate? Yes — that's what makes scores comparable, and it's the foundation of structured interviewing. The AI's follow-ups personalize the path through the questions without changing the questions themselves, which is the right balance of consistency and depth.
Can the AI ask questions about the candidate's CV? Yes — you can budget AI-generated CV questions per interview set, and the AI grounds them in the candidate's actual document. They're useful for probing claims ("you list X — walk me through your role in it") while the scripted core stays identical across candidates.
Write the questions once, weight what matters, and let the AI run every interview against them — with a scored, justified answer for every candidate. Start free →