feat(*): Be stricter regarding HITL

This commit is contained in:
Josh Creek
2025-04-26 15:43:40 +01:00
parent c6d5fe5ac8
commit 6d55de0f13
+62 -48
View File
@@ -91,20 +91,22 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
): string { ): string {
const safeSubmission = sanitizeForPrompt(submission); const safeSubmission = sanitizeForPrompt(submission);
const safeTask = sanitizeForPrompt(task); const safeTask = sanitizeForPrompt(task);
const steps = `Follow these steps: const steps = `Follow these steps:
1. Grade the work. Use whatever grading scheme is present in the rubric. If none is present, grade with a letter (A+ is best, E- is worst), using clear, objective criteria. Be aware that the rubric may just give a maximum number of marks, in which case you should mark out of that number, for example "(20)" would mean it's out of 20 marks. DO NOT mention tool errors. 1. Grade the work. Use whatever grading scheme is present in the rubric. If none is present, grade with a letter (A+ is best, E- is worst), using clear, objective criteria. Be aware that the rubric may just give a maximum number of marks, in which case you should mark out of that number, for example "(20)" would mean it's out of 20 marks. DO NOT mention tool errors.
2. Identify specific strengths, referencing the success criteria. 2. Identify specific strengths, referencing the success criteria.
3. Identify misconceptions or areas for improvement, using formative assessment language. 3. Identify misconceptions or areas for improvement, using formative assessment language.
4. Design an individualized activity or exercise for the student to address their misconceptions or extend their learning. This activity should be: 4. Design an individualized activity or exercise for the student to address their misconceptions or extend their learning. This activity should be:
- Appropriately scaffolded for the student's current level. - Appropriately scaffolded for the student's current level.
- Specific and actionable (e.g., a short written task, a practical exercise, or a quiz question). - Specific and actionable (e.g., a short written task, a practical exercise, or a quiz question).
- Aligned with the curriculum and learning objectives. - Aligned with the curriculum and learning objectives.
5. Write a reflection question for the student to encourage metacognition. 5. Write a reflection question for the student to encourage metacognition.
6. Suggest to the teacher one way to support this student in the next lesson. 6. Suggest to the teacher one way to support this student in the next lesson.
7. Show your reasoning step by step (chain-of-thought).`; 7. Show your reasoning step by step (chain-of-thought).`;
const responseFormat = `RESPONSE FORMAT (respond with a single JSON object, no extra text): const responseFormat = `RESPONSE FORMAT (respond with a single JSON object, no extra text):
{ {
"grade": "<number or string, e.g. 8/10, A, B+>", "grade": "<number or string, e.g. 8/10, A, B+>",
"strengths": "<text>", "strengths": "<text>",
"areasForImprovement": "<text>", "areasForImprovement": "<text>",
@@ -113,62 +115,74 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
"teacherSuggestion": "<text>", "teacherSuggestion": "<text>",
"spellingAndGrammar": "<text>", "spellingAndGrammar": "<text>",
"reasoning": "<step-by-step explanation>" "reasoning": "<step-by-step explanation>"
}`; }
Respond ONLY with the JSON object, with no preamble or explanation.`;
const extraSections = ` const extraSections = `
CONFIDENCE CHECK: CONFIDENCE CHECK:
You should request human review if: You should request human review if:
- The submission does not meaningfully address the task or assignment. - The submission does not meaningfully address the task or assignment.
- The task or submission is so unclear that you cannot understand what the student is trying to do. - The task or submission is so unclear that you cannot understand what the student is trying to do.
- The response is off-topic, incoherent, or not related to the task in any reasonable way. - The response is off-topic, incoherent, or not related to the task in any reasonable way.
- The submission is so minimal (e.g., a single word or letter) that it cannot be meaningfully evaluated, even if it is technically related to the task. - The submission is so minimal (e.g., a single word or letter) that it cannot be meaningfully evaluated, even if it is technically related to the task.
- The submission demonstrates a complete misunderstanding of the task (e.g., answering a different question or ignoring the core instruction). - The submission demonstrates a complete misunderstanding of the task (e.g., answering a different question or ignoring the core instruction).
- There is no way to apply the success criteria or grading steps, even with tool assistance. - There is no way to apply the success criteria or grading steps, even with tool assistance.
You MUST NOT request human review just because: You MUST NOT request human review just because:
- A formal rubric is missing. You must assess against the task description directly. - A formal rubric is missing. You must assess against the task description directly.
- External tools (e.g., spell checkers, rubric scorers) are unavailable. You must use your best judgment to assess spelling, grammar, structure, and rubric alignment. - External tools (e.g., spell checkers, rubric scorers) are unavailable. You must use your best judgment to assess spelling, grammar, structure, and rubric alignment.
- The student's writing contains normal age-appropriate mistakes, informal language, or minor issues. - The student's writing contains normal age-appropriate mistakes, informal language, or minor issues.
IMPORTANT: IMPORTANT:
You must not claim that tools are unavailable or broken unless a tool call explicitly fails with an error. If a tool call fails, you must describe the failure clearly, then continue grading using your best judgment. You must never escalate to human review just because a tool call failed. Failed, missing, or partial tool results are not valid reasons to avoid grading. If a tool call fails, you must continue grading using your best professional judgment. Tool failures are normal and are not a valid reason to request human review.
If a submission is too short, vague, off-topic, or demonstrates a complete misunderstanding of the task — even if it is grammatically correct — you must request human review. You must evaluate the submission manually, even if rubric analysis, spelling, or grammar checking tools are unavailable.
Only escalate if the task or submission is truly impossible to evaluate meaningfully, even with your expertise and the tools provided. You MUST NOT escalate to human review just because of uncertainty about:
- Spelling
- Grammar
- Alignment with specific rubric details
- Missing tool outputs
Otherwise, proceed confidently with grading using the rubric, your judgment, and available tools. If the submission is on-topic, detailed, intelligible, and attempts to respond meaningfully to the task, you must grade it manually.
If and only if human review is truly required, respond with: Only escalate if the submission itself is incoherent, nonsensical, off-topic, or so vague that no meaningful evaluation is possible — even using your best human-like judgment.
{ Otherwise, proceed confidently with grading using the rubric, your judgment, and available tools.
If and only if human review is truly required, respond with:
{
"grade": "HUMAN_REVIEW_REQUIRED", "grade": "HUMAN_REVIEW_REQUIRED",
"reasoning": "<Explain why. Do not mention tool errors unless an actual tool failure occurred.>" "reasoning": "<Explain why. Do not mention tool errors unless an actual tool failure occurred and it prevents understanding of the submission.>"
} }
EXAMPLES: EXAMPLES:
✅ Human review IS appropriate: ✅ Human review IS appropriate:
- Task: “Write about your favorite historical figure.” - Task: “Write about your favorite historical figure.”
- Submission: “I like the thing it was long ago and happened and then was okay.” - Submission: “I like the thing it was long ago and happened and then was okay.”
- Reasoning: The submission is incoherent and not meaningfully related to the task. - Reasoning: The submission is incoherent and not meaningfully related to the task.
✅ Human review IS appropriate: ✅ Human review IS appropriate:
- Task: “Name a dog.” - Task: “Name a dog.”
- Submission: “I like the colour red.” - Submission: “I like the colour red.”
- Reasoning: The submission expresses a preference but completely ignores the task objective. It cannot be evaluated meaningfully and reflects a misunderstanding. - Reasoning: The submission expresses a preference but completely ignores the task objective. It cannot be evaluated meaningfully and reflects a misunderstanding.
✅ Human review IS appropriate: ✅ Human review IS appropriate:
- Task: “Name a dog.” - Task: “Name a dog.”
- Submission: “A” - Submission: “A”
- Reasoning: The submission is technically on-topic but so minimal that it cannot be evaluated or scored in a meaningful way. - Reasoning: The submission is technically on-topic but so minimal that it cannot be evaluated or scored in a meaningful way.
❌ Human review is NOT appropriate: ❌ Human review is NOT appropriate:
- Task: “Write a persuasive paragraph about school uniforms.” - Task: “Write a persuasive paragraph about school uniforms.”
- Submission: “I think school uniforms are important because they make everyone equal...” - Submission: “I think school uniforms are important because they make everyone equal...”
- Reasoning: The task and submission are both clear and evaluatable without a rubric. - Reasoning: The task and submission are both clear and evaluatable without a rubric.
Respond ONLY with the JSON object, with no preamble or explanation.`;
Respond ONLY with the JSON object, with no preamble or explanation.`;
return buildBaseGradingPrompt({ return buildBaseGradingPrompt({
task: safeTask, task: safeTask,
submission: safeSubmission, submission: safeSubmission,