feat(*): Be stricter regarding HITL

This commit is contained in:
Josh Creek
2025-04-26 15:43:40 +01:00
parent c6d5fe5ac8
commit 6d55de0f13
+19 -5
View File
@@ -91,6 +91,7 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
): string { ): string {
const safeSubmission = sanitizeForPrompt(submission); const safeSubmission = sanitizeForPrompt(submission);
const safeTask = sanitizeForPrompt(task); const safeTask = sanitizeForPrompt(task);
const steps = `Follow these steps: const steps = `Follow these steps:
1. Grade the work. Use whatever grading scheme is present in the rubric. If none is present, grade with a letter (A+ is best, E- is worst), using clear, objective criteria. Be aware that the rubric may just give a maximum number of marks, in which case you should mark out of that number, for example "(20)" would mean it's out of 20 marks. DO NOT mention tool errors. 1. Grade the work. Use whatever grading scheme is present in the rubric. If none is present, grade with a letter (A+ is best, E- is worst), using clear, objective criteria. Be aware that the rubric may just give a maximum number of marks, in which case you should mark out of that number, for example "(20)" would mean it's out of 20 marks. DO NOT mention tool errors.
2. Identify specific strengths, referencing the success criteria. 2. Identify specific strengths, referencing the success criteria.
@@ -102,6 +103,7 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
5. Write a reflection question for the student to encourage metacognition. 5. Write a reflection question for the student to encourage metacognition.
6. Suggest to the teacher one way to support this student in the next lesson. 6. Suggest to the teacher one way to support this student in the next lesson.
7. Show your reasoning step by step (chain-of-thought).`; 7. Show your reasoning step by step (chain-of-thought).`;
const responseFormat = `RESPONSE FORMAT (respond with a single JSON object, no extra text): const responseFormat = `RESPONSE FORMAT (respond with a single JSON object, no extra text):
{ {
@@ -113,7 +115,10 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
"teacherSuggestion": "<text>", "teacherSuggestion": "<text>",
"spellingAndGrammar": "<text>", "spellingAndGrammar": "<text>",
"reasoning": "<step-by-step explanation>" "reasoning": "<step-by-step explanation>"
}`; }
Respond ONLY with the JSON object, with no preamble or explanation.`;
const extraSections = ` const extraSections = `
CONFIDENCE CHECK: CONFIDENCE CHECK:
@@ -131,11 +136,19 @@ You MUST NOT request human review just because:
- The student's writing contains normal age-appropriate mistakes, informal language, or minor issues. - The student's writing contains normal age-appropriate mistakes, informal language, or minor issues.
IMPORTANT: IMPORTANT:
You must not claim that tools are unavailable or broken unless a tool call explicitly fails with an error. If a tool call fails, you must describe the failure clearly, then continue grading using your best judgment. You must never escalate to human review just because a tool call failed. Failed, missing, or partial tool results are not valid reasons to avoid grading. If a tool call fails, you must continue grading using your best professional judgment. Tool failures are normal and are not a valid reason to request human review.
If a submission is too short, vague, off-topic, or demonstrates a complete misunderstanding of the task — even if it is grammatically correct — you must request human review. You must evaluate the submission manually, even if rubric analysis, spelling, or grammar checking tools are unavailable.
Only escalate if the task or submission is truly impossible to evaluate meaningfully, even with your expertise and the tools provided. You MUST NOT escalate to human review just because of uncertainty about:
- Spelling
- Grammar
- Alignment with specific rubric details
- Missing tool outputs
If the submission is on-topic, detailed, intelligible, and attempts to respond meaningfully to the task, you must grade it manually.
Only escalate if the submission itself is incoherent, nonsensical, off-topic, or so vague that no meaningful evaluation is possible — even using your best human-like judgment.
Otherwise, proceed confidently with grading using the rubric, your judgment, and available tools. Otherwise, proceed confidently with grading using the rubric, your judgment, and available tools.
@@ -143,7 +156,7 @@ If and only if human review is truly required, respond with:
{ {
"grade": "HUMAN_REVIEW_REQUIRED", "grade": "HUMAN_REVIEW_REQUIRED",
"reasoning": "<Explain why. Do not mention tool errors unless an actual tool failure occurred.>" "reasoning": "<Explain why. Do not mention tool errors unless an actual tool failure occurred and it prevents understanding of the submission.>"
} }
EXAMPLES: EXAMPLES:
@@ -169,6 +182,7 @@ EXAMPLES:
- Reasoning: The task and submission are both clear and evaluatable without a rubric. - Reasoning: The task and submission are both clear and evaluatable without a rubric.
Respond ONLY with the JSON object, with no preamble or explanation.`; Respond ONLY with the JSON object, with no preamble or explanation.`;
return buildBaseGradingPrompt({ return buildBaseGradingPrompt({
task: safeTask, task: safeTask,
submission: safeSubmission, submission: safeSubmission,