mirror of
https://github.com/jcreek/AutomatedAssessmentFeedbackAgent.git
synced 2026-07-12 18:43:49 +00:00
feat(*): Be stricter regarding HITL
This commit is contained in:
+19
-5
@@ -91,6 +91,7 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
|
||||
): string {
|
||||
const safeSubmission = sanitizeForPrompt(submission);
|
||||
const safeTask = sanitizeForPrompt(task);
|
||||
|
||||
const steps = `Follow these steps:
|
||||
1. Grade the work. Use whatever grading scheme is present in the rubric. If none is present, grade with a letter (A+ is best, E- is worst), using clear, objective criteria. Be aware that the rubric may just give a maximum number of marks, in which case you should mark out of that number, for example "(20)" would mean it's out of 20 marks. DO NOT mention tool errors.
|
||||
2. Identify specific strengths, referencing the success criteria.
|
||||
@@ -102,6 +103,7 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
|
||||
5. Write a reflection question for the student to encourage metacognition.
|
||||
6. Suggest to the teacher one way to support this student in the next lesson.
|
||||
7. Show your reasoning step by step (chain-of-thought).`;
|
||||
|
||||
const responseFormat = `RESPONSE FORMAT (respond with a single JSON object, no extra text):
|
||||
|
||||
{
|
||||
@@ -113,7 +115,10 @@ export function buildGradingPromptWithSupportForHumanInTheLoop(
|
||||
"teacherSuggestion": "<text>",
|
||||
"spellingAndGrammar": "<text>",
|
||||
"reasoning": "<step-by-step explanation>"
|
||||
}`;
|
||||
}
|
||||
|
||||
Respond ONLY with the JSON object, with no preamble or explanation.`;
|
||||
|
||||
const extraSections = `
|
||||
CONFIDENCE CHECK:
|
||||
|
||||
@@ -131,11 +136,19 @@ You MUST NOT request human review just because:
|
||||
- The student's writing contains normal age-appropriate mistakes, informal language, or minor issues.
|
||||
|
||||
IMPORTANT:
|
||||
You must not claim that tools are unavailable or broken unless a tool call explicitly fails with an error. If a tool call fails, you must describe the failure clearly, then continue grading using your best judgment. You must never escalate to human review just because a tool call failed. Failed, missing, or partial tool results are not valid reasons to avoid grading.
|
||||
If a tool call fails, you must continue grading using your best professional judgment. Tool failures are normal and are not a valid reason to request human review.
|
||||
|
||||
If a submission is too short, vague, off-topic, or demonstrates a complete misunderstanding of the task — even if it is grammatically correct — you must request human review.
|
||||
You must evaluate the submission manually, even if rubric analysis, spelling, or grammar checking tools are unavailable.
|
||||
|
||||
Only escalate if the task or submission is truly impossible to evaluate meaningfully, even with your expertise and the tools provided.
|
||||
You MUST NOT escalate to human review just because of uncertainty about:
|
||||
- Spelling
|
||||
- Grammar
|
||||
- Alignment with specific rubric details
|
||||
- Missing tool outputs
|
||||
|
||||
If the submission is on-topic, detailed, intelligible, and attempts to respond meaningfully to the task, you must grade it manually.
|
||||
|
||||
Only escalate if the submission itself is incoherent, nonsensical, off-topic, or so vague that no meaningful evaluation is possible — even using your best human-like judgment.
|
||||
|
||||
Otherwise, proceed confidently with grading using the rubric, your judgment, and available tools.
|
||||
|
||||
@@ -143,7 +156,7 @@ If and only if human review is truly required, respond with:
|
||||
|
||||
{
|
||||
"grade": "HUMAN_REVIEW_REQUIRED",
|
||||
"reasoning": "<Explain why. Do not mention tool errors unless an actual tool failure occurred.>"
|
||||
"reasoning": "<Explain why. Do not mention tool errors unless an actual tool failure occurred and it prevents understanding of the submission.>"
|
||||
}
|
||||
|
||||
EXAMPLES:
|
||||
@@ -169,6 +182,7 @@ EXAMPLES:
|
||||
- Reasoning: The task and submission are both clear and evaluatable without a rubric.
|
||||
|
||||
Respond ONLY with the JSON object, with no preamble or explanation.`;
|
||||
|
||||
return buildBaseGradingPrompt({
|
||||
task: safeTask,
|
||||
submission: safeSubmission,
|
||||
|
||||
Reference in New Issue
Block a user