mirror of
https://github.com/jcreek/AutomatedAssessmentFeedbackAgent.git
synced 2026-07-12 18:43:49 +00:00
Josh Creek
372 lines
14 KiB
Markdown
372 lines
14 KiB
Markdown
# AI-Powered Automated Assessment and Feedback Agent
|
||
|
||
|
||
|
||
> An intelligent, agentic AI system built by a teacher-turned-engineer to deliver instant grading, personalized feedback, and real-time transparent reasoning - saving teachers hours and improving student outcomes.
|
||
|
||
---
|
||
|
||
## 🏆 Hackathon Info
|
||
|
||
Built for the [Microsoft Hack Together: AI Agents Hackathon](https://microsoft.github.io/AI_Agents_Hackathon/) (April 8–30, 2025).
|
||
Status: Hackathon prototype/MVP.
|
||
See the [Official Rules](https://microsoft.github.io/AI_Agents_Hackathon/rules/).
|
||
|
||
Created using:
|
||
|
||
- JavaScript/TypeScript
|
||
- Azure AI Agent Service
|
||
|
||
---
|
||
|
||
## 📽️ Demonstration and Hackathon Experience Video
|
||
|
||
[Watch it on YouTube](https://www.youtube.com/watch?v=FAi9-s9WP84)
|
||
|
||
---
|
||
|
||
## ✨ Live Demos / Key Functionality
|
||
|
||
**Key Features:**
|
||
|
||
- Instant, AI-powered grading and feedback
|
||
- Human-in-the-Loop escalation for ambiguous cases
|
||
- Real-time agentic tool use and transparent reasoning
|
||
- Assessment history stored locally for privacy
|
||
- Fully tested and CI-backed for reliability
|
||
|
||
Below are GIF demonstrations of the Automated Assessment Feedback Agent in action, showcasing core features and real-world scenarios:
|
||
|
||
### 1. Human-in-the-Loop (HITL): Wrong Assignment Provided
|
||
|
||
_The agent detects when a student submits the wrong assignment and facilitates teacher intervention._
|
||
|
||
|
||
|
||
### 2. HITL: English as an Additional Language (EAL) Student
|
||
|
||
_Demonstrates tailored feedback and HITL support for a student with EAL needs._
|
||
|
||
|
||
|
||
### 3. Assessing a Wordy Essay-Type Task
|
||
|
||
_Shows the agent analyzing and providing feedback on a complex, open-ended essay response._
|
||
|
||
|
||
|
||
### 4. Assessing a Computer Science Coding Task
|
||
|
||
_The agent evaluates a programming submission, running code tests and generating feedback._
|
||
|
||
|
||
|
||
---
|
||
|
||
## 📁 Table of Contents
|
||
|
||
1. [Project Overview](#1-project-overview)
|
||
2. [Team Information](#2-team-information)
|
||
3. [What Makes This Unique](#3-what-makes-this-unique)
|
||
4. [Who Is This For?](#4-who-is-this-for)
|
||
5. [How It Works](#5-how-it-works)
|
||
6. [Technical Details](#6-technical-details)
|
||
7. [Automated Testing & Quality Assurance](#7-automated-testing--quality-assurance)
|
||
8. [Human-in-the-Loop Innovation](#8-human-in-the-loop-innovation)
|
||
9. [Accessibility and Responsible AI](#9-accessibility-and-responsible-ai)
|
||
10. [Security & Privacy](#10-security--privacy)
|
||
11. [Setup, Usage, and Testing](#11-setup-usage-and-testing)
|
||
12. [License](#12-license)
|
||
|
||
---
|
||
|
||
## 1. Project Overview
|
||
|
||
### Elevator Pitch
|
||
|
||
An agentic AI system for teachers that transforms grading and feedback.
|
||
Transparent, real-time tool use and reasoning builds trust—giving educators instant, individualized assessments and actionable feedback for students that teachers can understand, edit, and trust.
|
||
|
||
---
|
||
|
||
## 2. Team Information
|
||
|
||
Created and built entirely by me, **Josh Creek** - an ex-teacher and current software engineer.
|
||
Mission: build AI tools that **genuinely empower educators**, **save time**, and **improve student outcomes**.
|
||
|
||
---
|
||
|
||
## 3. What Makes This Unique
|
||
|
||
_Why is this different from other AI grading tools?_
|
||
|
||
- **Real-Time Transparency:** Teachers see every reasoning step and tool the agent chooses—live.
|
||
- **Personalized Feedback:** Detailed, contextual feedback plus strengths, improvements, follow-on activities, and suggestions.
|
||
- **Human-in-the-Loop Escalation:** The agent escalates edge cases to the teacher for review (never guessing blindly).
|
||
- **Resilient History Management:** All assessments stored locally for review, even during demos (no student data stored anywhere but the teacher's browser).
|
||
- **Accessibility First:** Full screen reader support, keyboard navigation, color contrast compliance.
|
||
- **Robust Error Handling:** Friendly, actionable error messages for upload, AI, or network issues.
|
||
- **Real-Time Agentic Progress Visualization:** Not just a loading spinner—teachers see, in real time, which tools and reasoning steps the agent chooses as it grades. This transparency builds trust and helps educators understand _how_ AI arrives at its conclusions.
|
||
- **Built by a Teacher, for Teachers:** Practical, realistic, classroom-aware design.
|
||
|
||
---
|
||
|
||
## 4. Who Is This For?
|
||
|
||
### Audience
|
||
|
||
- **Teachers** seeking to save time, improve feedback quality, and maintain control.
|
||
- **Schools/Educational Institutions** aiming to modernize and streamline assessment workflows.
|
||
|
||
### Educational Impact
|
||
|
||
- Reduces hours spent grading and marking.
|
||
- Improves quality and consistency of student feedback.
|
||
- Allows teachers more time to focus on direct student interaction and lesson planning.
|
||
|
||
---
|
||
|
||
## 5. How It Works
|
||
|
||
### Workflow At A Glance
|
||
|
||
1. Upload assignment instructions and student response (text for demo; file support planned).
|
||
2. Agent generates real-time, transparent reasoning and instant grading.
|
||
3. If the agent thinks it needs it, it can optionally ask the teacher for suppport (Human-In-The-Loop).
|
||
4. Teacher reviews, edits, or clears assessments from local history.
|
||
|
||
### Demo it!
|
||
|
||
> Upload a meaningful text (or minimal/off-topic text to trigger Human-in-the-Loop escalation).
|
||
> Watch live tool use and reasoning.
|
||
> Review or clear past assessments directly in history.
|
||
|
||
### System Flow Diagram (with HITL)
|
||
|
||
This shows how a user interacts with the system and how requests are processed, including HITL.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Teacher["Teacher (User)"]
|
||
Upload["Upload Page"]
|
||
Results["Results/History Page"]
|
||
AgenticProgress["AgenticProgress Component"]
|
||
LocalStorage["localStorage (Browser)"]
|
||
APIGrade["API: /api/grade"]
|
||
APIHITL["API: /api/hitl-review"]
|
||
EventStream["PartyKit WebSocket (Real-time Agent Progress)"]
|
||
OpenAI["Azure OpenAI (NLP, Grading, Feedback)"]
|
||
|
||
%% Standard Grading Flow
|
||
Teacher -->|Uploads assignment & student work| Upload
|
||
Upload -->|Calls| APIGrade
|
||
APIGrade -->|Sends to| OpenAI
|
||
APIGrade -->|Streams progress| EventStream
|
||
EventStream -->|Updates| AgenticProgress
|
||
APIGrade -->|Returns feedback| Results
|
||
Results -->|Saves| LocalStorage
|
||
Results -->|Displays| Teacher
|
||
|
||
%% HITL Escalation (Human-in-the-Loop Path)
|
||
APIGrade -- Escalates if unclear/minimal --> TeacherReview["Teacher Review (HITL Prompt)"]
|
||
TeacherReview -->|Submits review| APIHITL
|
||
APIHITL -->|Injects teacher feedback| OpenAI
|
||
APIHITL -->|Returns final feedback| Results
|
||
```
|
||
|
||
### Architecture Diagram
|
||
|
||
This shows the big-picture technical structure and how my chosen technologies fit together.
|
||
|
||
The architecture includes:
|
||
|
||
- Frontend: SvelteKit app (browser)
|
||
- Connects to PartyKit (for real-time events)
|
||
- Calls API endpoints (/api/grade, /api/hitl-review)
|
||
- Saves assessment history in browser localStorage
|
||
- Backend: SvelteKit API routes (serverless functions, e.g., Netlify)
|
||
- /api/grade: Handles grading, talks to Azure OpenAI
|
||
- /api/hitl-review: Handles teacher review, resumes AI grading
|
||
- Third-Party Services:
|
||
- Azure OpenAI (AI grading, feedback)
|
||
- Azure AI Agent Service (agentic workflow orchestration, tool use, LLM integration)
|
||
- PartyKit (WebSocket server for real-time updates)
|
||
- Data Storage: Only in browser (localStorage), no backend DB
|
||
|
||
System architecture diagram showing component hosting, communication flows (HTTP, WebSocket), and external AI service integrations:
|
||
|
||
|
||
|
||
---
|
||
|
||
## 6. Technical Details
|
||
|
||
- **Frontend:** [SvelteKit](https://kit.svelte.dev/) + [TypeScript](https://www.typescriptlang.org/)
|
||
- **Real-time Events:** [PartyKit](https://partykit.io/)
|
||
- **AI & Agent Orchestration:** [Azure AI Agent Service](https://azure.microsoft.com/en-us/products/ai-services/ai-agent-service) (agentic workflows, tool use) + [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) (LLMs)
|
||
|
||
---
|
||
|
||
## 7. Automated Testing & Quality Assurance
|
||
|
||
This project is built with a strong focus on reliability and correctness:
|
||
|
||
- **Comprehensive BDD Tests:** All core features are covered by automated, scenario-based tests using [Cucumber](https://cucumber.io/) and [Playwright](https://playwright.dev/).
|
||
- **Continuous Integration:** Every commit to the `main` branch triggers a full test suite via GitHub Actions ([see badge above](#ai-powered-automated-assessment-and-feedback-agent)), ensuring nothing breaks unexpectedly.
|
||
- **Real-World Scenarios:** Tests simulate real teacher/student interactions.
|
||
- **Easy to Run:**
|
||
```bash
|
||
pnpm run test:bdd
|
||
```
|
||
Or view the [test scripts](./tests/bdd).
|
||
|
||
**Result:** You can trust that this agent works—every time.
|
||
|
||
---
|
||
|
||
## 8. Human-in-the-Loop Innovation
|
||
|
||
### How It Works
|
||
|
||
- If a student submission is minimal/ambiguous, the agent returns `HUMAN_REVIEW_REQUIRED` and explains why.
|
||
- Teacher intervenes, providing direct feedback.
|
||
- Agent resumes, using the human input to complete grading and feedback generation.
|
||
|
||
### Why It Matters
|
||
|
||
- **Transparency:** Teachers always see _why_ the agent requests help, with clear reasoning.
|
||
- **Control:** Teachers remain in the loop for edge cases, ensuring fairness and pedagogical soundness.
|
||
- **Innovation:** This collaborative workflow demonstrates how agentic AI can augment, not replace, expert educators—addressing a key hackathon challenge.
|
||
|
||
---
|
||
|
||
## 9. Accessibility and Responsible AI
|
||
|
||
### Accessibility
|
||
|
||
- Screen reader and keyboard friendly.
|
||
- Color contrast meets WCAG AA standards.
|
||
- Accessible real-time agent progress updates.
|
||
- Tested with browser accessibility tools.
|
||
|
||
### Responsible AI
|
||
|
||
I am committed to responsible and ethical use of AI in education. This project:
|
||
|
||
- Uses Azure OpenAI and Cognitive Services, which comply with Microsoft's responsible AI principles.
|
||
- Does not retain or share student data beyond local processing in the browser (history is stored in localStorage only).
|
||
- Clearly communicates to users when they are interacting with AI-generated feedback.
|
||
- Is designed to minimize bias by providing transparent, explainable feedback and allowing teachers to review/edit results.
|
||
- Does not use student data for model training or any secondary purpose.
|
||
|
||
---
|
||
|
||
## 10. Security & Privacy
|
||
|
||
- **No Persistent Storage:** All assessment history and student data are stored only in the user’s browser (localStorage). Nothing is ever uploaded to or retained on any backend server or database.
|
||
- **Ephemeral AI Agent Threads:** AI agent threads and any temporary data created during assessment are deleted immediately after grading is complete.
|
||
- **No Data Sharing:** Student data is never shared with third parties, used for model training, or processed for any purpose other than providing instant feedback.
|
||
- **User Control:** Users can review and clear their assessment history at any time from within the app.
|
||
- **Secure Communication:** All interactions with the Azure AI Agent Service are performed over secure, encrypted channels (HTTPS/WSS).
|
||
- **Privacy by Design:** The system is architected to maximize privacy and minimize data exposure at every step.
|
||
|
||
---
|
||
|
||
## 11. Setup, Usage, and Testing
|
||
|
||
### Prerequisites
|
||
|
||
- [Node.js](https://nodejs.org/) (v23 or newer recommended)
|
||
- [pnpm](https://pnpm.io/installation)
|
||
|
||
### Running Locally
|
||
|
||
Once you've installed dependencies with `pnpm install`, start a development server:
|
||
|
||
```bash
|
||
pnpm run dev
|
||
# or start the server and open the app in a new browser tab
|
||
pnpm run dev -- --open
|
||
```
|
||
|
||
#### Building
|
||
|
||
To create a production version:
|
||
|
||
```bash
|
||
pnpm run build
|
||
```
|
||
|
||
You can preview the production build with `npm run preview`.
|
||
|
||
### PartyKit Setup for Real-Time Events
|
||
|
||
This project includes end-to-end BDD (Behavior-Driven Development) tests using [Cucumber.js](https://github.com/cucumber/cucumber-js) and [Playwright](https://playwright.dev/).
|
||
|
||
#### Prerequisites
|
||
|
||
- All application dependencies installed (see above)
|
||
- [Node.js](https://nodejs.org/) and [pnpm](https://pnpm.io/)
|
||
|
||
#### Install Playwright Browsers
|
||
|
||
If you haven't already, install Playwright's required browsers:
|
||
|
||
```bash
|
||
pnpm exec playwright install
|
||
```
|
||
|
||
#### Running the Tests
|
||
|
||
1. Start the SvelteKit dev server:
|
||
```bash
|
||
pnpm run dev
|
||
```
|
||
(Or use `pnpm run bdd:full` to auto-start the server and run tests.)
|
||
2. In a separate terminal, run the BDD tests:
|
||
```bash
|
||
pnpm run test:bdd
|
||
```
|
||
This will execute all feature files in `tests/bdd/features/` using step definitions in `tests/bdd/steps/`.
|
||
|
||
#### Test Output & Screenshots
|
||
|
||
- Test results will be shown in the terminal.
|
||
- On failure, a screenshot will be saved to the `screenshots/` directory in the project root (see `tests/bdd/support/hooks.ts`).
|
||
- Screenshot filenames are based on the scenario name.
|
||
|
||
#### Customizing/Debugging
|
||
|
||
- You can run a specific feature file:
|
||
```bash
|
||
pnpm run test:bdd -- tests/bdd/features/assessment_submission.feature
|
||
```
|
||
- For more verbose output, add `--format progress` or `--format summary`.
|
||
|
||
#### Project Scripts
|
||
|
||
- `pnpm run test:bdd` – Run all BDD tests
|
||
- `pnpm run bdd:full` – Start dev server and run all BDD tests (requires [start-server-and-test](https://github.com/jsdom/start-server-and-test))
|
||
|
||
For more information, see the `package.json` scripts section.
|
||
|
||
---
|
||
|
||
## 12. License
|
||
|
||
Licensed under the Business Source License 1.1.
|
||
See LICENSE file for details.
|
||
|
||
---
|
||
|
||
## 13. Future Enhancements
|
||
|
||
- Integration with major Learning Management Systems (LMS) for streamlined workflow.
|
||
- Expansion of supported assignment types and subjects.
|
||
- Development of analytics dashboards for deeper insights into class performance.
|
||
- Image upload for handwritten work, especially for teachers of younger students.
|
||
|
||
---
|
||
|
||
**Built with love for teachers.**
|