jcreek/AutomatedAssessmentFeedbackAgent
1
0
Fork 0
mirror of https://github.com/jcreek/AutomatedAssessmentFeedbackAgent.git synced 2026-07-12 18:43:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(*): Update readme to reflect latest functionality changes

Browse Source
This commit is contained in:
Josh Creek
2025-04-25 23:35:31 +01:00
parent 8394a24ac1
commit c495ba4bc0
1 changed files with 181 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+181 -187
View File
@@ -2,167 +2,197 @@
![BDD Tests](https://github.com/jcreek/AutomatedAssessmentFeedbackAgent/actions/workflows/ci.yml/badge.svg?branch=main)
> An intelligent, agentic AI system designed to significantly reduce teachers' workloads by providing instant assessment and personalized differentiated feedback and follow-on activities for student assignments.
> An intelligent, agentic AI system built by a teacher-turned-engineer to deliver instant grading, personalized feedback, and real-time transparent reasoning - saving teachers hours and improving student outcomes.
## Who is this for?
This project is designed for:
- **Teachers** who want to save time on grading and provide more consistent, individualized feedback to students.
- **Schools and educational institutions** seeking to improve the quality and efficiency of assessment and feedback workflows.
## ⚡ Workflow At a Glance
- Upload an assignment and student response (file or text)
- Instantly receive detailed, actionable feedback and differentiated individualized follow-on tasks and a grade
- Review, delete, or clear past assessments in the history
- All actions are accessible, error-proof, and demo resilient
---
## 🏆 Hackathon Info
This project was developed for the [Microsoft Hack Together: AI Agents Hackathon](https://microsoft.github.io/AI_Agents_Hackathon/) (April 8–30, 2025).
Built for the [Microsoft Hack Together: AI Agents Hackathon](https://microsoft.github.io/AI_Agents_Hackathon/) (April 8–30, 2025).
Status: Hackathon prototype/MVP.
See the [Official Rules](https://microsoft.github.io/AI_Agents_Hackathon/rules/).
- See the [Official Rules](https://microsoft.github.io/AI_Agents_Hackathon/rules/)
- Status: Hackathon prototype/MVP
---
## 📽️ Demonstration Video
## 🎯 What It Does (Key Features)
- **Automated Grading:** Instantly grades student submissions (text or file) for any assignment/task.
- **Personalized Feedback:** Actionable, contextual feedback including grade, strengths, areas for improvement, individualized activity, and teacher suggestion.
- **Assessment History:** All assessments are saved locally (browser localStorage) for later review and demo resilience.
- **History Management:** Delete individual assessments or clear all history, with confirmation dialogs for safety.
- **Navigation:** Seamless navigation between upload and results/history pages.
[Coming soon: View a full demonstration of the agent in action.]
---
## 📁 Table of Contents
1. [Project Overview](#1-project-overview)
2. [Team Information](#2-team-information)
3. [What Makes This Unique](#3-what-makes-this-unique)
4. [Who Is This For?](#4-who-is-this-for)
5. [How It Works](#5-how-it-works)
6. [Technical Details](#6-technical-details)
7. [Human-in-the-Loop Innovation](#7-human-in-the-loop-innovation)
8. [Accessibility and Responsible AI](#8-accessibility-and-responsible-ai)
9. [Setup, Usage, and Testing](#9-setup-usage-and-testing)
10. [License](#10-license)
---
## 1. Project Overview
### Elevator Pitch
An agentic AI system for teachers that transforms grading and feedback.
Transparent, real-time tool use and reasoning builds trust—giving educators instant, individualized assessments and actionable feedback for students that teachers can understand, edit, and trust.
---
## 2. Team Information
Created and built entirely by me, **Josh Creek** - an ex-teacher and current software engineer.
Mission: build AI tools that **genuinely empower educators**, **save time**, and **improve student outcomes**.
---
## 3. What Makes This Unique
_Why is this different from other AI grading tools?_
- **Real-Time Transparency:** Teachers see every reasoning step and tool the agent chooses—live.
- **Personalized Feedback:** Detailed, contextual feedback plus strengths, improvements, follow-on activities, and suggestions.
- **Human-in-the-Loop Escalation:** The agent escalates edge cases to the teacher for review (never guessing blindly).
- **Resilient History Management:** All assessments stored locally for review, even during demos (no student data stored anywhere but the teacher's browser).
- **Accessibility First:** Full screen reader support, keyboard navigation, color contrast compliance.
- **Robust Error Handling:** Friendly, actionable error messages for upload, AI, or network issues.
- **Loading Spinner:** Visual feedback while grading is in progress.
- **Accessibility:** Screen reader-friendly, keyboard-accessible, and color-contrast aware.
- **Real-Time Agentic Progress Visualization:** Not just a loading spinner—teachers see, in real time, which tools and reasoning steps the agent chooses as it grades. This transparency builds trust and helps educators understand _how_ AI arrives at its conclusions.
- **Built by a Teacher, for Teachers:** Practical, realistic, classroom-aware design.
## ⚙️ How It Works
---
1. **Teacher uploads a student submission** (file or text) and assignment description.
2. **AI (Azure OpenAI)** generates instant, individualized feedback and a grade.
3. **Results and assessment history** are displayed for review, deletion, or clearing.
4. **All data is stored locally** (no backend required for history/demo).
## 4. Who Is This For?
## 🚀 Technical Stack
### Audience
- **TypeScript:** Ensures reliability, maintainability, and scalability.
- **Azure OpenAI:** Provides advanced NLP capabilities for nuanced and accurate assessment.
- **Azure Cognitive Services:** Enhances semantic analysis for precise feedback generation.
- **Teachers** seeking to save time, improve feedback quality, and maintain control.
- **Schools/Educational Institutions** aiming to modernize and streamline assessment workflows.
## 📖 Educational Impact
### Educational Impact
- Reduces hours spent grading and marking.
- Improves quality and consistency of student feedback.
- Allows teachers more time to focus on direct student interaction and lesson planning.
---
## 5. How It Works
### Workflow At A Glance
1. Upload assignment instructions and student response (text for demo; file support planned).
2. Agent generates real-time, transparent reasoning and instant grading.
3. If the agent thinks it needs it, it can optionally ask the teacher for suppport (Human-In-The-Loop).
4. Teacher reviews, edits, or clears assessments from local history.
### Demo it!
> Upload a meaningful text (or minimal/off-topic text to trigger Human-in-the-Loop escalation).
> Watch live tool use and reasoning.
> Review or clear past assessments directly in history.
### Architecture Diagram
```mermaid
flowchart TD
Teacher["Teacher (User)"]
Upload["Upload Page"]
Results["Results/History Page"]
AgenticProgress["AgenticProgress Component"]
LocalStorage["localStorage (Browser)"]
APIGrade["API: /api/grade"]
APIHITL["API: /api/hitl-review"]
EventStream["PartyKit WebSocket (Real-time Agent Progress)"]
OpenAI["Azure OpenAI (NLP, Grading, Feedback)"]
%% Standard Grading Flow
Teacher -->|Uploads assignment & student work| Upload
Upload -->|Calls| APIGrade
APIGrade -->|Sends to| OpenAI
APIGrade -->|Streams progress| EventStream
EventStream -->|Updates| AgenticProgress
APIGrade -->|Returns feedback| Results
Results -->|Saves| LocalStorage
Results -->|Displays| Teacher
%% HITL Escalation (Human-in-the-Loop Path)
APIGrade -- Escalates if unclear/minimal --> TeacherReview["Teacher Review (HITL Prompt)"]
TeacherReview -->|Submits review| APIHITL
APIHITL -->|Injects teacher feedback| OpenAI
APIHITL -->|Returns final feedback| Results
```
Assessment history is **only stored locally** (no external storage).
---
## 6. Technical Details
- **Frontend:** [SvelteKit](https://kit.svelte.dev/) + [TypeScript](https://www.typescriptlang.org/)
- **Real-time Events:** [PartyKit](https://partykit.io/)
- **AI:** [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) + [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/ai-services/cognitive-services)
---
## 7. Human-in-the-Loop Innovation
### How It Works
- If a student submission is minimal/ambiguous, the agent returns `HUMAN_REVIEW_REQUIRED` and explains why.
- Teacher intervenes, providing direct feedback.
- Agent resumes, using the human input to complete grading and feedback generation.
### Why It Matters
- **Transparency:** Teachers always see _why_ the agent requests help, with clear reasoning.
- **Control:** Teachers remain in the loop for edge cases, ensuring fairness and pedagogical soundness.
- **Innovation:** This collaborative workflow demonstrates how agentic AI can augment, not replace, expert educators—addressing a key hackathon challenge.
### Demo
> Upload a blank or nonsense submission to trigger the HITL workflow and see the transparent escalation.
---
## 8. Accessibility and Responsible AI
### Accessibility
- Screen reader and keyboard friendly.
- Color contrast meets WCAG AA standards.
- Accessible real-time agent progress updates.
- Tested with browser accessibility tools.
### Responsible AI
## 🛠️ Responsible AI
I am committed to responsible and ethical use of AI in education. This project:
- Uses Azure OpenAI and Cognitive Services, which comply with Microsoft's responsible AI principles.
- Does not retain or share student data beyond local processing in the browser (history is stored in localStorage only).
- Clearly communicates to users when they are interacting with AI-generated feedback.
- Is designed to minimize bias by providing transparent, explainable feedback and allowing teachers to review/edit results.
- Does not use student data for model training or any secondary purpose.
## ♿ Accessibility
Accessibility is a core priority:
- The interface is screen reader-friendly, with proper semantic HTML and ARIA labels.
- All features are keyboard accessible (tab navigation, focus indicators).
- Color contrast meets WCAG AA standards for readability.
- Error messages and progress indicators are accessible to assistive technologies.
- The site has been tested with browser accessibility tools and screen readers.
---
## 🗺️ Architecture Diagram
```mermaid
flowchart TD
%% User
Teacher["Teacher (User)"]
%% Frontend
Upload["Upload Page"]
Results["Results/History Page"]
AgenticProgress["AgenticProgress Component"]
LocalStorage["localStorage (Browser)"]
%% API
APIEndpoint["API: /api/grade (Netlify serverless function)"]
APINote["API endpoints are Netlify serverless functions (SvelteKit endpoints)"]
APIEndpoint -.-> APINote
%% PartyKit
EventStream["PartyKit WebSocket (Real-time Agent Progress)"]
PartyKitNote["PartyKit provides WebSocket-based real-time updates on agent progress/tools."]
EventStream -.-> PartyKitNote
%% Azure
OpenAI["Azure OpenAI (NLP, Grading, Feedback)"]
%% Data Flow
Teacher -->|Uploads assignment & student work| Upload
Upload -->|Calls| APIEndpoint
APIEndpoint -->|Sends data & assignment description| OpenAI
APIEndpoint -->|Streams grading progress| EventStream
EventStream -->|Updates progress| AgenticProgress
APIEndpoint -->|Returns feedback & grade| Results
Results -->|Planned: Teacher reviews/edits feedback| Results
Results -->|Saves assessment| LocalStorage
Results -->|Displays feedback, history| Teacher
PrivacyNote["Assessment history is stored only in the user’s browser (localStorage), not sent to any backend."]
LocalStorage --> PrivacyNote
class Teacher user;
class Upload,Results,AgenticProgress,LocalStorage frontend;
class APIEndpoint api;
class EventStream partykit;
class OpenAI azure;
```
## 👥 Teacher Workflow Example
Here's an example of how a teacher might use the Automated Assessment and Feedback Agent:
1. The teacher uploads an assignment and a student response, and uploads them to the platform.
2. The platform generates instant, individualized feedback and a grade using Azure OpenAI.
3. The feedback is stored locally in the browser.
4. The teacher reviews the feedback and grade, and can edit or modify them as needed.
## 🔮 Future Enhancements
- Integration with major Learning Management Systems (LMS) for streamlined workflow.
- Expansion of supported assignment types and subjects.
- Development of analytics dashboards for deeper insights into class performance.
## 📽️ Demonstration Video
[Coming soon: View a full demonstration of the agent in action.]
## 👥 Team
- **Josh Creek**
[jcreek.co.uk](https://jcreek.co.uk)
## 🛠️ Getting Started
This project uses [SvelteKit](https://kit.svelte.dev/) and [TypeScript](https://www.typescriptlang.org/) with [pnpm](https://pnpm.io/) as the package manager.
## 9. Setup, Usage, and Testing
### Prerequisites
- [Node.js](https://nodejs.org/) (v18 or newer recommended)
- [Node.js](https://nodejs.org/) (v23 or newer recommended)
- [pnpm](https://pnpm.io/installation)
### Installation & Running Locally
#### Developing
### Running Locally
Once you've installed dependencies with `pnpm install`, start a development server:
```bash
pnpm run dev
# or start the server and open the app in a new browser tab
pnpm run dev -- --open
```
@@ -177,108 +207,72 @@ pnpm run build
You can preview the production build with `npm run preview`.
## 🕹️ Real-Time Events: PartyKit Setup
This project uses [PartyKit](https://partykit.io/) for real-time tool usage event streaming between the frontend and backend.
### Running PartyKit Locally
1. **Install dependencies** for PartyKit:
```sh
cd partykit
npm install
```
2. **Set up your `.env` file** (in the project root):
```env
VITE_PARTYKIT_BASE_URL=ws://127.0.0.1:1999
PARTYKIT_BASE_URL=ws://127.0.0.1:1999
```
These variables are required for both the SvelteKit frontend and backend to connect to your local PartyKit server.
3. **Start the PartyKit dev server**:
```sh
cd partykit
npm run dev
```
The server will be available at `ws://127.0.0.1:1999/party/<room>`.
4. **Start the SvelteKit frontend** (in a separate terminal):
```sh
pnpm run dev
```
### Deploying PartyKit to Production
1. **Update your `.env` for production**:
```env
VITE_PARTYKIT_BASE_URL=wss://<your-connection-string>.partykit.dev
PARTYKIT_BASE_URL=wss://<your-connection-string>.partykit.dev
```
2. **Deploy PartyKit**:
```sh
cd partykit
npm run deploy
```
Wait for the domain provisioning to complete.
3. **Update your frontend/backend to use the production WebSocket URL** (as above).
### Troubleshooting
- If you see `Invalid URL` errors, make sure your environment variables are set and that you have restarted your dev servers after editing `.env`.
- Always run the PartyKit dev server from the `partykit` directory.
See also `.env.example` for sample configuration.
## 🧪 Running BDD Tests (Cucumber + Playwright)
### PartyKit Setup for Real-Time Events
This project includes end-to-end BDD (Behavior-Driven Development) tests using [Cucumber.js](https://github.com/cucumber/cucumber-js) and [Playwright](https://playwright.dev/).
### Prerequisites
#### Prerequisites
- All application dependencies installed (see above)
- [Node.js](https://nodejs.org/) and [pnpm](https://pnpm.io/)
### Install Playwright Browsers
#### Install Playwright Browsers
If you haven't already, install Playwright's required browsers:
```bash
pnpm exec playwright install
```
### Running the Tests
#### Running the Tests
1. Start the SvelteKit dev server:
```bash
pnpm run dev
```
(Or use `pnpm run bdd:full` to auto-start the server and run tests.)
2. In a separate terminal, run the BDD tests:
```bash
pnpm run test:bdd
```
This will execute all feature files in `tests/bdd/features/` using step definitions in `tests/bdd/steps/`.
### Test Output & Screenshots
#### Test Output & Screenshots
- Test results will be shown in the terminal.
- On failure, a screenshot will be saved to the `screenshots/` directory in the project root (see `tests/bdd/support/hooks.ts`).
- Screenshot filenames are based on the scenario name.
### Customizing/Debugging
#### Customizing/Debugging
- You can run a specific feature file:
```bash
pnpm run test:bdd -- tests/bdd/features/assessment_submission.feature
```
- For more verbose output, add `--format progress` or `--format summary`.
### Project Scripts
#### Project Scripts
- `pnpm run test:bdd` – Run all BDD tests
- `pnpm run bdd:full` – Start dev server and run all BDD tests (requires [start-server-and-test](https://github.com/jsdom/start-server-and-test))
For more information, see the `package.json` scripts section.
## 📚 Resources
- [Hack Together: AI Agents Hackathon – Introduction & Getting Started](https://www.youtube.com/watch?v=RNphlRKvmJQ)
- [Hack Together: AI Agents Hackathon – Building Your Agent](https://www.youtube.com/watch?v=Aq30zfbWNSQ)
---
## 10. License
## 📌 License
Licensed under the Business Source License 1.1.
See LICENSE file for details.
---
## 🔮 Future Enhancements
- Integration with major Learning Management Systems (LMS) for streamlined workflow.
- Expansion of supported assignment types and subjects.
- Development of analytics dashboards for deeper insights into class performance.
---
**Built with love for teachers.**