From c495ba4bc0e9db83ee11812e6ba450e08c78de61 Mon Sep 17 00:00:00 2001 From: Josh Creek <8179928+jcreek@users.noreply.github.com> Date: Fri, 25 Apr 2025 23:35:31 +0100 Subject: [PATCH] docs(*): Update readme to reflect latest functionality changes --- README.md | 368 +++++++++++++++++++++++++++--------------------------- 1 file changed, 181 insertions(+), 187 deletions(-) diff --git a/README.md b/README.md index 670b358..bfa251b 100644 --- a/README.md +++ b/README.md @@ -2,167 +2,197 @@ ![BDD Tests](https://github.com/jcreek/AutomatedAssessmentFeedbackAgent/actions/workflows/ci.yml/badge.svg?branch=main) -> An intelligent, agentic AI system designed to significantly reduce teachers' workloads by providing instant assessment and personalized differentiated feedback and follow-on activities for student assignments. +> An intelligent, agentic AI system built by a teacher-turned-engineer to deliver instant grading, personalized feedback, and real-time transparent reasoning - saving teachers hours and improving student outcomes. - -## Who is this for? -This project is designed for: -- **Teachers** who want to save time on grading and provide more consistent, individualized feedback to students. -- **Schools and educational institutions** seeking to improve the quality and efficiency of assessment and feedback workflows. - - -## ⚡ Workflow At a Glance -- Upload an assignment and student response (file or text) -- Instantly receive detailed, actionable feedback and differentiated individualized follow-on tasks and a grade -- Review, delete, or clear past assessments in the history -- All actions are accessible, error-proof, and demo resilient +--- ## 🏆 Hackathon Info -This project was developed for the [Microsoft Hack Together: AI Agents Hackathon](https://microsoft.github.io/AI_Agents_Hackathon/) (April 8–30, 2025). +Built for the [Microsoft Hack Together: AI Agents Hackathon](https://microsoft.github.io/AI_Agents_Hackathon/) (April 8–30, 2025). +Status: Hackathon prototype/MVP. +See the [Official Rules](https://microsoft.github.io/AI_Agents_Hackathon/rules/). -- See the [Official Rules](https://microsoft.github.io/AI_Agents_Hackathon/rules/) -- Status: Hackathon prototype/MVP +--- +## 📽️ Demonstration Video -## 🎯 What It Does (Key Features) -- **Automated Grading:** Instantly grades student submissions (text or file) for any assignment/task. -- **Personalized Feedback:** Actionable, contextual feedback including grade, strengths, areas for improvement, individualized activity, and teacher suggestion. -- **Assessment History:** All assessments are saved locally (browser localStorage) for later review and demo resilience. -- **History Management:** Delete individual assessments or clear all history, with confirmation dialogs for safety. -- **Navigation:** Seamless navigation between upload and results/history pages. +[Coming soon: View a full demonstration of the agent in action.] + +--- + +## 📁 Table of Contents + +1. [Project Overview](#1-project-overview) +2. [Team Information](#2-team-information) +3. [What Makes This Unique](#3-what-makes-this-unique) +4. [Who Is This For?](#4-who-is-this-for) +5. [How It Works](#5-how-it-works) +6. [Technical Details](#6-technical-details) +7. [Human-in-the-Loop Innovation](#7-human-in-the-loop-innovation) +8. [Accessibility and Responsible AI](#8-accessibility-and-responsible-ai) +9. [Setup, Usage, and Testing](#9-setup-usage-and-testing) +10. [License](#10-license) + +--- + +## 1. Project Overview + +### Elevator Pitch + +An agentic AI system for teachers that transforms grading and feedback. +Transparent, real-time tool use and reasoning builds trust—giving educators instant, individualized assessments and actionable feedback for students that teachers can understand, edit, and trust. + +--- + +## 2. Team Information + +Created and built entirely by me, **Josh Creek** - an ex-teacher and current software engineer. +Mission: build AI tools that **genuinely empower educators**, **save time**, and **improve student outcomes**. + +--- + +## 3. What Makes This Unique + +_Why is this different from other AI grading tools?_ + +- **Real-Time Transparency:** Teachers see every reasoning step and tool the agent chooses—live. +- **Personalized Feedback:** Detailed, contextual feedback plus strengths, improvements, follow-on activities, and suggestions. +- **Human-in-the-Loop Escalation:** The agent escalates edge cases to the teacher for review (never guessing blindly). +- **Resilient History Management:** All assessments stored locally for review, even during demos (no student data stored anywhere but the teacher's browser). +- **Accessibility First:** Full screen reader support, keyboard navigation, color contrast compliance. - **Robust Error Handling:** Friendly, actionable error messages for upload, AI, or network issues. -- **Loading Spinner:** Visual feedback while grading is in progress. -- **Accessibility:** Screen reader-friendly, keyboard-accessible, and color-contrast aware. +- **Real-Time Agentic Progress Visualization:** Not just a loading spinner—teachers see, in real time, which tools and reasoning steps the agent chooses as it grades. This transparency builds trust and helps educators understand _how_ AI arrives at its conclusions. +- **Built by a Teacher, for Teachers:** Practical, realistic, classroom-aware design. -## ⚙️ How It Works +--- -1. **Teacher uploads a student submission** (file or text) and assignment description. -2. **AI (Azure OpenAI)** generates instant, individualized feedback and a grade. -3. **Results and assessment history** are displayed for review, deletion, or clearing. -4. **All data is stored locally** (no backend required for history/demo). +## 4. Who Is This For? -## 🚀 Technical Stack +### Audience -- **TypeScript:** Ensures reliability, maintainability, and scalability. -- **Azure OpenAI:** Provides advanced NLP capabilities for nuanced and accurate assessment. -- **Azure Cognitive Services:** Enhances semantic analysis for precise feedback generation. +- **Teachers** seeking to save time, improve feedback quality, and maintain control. +- **Schools/Educational Institutions** aiming to modernize and streamline assessment workflows. - -## 📖 Educational Impact +### Educational Impact - Reduces hours spent grading and marking. - Improves quality and consistency of student feedback. - Allows teachers more time to focus on direct student interaction and lesson planning. +--- + +## 5. How It Works + +### Workflow At A Glance + +1. Upload assignment instructions and student response (text for demo; file support planned). +2. Agent generates real-time, transparent reasoning and instant grading. +3. If the agent thinks it needs it, it can optionally ask the teacher for suppport (Human-In-The-Loop). +4. Teacher reviews, edits, or clears assessments from local history. + +### Demo it! + +> Upload a meaningful text (or minimal/off-topic text to trigger Human-in-the-Loop escalation). +> Watch live tool use and reasoning. +> Review or clear past assessments directly in history. + +### Architecture Diagram + +```mermaid +flowchart TD + Teacher["Teacher (User)"] + Upload["Upload Page"] + Results["Results/History Page"] + AgenticProgress["AgenticProgress Component"] + LocalStorage["localStorage (Browser)"] + APIGrade["API: /api/grade"] + APIHITL["API: /api/hitl-review"] + EventStream["PartyKit WebSocket (Real-time Agent Progress)"] + OpenAI["Azure OpenAI (NLP, Grading, Feedback)"] + + %% Standard Grading Flow + Teacher -->|Uploads assignment & student work| Upload + Upload -->|Calls| APIGrade + APIGrade -->|Sends to| OpenAI + APIGrade -->|Streams progress| EventStream + EventStream -->|Updates| AgenticProgress + APIGrade -->|Returns feedback| Results + Results -->|Saves| LocalStorage + Results -->|Displays| Teacher + + %% HITL Escalation (Human-in-the-Loop Path) + APIGrade -- Escalates if unclear/minimal --> TeacherReview["Teacher Review (HITL Prompt)"] + TeacherReview -->|Submits review| APIHITL + APIHITL -->|Injects teacher feedback| OpenAI + APIHITL -->|Returns final feedback| Results +``` + +Assessment history is **only stored locally** (no external storage). + +--- + +## 6. Technical Details + +- **Frontend:** [SvelteKit](https://kit.svelte.dev/) + [TypeScript](https://www.typescriptlang.org/) +- **Real-time Events:** [PartyKit](https://partykit.io/) +- **AI:** [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) + [Azure Cognitive Services](https://azure.microsoft.com/en-us/products/ai-services/cognitive-services) + +--- + +## 7. Human-in-the-Loop Innovation + +### How It Works + +- If a student submission is minimal/ambiguous, the agent returns `HUMAN_REVIEW_REQUIRED` and explains why. +- Teacher intervenes, providing direct feedback. +- Agent resumes, using the human input to complete grading and feedback generation. + +### Why It Matters + +- **Transparency:** Teachers always see _why_ the agent requests help, with clear reasoning. +- **Control:** Teachers remain in the loop for edge cases, ensuring fairness and pedagogical soundness. +- **Innovation:** This collaborative workflow demonstrates how agentic AI can augment, not replace, expert educators—addressing a key hackathon challenge. + +### Demo + +> Upload a blank or nonsense submission to trigger the HITL workflow and see the transparent escalation. + +--- + +## 8. Accessibility and Responsible AI + +### Accessibility + +- Screen reader and keyboard friendly. +- Color contrast meets WCAG AA standards. +- Accessible real-time agent progress updates. +- Tested with browser accessibility tools. + +### Responsible AI -## 🛠️ Responsible AI I am committed to responsible and ethical use of AI in education. This project: + - Uses Azure OpenAI and Cognitive Services, which comply with Microsoft's responsible AI principles. - Does not retain or share student data beyond local processing in the browser (history is stored in localStorage only). - Clearly communicates to users when they are interacting with AI-generated feedback. - Is designed to minimize bias by providing transparent, explainable feedback and allowing teachers to review/edit results. - Does not use student data for model training or any secondary purpose. -## ♿ Accessibility -Accessibility is a core priority: -- The interface is screen reader-friendly, with proper semantic HTML and ARIA labels. -- All features are keyboard accessible (tab navigation, focus indicators). -- Color contrast meets WCAG AA standards for readability. -- Error messages and progress indicators are accessible to assistive technologies. -- The site has been tested with browser accessibility tools and screen readers. +--- - -## 🗺️ Architecture Diagram - -```mermaid -flowchart TD - %% User - Teacher["Teacher (User)"] - - %% Frontend - Upload["Upload Page"] - Results["Results/History Page"] - AgenticProgress["AgenticProgress Component"] - LocalStorage["localStorage (Browser)"] - - %% API - APIEndpoint["API: /api/grade (Netlify serverless function)"] - APINote["API endpoints are Netlify serverless functions (SvelteKit endpoints)"] - APIEndpoint -.-> APINote - - %% PartyKit - EventStream["PartyKit WebSocket (Real-time Agent Progress)"] - PartyKitNote["PartyKit provides WebSocket-based real-time updates on agent progress/tools."] - EventStream -.-> PartyKitNote - - %% Azure - OpenAI["Azure OpenAI (NLP, Grading, Feedback)"] - - %% Data Flow - Teacher -->|Uploads assignment & student work| Upload - Upload -->|Calls| APIEndpoint - APIEndpoint -->|Sends data & assignment description| OpenAI - APIEndpoint -->|Streams grading progress| EventStream - EventStream -->|Updates progress| AgenticProgress - APIEndpoint -->|Returns feedback & grade| Results - Results -->|Planned: Teacher reviews/edits feedback| Results - Results -->|Saves assessment| LocalStorage - Results -->|Displays feedback, history| Teacher - - PrivacyNote["Assessment history is stored only in the user’s browser (localStorage), not sent to any backend."] - LocalStorage --> PrivacyNote - - class Teacher user; - class Upload,Results,AgenticProgress,LocalStorage frontend; - class APIEndpoint api; - class EventStream partykit; - class OpenAI azure; -``` - -## 👥 Teacher Workflow Example - -Here's an example of how a teacher might use the Automated Assessment and Feedback Agent: - -1. The teacher uploads an assignment and a student response, and uploads them to the platform. -2. The platform generates instant, individualized feedback and a grade using Azure OpenAI. -3. The feedback is stored locally in the browser. -4. The teacher reviews the feedback and grade, and can edit or modify them as needed. - -## 🔮 Future Enhancements - -- Integration with major Learning Management Systems (LMS) for streamlined workflow. -- Expansion of supported assignment types and subjects. -- Development of analytics dashboards for deeper insights into class performance. - - -## 📽️ Demonstration Video - -[Coming soon: View a full demonstration of the agent in action.] - - -## 👥 Team -- **Josh Creek** - [jcreek.co.uk](https://jcreek.co.uk) - - -## 🛠️ Getting Started - -This project uses [SvelteKit](https://kit.svelte.dev/) and [TypeScript](https://www.typescriptlang.org/) with [pnpm](https://pnpm.io/) as the package manager. +## 9. Setup, Usage, and Testing ### Prerequisites -- [Node.js](https://nodejs.org/) (v18 or newer recommended) + +- [Node.js](https://nodejs.org/) (v23 or newer recommended) - [pnpm](https://pnpm.io/installation) -### Installation & Running Locally - -#### Developing +### Running Locally Once you've installed dependencies with `pnpm install`, start a development server: ```bash pnpm run dev - # or start the server and open the app in a new browser tab pnpm run dev -- --open ``` @@ -177,108 +207,72 @@ pnpm run build You can preview the production build with `npm run preview`. - -## 🕹️ Real-Time Events: PartyKit Setup - -This project uses [PartyKit](https://partykit.io/) for real-time tool usage event streaming between the frontend and backend. - -### Running PartyKit Locally - -1. **Install dependencies** for PartyKit: - ```sh - cd partykit - npm install - ``` -2. **Set up your `.env` file** (in the project root): - ```env - VITE_PARTYKIT_BASE_URL=ws://127.0.0.1:1999 - PARTYKIT_BASE_URL=ws://127.0.0.1:1999 - ``` - These variables are required for both the SvelteKit frontend and backend to connect to your local PartyKit server. -3. **Start the PartyKit dev server**: - ```sh - cd partykit - npm run dev - ``` - The server will be available at `ws://127.0.0.1:1999/party/`. -4. **Start the SvelteKit frontend** (in a separate terminal): - ```sh - pnpm run dev - ``` - -### Deploying PartyKit to Production - -1. **Update your `.env` for production**: - ```env - VITE_PARTYKIT_BASE_URL=wss://.partykit.dev - PARTYKIT_BASE_URL=wss://.partykit.dev - ``` -2. **Deploy PartyKit**: - ```sh - cd partykit - npm run deploy - ``` - Wait for the domain provisioning to complete. -3. **Update your frontend/backend to use the production WebSocket URL** (as above). - -### Troubleshooting -- If you see `Invalid URL` errors, make sure your environment variables are set and that you have restarted your dev servers after editing `.env`. -- Always run the PartyKit dev server from the `partykit` directory. - -See also `.env.example` for sample configuration. - -## 🧪 Running BDD Tests (Cucumber + Playwright) +### PartyKit Setup for Real-Time Events This project includes end-to-end BDD (Behavior-Driven Development) tests using [Cucumber.js](https://github.com/cucumber/cucumber-js) and [Playwright](https://playwright.dev/). -### Prerequisites +#### Prerequisites + - All application dependencies installed (see above) - [Node.js](https://nodejs.org/) and [pnpm](https://pnpm.io/) -### Install Playwright Browsers +#### Install Playwright Browsers + If you haven't already, install Playwright's required browsers: ```bash pnpm exec playwright install ``` -### Running the Tests +#### Running the Tests + 1. Start the SvelteKit dev server: ```bash pnpm run dev ``` (Or use `pnpm run bdd:full` to auto-start the server and run tests.) - 2. In a separate terminal, run the BDD tests: ```bash pnpm run test:bdd ``` This will execute all feature files in `tests/bdd/features/` using step definitions in `tests/bdd/steps/`. -### Test Output & Screenshots +#### Test Output & Screenshots + - Test results will be shown in the terminal. - On failure, a screenshot will be saved to the `screenshots/` directory in the project root (see `tests/bdd/support/hooks.ts`). - Screenshot filenames are based on the scenario name. -### Customizing/Debugging +#### Customizing/Debugging + - You can run a specific feature file: ```bash pnpm run test:bdd -- tests/bdd/features/assessment_submission.feature ``` - For more verbose output, add `--format progress` or `--format summary`. -### Project Scripts +#### Project Scripts + - `pnpm run test:bdd` – Run all BDD tests - `pnpm run bdd:full` – Start dev server and run all BDD tests (requires [start-server-and-test](https://github.com/jsdom/start-server-and-test)) For more information, see the `package.json` scripts section. -## 📚 Resources -- [Hack Together: AI Agents Hackathon – Introduction & Getting Started](https://www.youtube.com/watch?v=RNphlRKvmJQ) -- [Hack Together: AI Agents Hackathon – Building Your Agent](https://www.youtube.com/watch?v=Aq30zfbWNSQ) +--- +## 10. License -## 📌 License Licensed under the Business Source License 1.1. See LICENSE file for details. +--- + +## 🔮 Future Enhancements + +- Integration with major Learning Management Systems (LMS) for streamlined workflow. +- Expansion of supported assignment types and subjects. +- Development of analytics dashboards for deeper insights into class performance. + +--- + +**Built with love for teachers.**