diff --git a/README.md b/README.md index 8431673..633885b 100644 --- a/README.md +++ b/README.md @@ -1 +1,408 @@ -# Microsoft-AI-Agents-Hackathon-Submission \ No newline at end of file +### Project Name + +AutomatedAssessmentFeedbackAgent + +### Description + +# AI-Powered Automated Assessment and Feedback Agent + +![BDD Tests](https://github.com/jcreek/AutomatedAssessmentFeedbackAgent/actions/workflows/ci.yml/badge.svg?branch=main) + +> An intelligent, agentic AI system built by a teacher-turned-engineer to deliver instant grading, personalized feedback, and real-time transparent reasoning - saving teachers hours and improving student outcomes. + +--- + +## 🏆 Hackathon Info + +Built for the [Microsoft Hack Together: AI Agents Hackathon](https://microsoft.github.io/AI_Agents_Hackathon/) (April 8–30, 2025). +Status: Hackathon prototype/MVP. +See the [Official Rules](https://microsoft.github.io/AI_Agents_Hackathon/rules/). + +Created using: + +- JavaScript/TypeScript +- Azure AI Agent Service + +--- + +## 📽️ Demonstration and Hackathon Experience Video + +[Watch it on YouTube](https://www.youtube.com/watch?v=FAi9-s9WP84) + +--- + +## ✨ Live Demos / Key Functionality + +**Key Features:** + +- Instant, AI-powered grading and feedback +- Human-in-the-Loop escalation for ambiguous cases +- Real-time agentic tool use and transparent reasoning +- Assessment history stored locally for privacy +- Fully tested and CI-backed for reliability + +Below are GIF demonstrations of the Automated Assessment Feedback Agent in action, showcasing core features and real-world scenarios: + +### 1. Human-in-the-Loop (HITL): Wrong Assignment Provided + +_The agent detects when a student submits the wrong assignment and facilitates teacher intervention._ + +![HITL: Wrong Assignment](https://raw.githubusercontent.com/jcreek/AutomatedAssessmentFeedbackAgent/refs/heads/main/docs/images/hack1.gif) + +### 2. HITL: English as an Additional Language (EAL) Student + +_Demonstrates tailored feedback and HITL support for a student with EAL needs._ + +![HITL: EAL Student](https://raw.githubusercontent.com/jcreek/AutomatedAssessmentFeedbackAgent/refs/heads/main/docs/images/hack2.gif) + +### 3. Assessing a Wordy Essay-Type Task + +_Shows the agent analyzing and providing feedback on a complex, open-ended essay response._ + +![Essay Assessment](https://raw.githubusercontent.com/jcreek/AutomatedAssessmentFeedbackAgent/refs/heads/main/docs/images/hack3.gif) + +### 4. Assessing a Computer Science Coding Task + +_The agent evaluates a programming submission, running code tests and generating feedback._ + +![Coding Task Assessment](https://raw.githubusercontent.com/jcreek/AutomatedAssessmentFeedbackAgent/refs/heads/main/docs/images/hack4.gif) + +--- + +## 📁 Table of Contents + +1. [Project Overview](#1-project-overview) +2. [Team Information](#2-team-information) +3. [What Makes This Unique](#3-what-makes-this-unique) +4. [Who Is This For?](#4-who-is-this-for) +5. [How It Works](#5-how-it-works) +6. [Technical Details](#6-technical-details) +7. [Automated Testing & Quality Assurance](#7-automated-testing--quality-assurance) +8. [Human-in-the-Loop Innovation](#8-human-in-the-loop-innovation) +9. [Accessibility and Responsible AI](#9-accessibility-and-responsible-ai) +10. [Security & Privacy](#10-security--privacy) +11. [Setup, Usage, and Testing](#11-setup-usage-and-testing) +12. [License](#12-license) + +--- + +## 1. Project Overview + +### Elevator Pitch + +An agentic AI system for teachers that transforms grading and feedback. +Transparent, real-time tool use and reasoning builds trust—giving educators instant, individualized assessments and actionable feedback for students that teachers can understand, edit, and trust. + +--- + +## 2. Team Information + +Created and built entirely by me, **Josh Creek** - an ex-teacher and current software engineer. +Mission: build AI tools that **genuinely empower educators**, **save time**, and **improve student outcomes**. + +--- + +## 3. What Makes This Unique + +_Why is this different from other AI grading tools?_ + +- **Real-Time Transparency:** Teachers see every reasoning step and tool the agent chooses—live. +- **Personalized Feedback:** Detailed, contextual feedback plus strengths, improvements, follow-on activities, and suggestions. +- **Human-in-the-Loop Escalation:** The agent escalates edge cases to the teacher for review (never guessing blindly). +- **Resilient History Management:** All assessments stored locally for review, even during demos (no student data stored anywhere but the teacher's browser). +- **Accessibility First:** Full screen reader support, keyboard navigation, color contrast compliance. +- **Robust Error Handling:** Friendly, actionable error messages for upload, AI, or network issues. +- **Real-Time Agentic Progress Visualization:** Not just a loading spinner—teachers see, in real time, which tools and reasoning steps the agent chooses as it grades. This transparency builds trust and helps educators understand _how_ AI arrives at its conclusions. +- **Built by a Teacher, for Teachers:** Practical, realistic, classroom-aware design. + +--- + +## 4. Who Is This For? + +### Audience + +- **Teachers** seeking to save time, improve feedback quality, and maintain control. +- **Schools/Educational Institutions** aiming to modernize and streamline assessment workflows. + +### Educational Impact + +- Reduces hours spent grading and marking. +- Improves quality and consistency of student feedback. +- Allows teachers more time to focus on direct student interaction and lesson planning. + +--- + +## 5. How It Works + +### Workflow At A Glance + +1. Upload assignment instructions and student response (text for demo; file support planned). +2. Agent generates real-time, transparent reasoning and instant grading. +3. If the agent thinks it needs it, it can optionally ask the teacher for suppport (Human-In-The-Loop). +4. Teacher reviews, edits, or clears assessments from local history. + +### Demo it! + +> Upload a meaningful text (or minimal/off-topic text to trigger Human-in-the-Loop escalation). +> Watch live tool use and reasoning. +> Review or clear past assessments directly in history. + +### System Flow Diagram (with HITL) + +This shows how a user interacts with the system and how requests are processed, including HITL. + +```mermaid +flowchart TD + Teacher["Teacher (User)"] + Upload["Upload Page"] + Results["Results/History Page"] + AgenticProgress["AgenticProgress Component"] + LocalStorage["localStorage (Browser)"] + APIGrade["API: /api/grade"] + APIHITL["API: /api/hitl-review"] + EventStream["PartyKit WebSocket (Real-time Agent Progress)"] + OpenAI["Azure OpenAI (NLP, Grading, Feedback)"] + + %% Standard Grading Flow + Teacher -->|Uploads assignment & student work| Upload + Upload -->|Calls| APIGrade + APIGrade -->|Sends to| OpenAI + APIGrade -->|Streams progress| EventStream + EventStream -->|Updates| AgenticProgress + APIGrade -->|Returns feedback| Results + Results -->|Saves| LocalStorage + Results -->|Displays| Teacher + + %% HITL Escalation (Human-in-the-Loop Path) + APIGrade -- Escalates if unclear/minimal --> TeacherReview["Teacher Review (HITL Prompt)"] + TeacherReview -->|Submits review| APIHITL + APIHITL -->|Injects teacher feedback| OpenAI + APIHITL -->|Returns final feedback| Results +``` + +### Architecture Diagram + +This shows the big-picture technical structure and how my chosen technologies fit together. + +The architecture includes: + +- Frontend: SvelteKit app (browser) + - Connects to PartyKit (for real-time events) + - Calls API endpoints (/api/grade, /api/hitl-review) + - Saves assessment history in browser localStorage +- Backend: SvelteKit API routes (serverless functions, e.g., Netlify) + - /api/grade: Handles grading, talks to Azure OpenAI + - /api/hitl-review: Handles teacher review, resumes AI grading +- Third-Party Services: + - Azure OpenAI (AI grading, feedback) + - Azure AI Agent Service (agentic workflow orchestration, tool use, LLM integration) + - PartyKit (WebSocket server for real-time updates) +- Data Storage: Only in browser (localStorage), no backend DB + +System architecture diagram showing component hosting, communication flows (HTTP, WebSocket), and external AI service integrations: + +![Architecture Diagram](https://raw.githubusercontent.com/jcreek/AutomatedAssessmentFeedbackAgent/refs/heads/main/docs/images/hackathon-architecture.png) + +--- + +## 6. Technical Details + +- **Frontend:** [SvelteKit](https://kit.svelte.dev/) + [TypeScript](https://www.typescriptlang.org/) +- **Real-time Events:** [PartyKit](https://partykit.io/) +- **AI & Agent Orchestration:** [Azure AI Agent Service](https://azure.microsoft.com/en-us/products/ai-services/ai-agent-service) (agentic workflows, tool use) + [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) (LLMs) + +--- + +## 7. Automated Testing & Quality Assurance + +This project is built with a strong focus on reliability and correctness: + +- **Comprehensive BDD Tests:** All core features are covered by automated, scenario-based tests using [Cucumber](https://cucumber.io/) and [Playwright](https://playwright.dev/). +- **Continuous Integration:** Every commit to the `main` branch triggers a full test suite via GitHub Actions ([see badge above](#ai-powered-automated-assessment-and-feedback-agent)), ensuring nothing breaks unexpectedly. +- **Real-World Scenarios:** Tests simulate real teacher/student interactions. +- **Easy to Run:** + ```bash + pnpm run test:bdd + ``` + Or view the [test scripts](./tests/bdd). + +**Result:** You can trust that this agent works—every time. + +--- + +## 8. Human-in-the-Loop Innovation + +### How It Works + +- If a student submission is minimal/ambiguous, the agent returns `HUMAN_REVIEW_REQUIRED` and explains why. +- Teacher intervenes, providing direct feedback. +- Agent resumes, using the human input to complete grading and feedback generation. + +### Why It Matters + +- **Transparency:** Teachers always see _why_ the agent requests help, with clear reasoning. +- **Control:** Teachers remain in the loop for edge cases, ensuring fairness and pedagogical soundness. +- **Innovation:** This collaborative workflow demonstrates how agentic AI can augment, not replace, expert educators—addressing a key hackathon challenge. + +--- + +## 9. Accessibility and Responsible AI + +### Accessibility + +- Screen reader and keyboard friendly. +- Color contrast meets WCAG AA standards. +- Accessible real-time agent progress updates. +- Tested with browser accessibility tools. + +### Responsible AI + +I am committed to responsible and ethical use of AI in education. This project: + +- Uses Azure OpenAI and Cognitive Services, which comply with Microsoft's responsible AI principles. +- Does not retain or share student data beyond local processing in the browser (history is stored in localStorage only). +- Clearly communicates to users when they are interacting with AI-generated feedback. +- Is designed to minimize bias by providing transparent, explainable feedback and allowing teachers to review/edit results. +- Does not use student data for model training or any secondary purpose. + +--- + +## 10. Security & Privacy + +- **No Persistent Storage:** All assessment history and student data are stored only in the user’s browser (localStorage). Nothing is ever uploaded to or retained on any backend server or database. +- **Ephemeral AI Agent Threads:** AI agent threads and any temporary data created during assessment are deleted immediately after grading is complete. +- **No Data Sharing:** Student data is never shared with third parties, used for model training, or processed for any purpose other than providing instant feedback. +- **User Control:** Users can review and clear their assessment history at any time from within the app. +- **Secure Communication:** All interactions with the Azure AI Agent Service are performed over secure, encrypted channels (HTTPS/WSS). +- **Privacy by Design:** The system is architected to maximize privacy and minimize data exposure at every step. + +--- + +## 11. Setup, Usage, and Testing + +### Prerequisites + +- [Node.js](https://nodejs.org/) (v23 or newer recommended) +- [pnpm](https://pnpm.io/installation) + +### Running Locally + +Once you've installed dependencies with `pnpm install`, start a development server: + +```bash +pnpm run dev +# or start the server and open the app in a new browser tab +pnpm run dev -- --open +``` + +#### Building + +To create a production version: + +```bash +pnpm run build +``` + +You can preview the production build with `npm run preview`. + +### PartyKit Setup for Real-Time Events + +This project includes end-to-end BDD (Behavior-Driven Development) tests using [Cucumber.js](https://github.com/cucumber/cucumber-js) and [Playwright](https://playwright.dev/). + +#### Prerequisites + +- All application dependencies installed (see above) +- [Node.js](https://nodejs.org/) and [pnpm](https://pnpm.io/) + +#### Install Playwright Browsers + +If you haven't already, install Playwright's required browsers: + +```bash +pnpm exec playwright install +``` + +#### Running the Tests + +1. Start the SvelteKit dev server: + ```bash + pnpm run dev + ``` + (Or use `pnpm run bdd:full` to auto-start the server and run tests.) +2. In a separate terminal, run the BDD tests: + ```bash + pnpm run test:bdd + ``` + This will execute all feature files in `tests/bdd/features/` using step definitions in `tests/bdd/steps/`. + +#### Test Output & Screenshots + +- Test results will be shown in the terminal. +- On failure, a screenshot will be saved to the `screenshots/` directory in the project root (see `tests/bdd/support/hooks.ts`). +- Screenshot filenames are based on the scenario name. + +#### Customizing/Debugging + +- You can run a specific feature file: + ```bash + pnpm run test:bdd -- tests/bdd/features/assessment_submission.feature + ``` +- For more verbose output, add `--format progress` or `--format summary`. + +#### Project Scripts + +- `pnpm run test:bdd` – Run all BDD tests +- `pnpm run bdd:full` – Start dev server and run all BDD tests (requires [start-server-and-test](https://github.com/jsdom/start-server-and-test)) + +For more information, see the `package.json` scripts section. + +--- + +## 12. License + +Licensed under the Business Source License 1.1. +See LICENSE file for details. + +--- + +## 13. Future Enhancements + +- Integration with major Learning Management Systems (LMS) for streamlined workflow. +- Expansion of supported assignment types and subjects. +- Development of analytics dashboards for deeper insights into class performance. +- Image upload for handwritten work, especially for teachers of younger students. + +--- + +**Built with love for teachers.** + + +### Language & Framework + +- [ ] Python +- [ ] C# +- [ ] Java +- [x] JavaScript/TypeScript +- [ ] Microsoft Copilot Studio +- [ ] Microsoft 365 Agents SDK +- [x] Azure AI Agent Service + +### Project Repository URL + +https://github.com/jcreek/AutomatedAssessmentFeedbackAgent + +### Deployed Endpoint URL + +_No response_ + +### Project Video + +https://www.youtube.com/watch?v=FAi9-s9WP84 + +### Team Members + +jcreek + +### Registration Check + +- [x] Each of my team members has filled out the registration form