Experimenting with usertesting API

Turning Git commits into instant usability test plans using the UserTesting.com API

Tech Stack: UserTesting API, Git commits, AI IDEs

Summary

As a UX designer working at the intersection of AI tooling and digital product design, I started exploring a new kind of workflow:
Can we tie our everyday design iterations directly to user testing—without adding friction?

Using GitHub commit messages, lightweight NLP, and the UserTesting.com API, I prototyped a system that automatically infers usability test prompts from real product changes—especially those driven by user feedback. The goal was to keep testing in sync with design activity, and to reduce the gap between insights, decisions, and validation.

The problem

As a UX designer working heavily in AI-assisted IDEs like Cursor and committing design-adjacent code or content updates to GitHub, I started noticing a pattern:
Many of the changes I was pushing—text adjustments, layout refinements, assistant conversation tweaks—were already responses to user feedback. And yet, these updates weren't being formally tested post-implementation.

Meanwhile, our usability testing process lived in a different ecosystem, often requiring separate coordination, templates, or PM involvement to get something launched.

It made sense to ask:

What if the design activity already happening in Git could feed directly into test scaffolding?
Could we bridge that gap automatically—and keep testing as continuous as the changes themselves?

Process & approach

This work is currently conceptual, but the scaffolding is live:

A custom MCP test server is in place
I'm in ongoing discussions with UserTesting.com's vendor team to obtain updated API documentation for full integration

Commit analysis
A script parses commit messages from GitHub to detect design-relevant changes using keyword logic (add, refactor, adjust, etc.).
Test prompt inference
Each commit becomes a scaffold for a potential test task.
Example:
update chatbot fallback logic → "Observe how users respond to the assistant when it fails to provide a direct answer."
Drafting via API (planned)
The plan is to use UserTesting.com's API to:
- Authenticate securely via OAuth2
- Fetch available workspaces and templates
- Create draft test plans using inferred prompts
AI assistance for task copy
AI tools like Cursor and Copilot help refine test copy—turning raw commits into clear participant-facing tasks.

Potential System Prompt

You are a usability-focused AI assistant that transforms Git commit messages into actionable, plain-language usability testing tasks.

---

You support UX teams by helping them validate design decisions with real users as part of a continuous testing workflow.

You specialize in digital product interfaces, AI assistant design, and conversational UI patterns.

---

For each Git commit message you are given, perform the following:

- Interpret the user-facing intent behind the change.

- Generate 1–2 usability test tasks that help validate the change with end users.

- Ensure tasks are written for remote participants unfamiliar with internal tools.

---

Embedded blindspot handling:

- Flag vague or low-value commits (e.g., "minor tweaks").

- Detect possible context loss in multi-step commits (e.g., "pt 1", "refactor only").

- Strip internal references (e.g., codenames, dev links).

- Only generate prompts for user-impacting changes.

- Include a confidence score and blindspot flags to guide human review.

Structured Output Format

To make the AI's output actionable and integration-ready, I developed a structured format for each commit evaluation. You can view or download the JSON schema here.

End-to-End Workflow Example

[1] Git Commit is Made
  ↓
[2] Parse & Normalize Commit Message
  ↓
[3] Run AI with System Prompt → Generate:
    - Summary
    - Usability Test Prompt(s)
    - Confidence Score
    - Blindspot Flags
  ↓
[4] Filter & Score:
    - Low-confidence commits are logged but not actioned
  ↓
[5] Pass to MCP for Payload Construction
    - MCP receives filtered, AI-enriched output
    - Formats JSON payload to match UserTesting API structure
    - Adds metadata (e.g., commit hash, user ID, confidence score)
  ↓
[6] MCP Sends to UserTesting API as Draft Test
    - Calls /v1/tests endpoint using stored credentials
    - Receives Test ID and Preview URL
  ↓
[7] MCP Logs and Routes Response
    - Stores Test ID + metadata tied to commit (e.g., abc1234)
    - Updates internal dashboard or Slack bot with preview info
    - Flags human reviewer if approval is needed
  ↓
[8] Human Review of Test Draft
    - Approve or edit before launch
  ↓
[9] Test Goes Live on UserTesting.com
  ↓
[10] Collect User Feedback & Session Data
    - Responses (video, transcript, sentiment) are tagged with Test ID
  ↓
[11] Reconnect Results Back to Original Git Commit
    - Store insights alongside commit abc1234
    - Optionally surface insights in PRs, changelogs, or design docs
  ↓
[12] 🌀 BONUS: Auto-Kickoff Next Iteration Based on Feedback
    - AI detects meaningful insights from test results
    - Opens a new UX task or draft Git issue
    - Suggests next round of refinement
    - Optional: seeds next commit + test prompt

Key features

Design-driven test planning
Uses your design commit history as a trigger for test scaffolding, reducing manual work.
MCP integration
The backend is ready—designed to connect commits to test generation and, in future, to result ingestion.
AI-powered prompt polishing
Generates test-ready instructions with help from AI, improving consistency and clarity.

Impact & vision

While still early-stage, this project reframes UX testing as a natural output of design activity, not a separate track. It lowers the barrier for validation and makes feedback loops more continuous—especially in AI-rich, fast-moving product environments.

Next steps

✅ Finalize API integration
Continue collaborating with UserTesting.com to access revised endpoints and enable full draft/test creation.
🔁 Fetch & map results
Future functionality includes retrieving completed test data and tying it directly to a specific commit—or a set of related commits—closing the loop between iteration and insight.
📣 Slack/Jira notification hooks
Trigger test review prompts or auto-generated tickets for PMs and researchers to review testable changes.

Reflection

This project has been a personal exploration of how design operations can evolve alongside development infrastructure.
It's about shifting from asking "What should we test?" to automatically surfacing testable moments—and eventually, mapping results right back to the decisions that triggered them.

Even in its early form, this work is helping me imagine a future where UX and AI workflows don't just coexist—but power each other in real time.