Case Study · AltaML · 2026
Turning unstructured feedback into prioritized product signal.
I designed and built an AI-assisted UAT triage system that helped PMs move from manually sorting feedback responses to reviewing ranked product signal — reducing triage time by 92% across multiple client projects.
Context
The product we were testing was an AI agent for digital ad operations.
At AltaML, I worked on a Meta Ads automation agent built for digital marketing teams. The agent took an advertisement campaign plan (a spreadsheet with details on target audiences, budget, assets, placements, and optimization) from a digital marketing agency and converted it directly into ads inside Meta’s platform.
The product also helped teams evaluate campaign performance after launch by looking at signals like reach, views, and campaign success. From there, it could recommend optimization actions such as reallocating budget, pausing underperforming ads, or adjusting campaign strategy.
Because the agent touched high-impact workflows — campaign setup, publishing, performance review, and budget decisions — testing could not just be a final QA checkbox. We needed to understand where users got blocked, where the agent created incorrect outputs, and where the workflow felt confusing before scaling the product more broadly.
Why UAT mattered
UAT is where product assumptions meet real user behaviour.
User Acceptance Testing, or UAT, is the stage where real users test whether a product works in the context of their actual workflows. It is different from internal QA because the goal is to understand whether the product is usable, trustworthy, and valuable in the environment it was built for rather than just testing for bugs.
For an AI agent, this feedback loop was especially important. A technically successful output could still fail the user if it was hard to understand, required too much manual cleanup, or made an optimization suggestion the team did not trust.
That made UAT feedback one of the most important inputs into the product lifecycle. It helped us decide what needed to be fixed immediately, what patterns were emerging across users, and what gaps should influence the roadmap before broader rollout.
The problem
Product feedback was scaling faster than teams could process it.
During the staged rollout, we were collecting large volumes of feedback (600+ feedback submissions) on just the Meta ads agent. There was a ton of good feedback. But, the problem was extracting usable decisions from the messy, inconsistent feedback. And, quickly enough to keep product iteration moving.
PMs were spending hours each week manually reading submissions, identifying duplicates, interpreting vague or insufficient comments, prioritizing issues, and routing work to engineering. Every reviewer had slightly different judgment calls, which meant prioritization quality varied depending on who was triaging that day.
The operational cost became more obvious as beta velocity increased. Critical issues could sit buried inside low-quality submissions for hours, duplicate reports distorted prioritization discussions, and roadmap conversations slowed because teams were still trying to agree on what the feedback actually meant.
After seeing the same workflow break across multiple client projects, I realized this wasn’t a Meta ads agent specific process issue — it was a product infrastructure issue.
Approach
Discovery first, then design around the decision-making workflow.
Before building anything, I interviewed PMs across multiple client engagements to understand where the triage workflow was actually breaking down. I wanted to separate “AI sounds useful here” from the real operational bottleneck.
The pattern became clear quickly: PMs were generally aligned on how to prioritize issues once signal was clear. The expensive part was everything before that: reading vague submissions, identifying duplicates, extracting themes, and determining which items were even worth discussing.
To better understand the workflow, I mapped the end-to-end feedback lifecycle in Lucidchart. This helped surface key operational gaps, including inconsistent prioritization logic, duplicate issue fragmentation, unclear escalation paths, and the amount of manual effort spent preparing feedback before PM review.
From there, I translated those pain points into system requirements and designed the architecture for the automation itself. Instead of trying to automate product judgment, I focused on reducing the operational overhead around it.
I intentionally avoided fully autonomous prioritization. PM trust mattered more than automation coverage, so ambiguous submissions were escalated for human review instead of silently classified with high confidence.
I chose to build the system in n8n rather than custom infrastructure for three reasons: PM teams could fork workflows without engineering support, integrations already existed across our intake stack, and scoring logic could evolve quickly as different client teams calibrated priorities differently.
Identify operational bottlenecks
Interviewed PMs across multiple projects and mapped the full UAT feedback lifecycle to identify friction points, workflow gaps, and repeated manual tasks.
Define the automation architecture
Translated workflow pain points into system requirements and designed the end-to-end n8n pipeline, including classification, enrichment, scoring, duplicate handling, and routing logic.
Turn messy feedback into ranked signal
Built the n8n automation that classified submissions, escalated ambiguous cases, clustered duplicates, and ranked issues based on severity, reach, effort, and frequency.
Make the workflow reusable
Designed the system so PM teams could duplicate and recalibrate workflows independently across projects without requiring engineering support.
Click the diagram to open it full size — then zoom in to read each node.
Key decisions
The judgment calls that actually shipped this.
Structured scoring over AI summaries.
Early on, I realized free-text summaries created more reading, not less. PMs didn’t need prettier feedback. A consistent way to rank competing issues was more important. I designed a weighted scoring rubric around severity, reach, effort, and frequency so the output supported prioritization decisions directly.
Optimize for trust, not maximum automation.
One of the biggest risks was the system confidently misclassifying ambiguous feedback. I introduced confidence-based escalation rules and an “insufficient” state so unclear submissions stayed human-reviewed instead of silently routed incorrectly.
Design for operational reuse, not one-off automation.
Each client team prioritized issues differently, so I built a reusable workflow architecture that PMs could duplicate and recalibrate without engineering support. This ensured the automation was actually a scalable feedback operations tool.
Tradeoffs & constraints
The messy parts that shaped the system.
One of the hardest parts of the project was balancing automation coverage with reliability. Vague feedback often lacked enough detail to classify confidently, but aggressively “enhancing” submissions risked introducing invented context or overconfident prioritization.
I added strict prompt constraints and a fallback “insufficient” state because preserving feedback fidelity mattered more than maximizing automation rates. In practice, this meant intentionally accepting some manual review in exchange for higher PM trust in the outputs.
Another challenge was duplicate clustering. Different testers often described the same workflow issue in completely different language, which made simple keyword matching unreliable. I ended up using semantic comparison logic paired with feature-area constraints to reduce false duplicate grouping.
Impact
How it landed.
The biggest shift was how PMs interacted with feedback entirely. Instead of spending hours manually parsing submissions, teams moved toward reviewing ranked product signal and discussing prioritization decisions earlier in the cycle.
The workflow became especially valuable during high-volume testing periods where duplicate issues and vague feedback previously slowed iteration speed and roadmap alignment discussions.
On the Meta Ads agent specifically, I also synthesized 150+ telemetry logs alongside the UAT signal. Together, the two data sources helped identify system gaps, re-prioritize the roadmap, and strengthen launch readiness before broader rollout.
After launch
What changed beyond the metrics.
- PM reviews shifted from categorization to prioritization. Teams spent less time interpreting feedback and more time discussing tradeoffs and roadmap impact.
- Recurring workflow friction became easier to identify. Semantic duplicate clustering surfaced patterns that were previously hidden across fragmented submissions.
- Feedback quality improved over time. As testers saw which submissions they got notified for follow ups on, the quality and consistency of incoming feedback increased naturally.
- The workflow became reusable infrastructure. Other client teams could adapt the system for their own UAT pipelines rather than rebuilding triage processes from scratch.
Reflection
Key learnings from building AI-assisted product infrastructure.
The hardest problems weren’t technical.
Going into the project, I assumed most of the complexity would come from the LLM workflows themselves. In reality, the harder challenge was defining what “high-signal feedback” actually meant across different PMs and client teams. A large part of the project became aligning on prioritization logic, escalation paths, and what level of uncertainty teams were comfortable automating.
Trust matters more than automation coverage.
One of the biggest mindset shifts for me was realizing that a partially automated workflow people trust is far more valuable than a fully autonomous workflow people constantly second guess. That changed how I approached edge cases, confidence thresholds, and human review throughout the system.
Operational workflows are product experiences too.
Before this project, I mostly thought about user experience in the context of customer-facing interfaces. This project pushed me to think more deeply about internal operational systems — how PMs consume information, how teams make prioritization decisions, and how workflow design directly affects product velocity.
Personal growth
What I’ll carry into future product work.
Spend more time understanding the workflow before building.
The workflow mapping and discovery phase ended up being one of the most valuable parts of the project. Taking the time to understand how PMs actually triaged feedback made the final system significantly more aligned with how teams operated in practice.
Good PM work is often infrastructure work.
This project changed how I think about product management. Some of the highest-leverage work is improving the systems teams use to learn, prioritize, and make decisions faster.
AI systems need operational guardrails.
Working on this project made me much more thoughtful about how AI systems should behave in production workflows. Designing fallback states, escalation paths, and evaluation coverage became just as important as the model outputs themselves.