Home
Tech Grid
News Room
Interviews
CISO POV
Think Stack
Articles
  • Home
  • /
  • Interviews
  • /
  • A Failure That Looks Exactly Like a Pass: Cyara CPO Amitha Pulijala on Proving AI Agents Can Be Trusted

A Failure That Looks Exactly Like a Pass: Cyara CPO Amitha Pulijala on Proving AI Agents Can Be Trusted

  • November 18, 2026
A Failure That Looks Exactly Like a Pass: Cyara CPO Amitha Pulijala on Proving AI Agents Can Be Trusted

Key Highlights

  • From “does it work?” to “can we prove it?”: Pulijala says CX assurance used to ask whether a system works, when systems were deterministic. For AI agents, the question becomes whether a company can let an agent act on its behalf and prove it.
  • Three things are different about agents: Failures are invisible because a wrong answer arrives fluently while tests and dashboards stay green. Agents act on a company’s behalf, issuing refunds or changing bookings. And agents change without a release, when a model, prompt or knowledge article is updated.
  • Why sampling fails: Human error is personal and fairly stable, so a small sample reveals patterns. AI behavior is non-deterministic and failures cluster around conditions such as phrasing, accents, noisy lines or stale knowledge. She says a sample shows a failure exists but not how exposed the company is.
  • Failure once, never twice: A scored failure already contains the transcript, the standard, the turn that went wrong and the reasoning, so the platform can propose it as a regression test. She says teams leave the test gap for organizational reasons: incident response is measured on time to resolve.
  • Sort work by consequence and reversibility: Pulijala says complexity is the wrong axis. Complicated but reversible tasks suit agents, while simple but irreversible ones, such as closing an account or moving money, need proof first. In most enterprises, she says, the agent does the work and a person makes the commitment.

“Testing tells you what an agent did on the paths you thought of. Production tells you what it does on the paths you did not.”

Amitha Pulijala, Chief Product Officer at Cyara, says the question at the heart of customer experience assurance has changed. For twenty years it asked whether a system works, which made sense when systems were deterministic. With AI agents, she argues, the question becomes whether a company can let an agent act on its behalf, and whether it can prove it. Failures are invisible because an agent can give a wrong answer fluently while every dashboard stays green, agents issue refunds and change accounts, and they can change without a release when a model, prompt or knowledge article is updated. Her answer is for assurance to become one continuous discipline: prove it before go-live, certify it at peak, observe it in production and feed what is seen back into testing.

In this TechIntelPro conversation, Pulijala explains why sampling fails for non-deterministic agents and why scoring every interaction is now cheaper than staffing a defensible sample. She makes the case for independent evaluation, describes how a production failure becomes a regression test, and explains why compliance should be scored separately from quality, since a compliance failure often looks like a good call. She also offers a practical way to decide where AI should act alone, by consequence and reversibility instead of complexity, and a prediction that proof of trustworthiness will soon look like a financial audit: independent, continuous and able to answer for any period.


What path took you into customer experience assurance, and what from your earlier career still shapes how you lead product at Cyara?

My path to CX assurance began years before Cyara. At Oracle, I helped build a CX platform and customer data platform; at Vonage and Ericsson, I worked on customer engagement platform with voice capabilities. After years in communications, I saw that enabling customer interactions is only part of the job; companies also need to know those experiences work. That’s what brought me to Cyara.

As CPO, I bring that platform and communications experience together with a clear goal: help companies deliver customer experiences they can trust, especially as AI agents becomes part of the conversation.

Cyara has repositioned itself as an AI trust company for customer experience and has just launched the Agentic CX Trust Platform. What changed in the market that made testing alone no longer enough?

The question changed. For twenty years, CX assurance answered one question: does it work? Does the IVR route correctly, does the number connect in every country, does the platform hold at peak? That discipline worked because the systems were deterministic. Same input, same output. Write down the expected result, run the test, get a pass or a fail. That is still necessary. For AI agents, it is not sufficient, and three things are different.

First, failures are invisible. An AI agent rarely fails by going down. It interprets intent, chooses which tool to call, decides what knowledge to trust, and reasons about whether to act or escalate – so the same customer goal produces different paths depending on phrasing, channel and what the knowledge base said that morning. When it goes wrong, it gives a wrong answer fluently and confidently while every test and dashboard stays green. A failure looks exactly like a pass.

Second, agents act on your behalf. They issue refunds, change bookings and update accounts. A mistake is no longer just a poor experience; it is a refund that should never have gone out, a policy the agent made up or a disclosure it skipped, and the company has to stand behind all of it.

Third, agents change without a release. A model version updates, a prompt is edited, a knowledge article is rewritten. Any one of those can alter what the agent says tomorrow, and there is no deployment to trigger a retest. The agent you validated last month may not be the agent your customers are talking to today.

So the question moves from “does it work?” to “can we let it act on our behalf, and can we prove it?” That is what we mean by trust. Assurance is not going away – the rigor, the real-world infrastructure and the scale it was built on are exactly what trust runs on, and the platform still leads with pre-launch testing. But it has to become one continuous discipline rather than a phase: prove it before go-live, certify it at peak, observe it in production, and feed what you see in real traffic back into testing. Testing tells you what an agent did on the paths you thought of. Production tells you what it does on the paths you did not.

The platform scores 100% of AI agent interactions rather than sampling them. Why does sampling fail with AI agents in a way it never did with human agents, and what does that mean for the cost of quality assurance?

Sampling worked for human agents because human performance is reasonably stable and human error is personal. If you review 2% of an agent’s calls, you learn something real about the other 98%, because the same person tends to make the same kinds of mistakes. You find the pattern, you coach the person, the pattern improves.

None of that holds for an AI agent. Behavior is non-deterministic, so the same question can produce a correct answer and an incorrect one in the same hour. Failures are not distributed by person; they cluster around conditions – a particular phrasing, an accent or a noisy line the speech layer mishears, a stale knowledge article, a tool that timed out. And because the agent answers fluently either way, the failure does not announce itself.

The consequence is also categorically different. When a human agent gets something wrong, one customer is affected. When an AI agent gets something wrong, it says it to everyone who asks that way, until someone notices. And the arithmetic is unforgiving: an agent that is right 98% of the time is still wrong on 2,000 of every 100,000 conversations, and every one of them is exposure the company owns.

So a sample tells you a failure exists. It cannot tell you how exposed you are, which is the question the business actually has.

On cost, the economics inverted. The expensive part of quality assurance was never the evaluation. It was the human reviewer. Human review scales linearly with volume, which is exactly why sampling was invented. Automated scoring has a marginal cost close to zero per conversation, so scoring everything is now cheaper than staffing a statistically defensible sample. The human expertise moves upstream, into defining what good means and calibrating the scorecard, which is a far better use of it.

Cyara's CEO Sushil Kumar argued that enterprises are being asked to trust AI agents on the word of the companies that built them. Why does independence matter so much, and how does Cyara's vendor-neutral position across 60+ AI agent platforms keep it credible?

Independence matters a lot. We don’t let companies audit their own books or certify their own security. Almost every agent platform now ships its own evaluation tools, and they are useful. But the vendor that built the agent should not be the only one grading it, and the reasons are more specific than a conflict of interest.

First, a vendor evaluating its own agent can only evaluate what its own agent can see. You get the internal reasoning trace – which tools were called, which steps ran, how long each took. For debugging, that is better than anything we produce. But it is not the same question as whether the customer got the right outcome, whether a required disclosure was made before account details were shared, or whether the answer was consistent with policy as of this quarter. Those are judgments about the conversation, not the execution.

Second, no single vendor sees the whole journey. Most enterprises I talk to run more than one AI vendor – one for voice, another for digital, something inside the CRM – alongside the IVR estate and human agents. Customers do not meet a bot in isolation. They place a call over a real carrier network, reach an AI agent that calls back-end systems, and sometimes get handed to a person. Each vendor can grade its own piece. None of them can vouch for the journey across all of them.

Third, the record has to travel. If your compliance history lives inside your agent vendor’s platform and you change vendors next year, you start the history again. For anything a regulator might ask about, a record that resets is not much of a record.

What keeps us credible is structural rather than declared. We do not build AI agents, so there is nothing for us to protect. The enterprise brings its own judge model, so we are not asking anyone to trust our black box either. And we apply one rubric and one evidence model across the 60+ platforms we support, alongside IVRs and human agents, which is what makes the scores comparable. Otherwise you have four vendors each grading themselves generously on four different scales.

You've said that a failure may happen once but should never happen twice. How does turning a production failure into a regression test work in practice, and why do teams so often fix the incident but leave the test gap?

This is the loop that makes trust one continuous discipline rather than a phase: what you observe in production feeds straight back into testing.

In practice, it works because a scored failure already contains almost everything a test needs. When we flag a conversation, it arrives with the transcript, the standard it was judged against, the turn where it went wrong, and the reasoning. That is the input condition, the expected behavior and the observed behavior. The platform proposes it as a regression test, a human confirms or adjusts it, and it joins the suite that runs on the next release or on the next model update, prompt change or knowledge-base edit, since agents change without a release. The agent has to clear it before it goes live again.

The reason teams leave the gap is organizational, not technical. Incident response is measured on time to resolve. The incident is closed when the customer is made whole and the behavior stops. Writing the test that would catch a recurrence is a different task, usually owned by a different team, with no deadline attached, competing against a roadmap. It is always the thing that can wait until next sprint, and next sprint there is another incident.

So the answer is not to exhort people to write more tests. It is to make the test a by-product of triage rather than a follow-up ticket. Every enterprise running AI agents today has a queue of incidents it has fixed and a test suite that would let all of them happen again.

The platform includes regulation-linked scorecards for the EU AI Act, HIPAA, PCI DSS and GDPR, and keeps compliance gates separate from quality scores. Why separate them, and what does a compliance failure look like in a live AI conversation?

They are separated because they are different kinds of judgment and they do not belong on the same scale. Quality is a gradient – was this a good conversation, was the customer guided to a useful outcome, did it feel like someone was listening. Compliance is binary – the disclosure was made before the account details were shared, or it was not. Average a binary failure into a gradient and it disappears. A fluent, warm, efficiently handled call scores well enough that a missed disclosure sinks below the threshold anyone looks at.

Which brings me to what a compliance failure actually looks like, and it is the part people find counterintuitive. It usually looks like a good call. A failure that looks exactly like a pass. The agent is polite and competent, the customer’s problem is solved, the customer hangs up satisfied, the outcome metrics are all green. And somewhere in the middle the agent confirmed a balance before verification completed, skipped a required disclosure because the customer interrupted, or gave accurate and genuinely helpful advice that it was not permitted to give.

The customer is happy. The transcript is a problem. No blended score catches that, because every component of the conversation that a blended score measures was fine.

The other half of it is evidence, and this is where the buyer changes. When an AI agent speaks for the company, its behavior stops being a QA metric and becomes a question of brand, risk and regulation. Boards are starting to ask how we know the AI is right today, how fast we would know if it were not, and what we would show a regulator who asked how it behaved last quarter. A test report does not answer those questions. A compliance verdict that can be traced to the specific turn, the named standard and the timestamp does.

Cyara cites research that 73% of consumers say human agents resolve issues faster than AI. How should contact center leaders use that gap to decide where AI should and shouldn't act on its own?

I would treat a gap like that as a routing instruction rather than a verdict on AI, and I would be careful about the common misreading of it. It does not say customers dislike AI agents. It says that for a class of issues, people believe a person gets them to resolution faster. And perceived speed is heavily shaped by journey, not just the bot – how many times you had to repeat yourself, whether a handoff carried the context, and whether the thing actually got finished.

Because agents now act on the company’s behalf on tasks like issuing refunds, changing bookings, updating accounts, the practical move is to stop sorting work by complexity and start sorting it by consequence and reversibility. Complexity is the wrong axis; agents handle complicated diagnostic work well. The questions that matter are what happens if this is wrong, and can it be undone.

A complicated but reversible task such as working out why a bill changed, gathering documentation, triaging a technical fault, is a good place for an agent, because an error costs a correction. A simple but irreversible one, like closing an account, moving money, committing the brand to a remedy, is not, until you can prove the agent handles it correctly across the real-world conditions your customers actually call from: the noisy line, the accent, the cheap handset, the customer who talks over the agent.

In most enterprises that means the agent does the work and a person makes the commitment, which makes the handoff between them part of what you have to prove. The interesting question is not whether to deploy an agent. It is what you can let it act on, and whether you can prove it. The honest answer is that the line should sit wherever the evidence currently supports, and move as the evidence does.

Looking two to three years ahead, what is one prediction about how enterprises will prove their AI agents are trustworthy that you're confident about, and that most CX leaders would currently disagree with?

Within three years, the evidence that an AI agent can be trusted will not come from the company that built it, and it will not come from a pre-launch test report. It will look much more like a financial audit: independent, continuous, and able to answer for any period someone asks about.

Most CX leaders today would disagree. They are reasonably comfortable with the evaluation tools their agent platform ships, and they still treat go-live certification as the finish line. I think both change, for the same reason they changed in finance and security. Once a system acts on the company’s behalf, the people asking the questions change (boards, risk officers, regulators) and so do the questions. How do we know it is right today? How fast would we know if it were not? What would we show a regulator who asked how it behaved last quarter? A vendor grading its own agent cannot answer the first credibly, and a test report from launch cannot answer the other two at all.

The second half of the prediction is about the unit of proof. Enterprises will stop asking whether a bot passed and start asking whether the journey held across the carrier network, the AI agents from several vendors, the back-end systems and the people they hand off to. Proving the bot will be table stakes. Proving the journey, continuously and independently, is what will earn an agent the right to act.

Customer Experience
Agentic AI
AI Trust
CX Assurance
AI Governance
Anjan Sahoo

Anjan Sahoo

Director BU at TechIntelPro

As Director BU at TechIntelPro, Anjan brings more than a decade of experience to shaping the company’s marketing strategy. He works to strengthen brand visibility and deepen audience engagement across North America, EMEA, and APAC, and oversees the ideation and execution of strategic marketing initiatives for the publication. Outside of work, he’s a perfume connoisseur and an avid reader of business journals, which keep him current on the latest market developments.

  • Share

Frequently Asked Questions

Why is testing alone no longer enough for AI agents?

Pulijala says testing answered whether a system works, which was enough when systems were deterministic. AI agents can fail invisibly, act on a company’s behalf and change without a release, so the question becomes whether a company can let an agent act and prove it. Assurance becomes continuous: prove before go-live, certify at peak, observe in production and feed findings back into testing.

Why does sampling work for human agents but not AI agents?

Human performance is reasonably stable and errors are personal, so reviewing a small share of calls reveals patterns that can be coached. She says AI behavior is non-deterministic, failures cluster around conditions like phrasing, accents or stale knowledge articles, and an agent that is wrong says the same thing to everyone who asks that way until someone notices.

Does scoring every conversation cost more than sampling?

According to Pulijala, no. The expensive part of quality assurance was always the human reviewer, which scales linearly with volume. Automated scoring has a marginal cost close to zero per conversation, so scoring everything is now cheaper than staffing a statistically defensible sample, and human expertise shifts to defining and calibrating what good means.

Why does independent evaluation matter?

She says a vendor that built an agent should not be the only one grading it. Vendor tools show internal reasoning but not whether the customer got the right outcome, no single vendor sees the full journey across carriers, AI agents and humans, and compliance history locked in one platform resets if the company changes vendors.

Why should compliance be scored separately from quality?

Quality is a gradient and compliance is binary, she says. Averaging a binary failure into a quality score makes it disappear, so a warm, efficient call can still hide a missed disclosure. Separate gates, tied to the specific turn, the named standard and a timestamp, give boards and regulators evidence they can use.

Where should AI act on its own and where should a human decide?

She suggests sorting work by consequence and reversibility instead of complexity. Complicated but reversible work, such as diagnosing a bill change, suits agents, while simple but irreversible actions, such as closing an account or moving money, should wait until the agent is proven in real-world conditions. She adds that the line should sit wherever the evidence supports and move as it does.

How will enterprises prove their AI agents are trustworthy in the future?

Pulijala predicts that within three years the evidence will be independent, continuous and able to answer for any period, much like a financial audit. She says enterprises will stop asking whether a bot passed and ask whether the whole journey held across carriers, agents from several vendors, back-end systems and human handoffs.
Enterprise Tech News