Your AI Agent Is Up. That Doesn’t Mean It Is Working

Imagine an enterprise AI agent handling a customer request. It responds in 300 milliseconds. The application returns no errors. The APIs it calls are available, infrastructure utilization is normal and every service in the dependency chain is healthy. The dashboards are green.

The agent also gives the customer the wrong answer and initiates a business process that should never have been started.

From the perspective of conventional observability, the application worked. From the perspective of the customer and the business, it failed.

That gap is quickly becoming one of the defining challenges for platform engineering. The systems being deployed are no longer limited to executing deterministic instructions and returning recognizable errors when something breaks. AI applications are probabilistic and context-sensitive. They can retrieve the wrong information, misunderstand an instruction, select the wrong tool or follow a plausible but incorrect path without ever crashing.

The application can be up, fast and completely wrong.

That is why Dynatrace’s agreement to acquire Arize for $915 million is more significant than another acquisition in the already crowded observability market. The deal is the news hook, but the larger story is what it says about the changing definition of software reliability.

Observability Meets Evaluation

Dynatrace has signed a definitive agreement to acquire Arize for approximately $815 million in cash plus replacement equity awards for Arize employees joining Dynatrace. The transaction has not yet closed and remains subject to regulatory review and customary closing conditions.

Dynatrace comes from the world of production observability: Applications, infrastructure, dependencies, performance, user experience and business processes. Arize comes from the newer world of AI evaluation and observability: Model behavior, agent traces, retrieved context, output quality and the difficult work of determining why an AI system produced the result it did.

Bringing those capabilities together reflects an increasingly obvious reality. Enterprises cannot operate application observability and AI evaluation as unrelated disciplines.

Traditional observability can tell us whether the model endpoint responded, how long the call took, whether an API failed and how many infrastructure resources the application consumed. Those remain essential signals. But they cannot independently tell us whether the model’s answer was correct, whether the retrieved context was relevant or whether an agent chose an appropriate course of action.

Arize CEO Jason Lopatecki put the problem in straightforward terms in his comments on the transaction. AI systems do not behave like traditional software, and visibility into why they fail is neither as straightforward nor as consistent.

That is the strategic logic behind the deal. One of the established names in production observability has concluded that observing whether AI software is running is not enough. Enterprises also need to evaluate whether it is working.

Reliability Now Has Three Layers

For years, platform teams have built around a relatively stable operational definition of reliability. Is the application available? Is it responding within the expected latency? Are its infrastructure and dependencies healthy? Is it generating errors?

AI agents do not eliminate those questions, but they add two more layers.

The first remains system reliability. Is the application available and responsive? Are its models, databases, APIs and tools reachable? Are resource utilization and error rates within acceptable ranges?

The second is behavioral reliability. Did the agent retrieve the appropriate information? Did it select the correct tool? Did it follow the right policies? Did it take a sensible sequence of steps, or did it wander through repeated calls and unnecessary retries before stumbling into an answer?

The third is business reliability. Did the agent accomplish the intended task accurately, safely and at an acceptable cost? Did it resolve the customer’s problem, identify the actual security incident, complete the transaction correctly or produce a usable software change?

A system can pass the first test while failing the other two.

That distinction matters because an AI failure may not produce an exception, a timeout or an HTTP 500 error. The output may be fluent, properly formatted and completely plausible. The agent may finish its workflow, declare success and record no technical failure at all.

The most dangerous AI failures may be the ones that look perfectly healthy on the operations dashboard.

This Is a Platform Problem

The immediate temptation will be to treat AI evaluation as the responsibility of every individual AI development team. Let each group select its own tracing framework, define its own evaluation metrics, maintain its own prompt registry and build its own cost dashboards.

We have seen this movie before. It leads to duplicated work, inconsistent controls, incompatible telemetry and an assortment of tools that become increasingly difficult to govern. Platform engineering exists in large part to keep every application team from assembling the same operational foundation independently.

AI reliability should follow the same path as CI/CD, secrets management, identity, policy enforcement and conventional observability. It should become a shared platform capability.

That does not mean platform engineers must personally decide whether every customer-service answer, legal summary or medical recommendation is correct. Product owners and domain experts must define acceptable outcomes for their applications.

The platform team’s role is to provide the common machinery through which those definitions can be implemented, tested and monitored consistently.

That machinery begins with tracing, but it cannot end there. Platform teams need to give application developers a way to reconstruct an agent’s complete execution path, including its model calls, retrieved context, tool selections, arguments, results, branches and retries.

They also need lineage. Which model version was running? Which prompt and system instructions were used? What context was retrieved? Which data sources, tools and policies were available? What changed between the version that passed yesterday and the version that failed today?

Then comes evaluation. Development teams need offline tests and regression gates before deployment, but they also need continuous evaluation in production. An agent that performs well against a static test set may behave differently when it encounters live data, ambiguous requests and changing business conditions.

Those evaluations may measure correctness, relevance, groundedness, completeness, policy compliance and task completion. Some can be automated. Others will require sampling, human review and feedback from users or business systems.

Cost must be part of the same picture. An agent can reach the correct answer while taking 50 unnecessary steps, invoking an expensive model repeatedly and calling tools whose results it never uses. That is a functional success and an operational failure. Token usage, inference costs, retries and tool calls need to be connected to the outcome rather than reported as isolated consumption metrics.

Tracing Is Not Evaluation

It is important not to collapse all of this into the word “observability.”

A trace tells us what the agent did. An evaluation helps determine whether what it did was acceptable. Business measurement tells us whether the result created value. These capabilities reinforce one another, but they are not interchangeable.

The distinction is similar to watching a human employee complete a task. A detailed record might show every application opened, every database queried and every action taken. That record still does not tell us whether the employee made the right decision. We need both the process and the result.

Guardrails play another role. They establish what an agent is permitted to do and can prevent certain dangerous actions. But a guardrail cannot guarantee that every permitted action is sensible or that every generated answer is accurate.

Guardrails constrain behavior. Tracing records behavior. Evaluation judges behavior. Business measurement determines whether the behavior produced a worthwhile result.

A complete AI platform will need all four.

A Common Telemetry Foundation Is Emerging

The industry is beginning to build the standards needed to make these capabilities portable. OpenTelemetry’s evolving GenAI observability conventions provide standardized ways to record model calls, token usage and, when organizations choose to capture them, prompts, completions, tool calls and tool results.

The broader collection of GenAI semantic attributes includes agent and workflow identity, prompt names, conversation identifiers, data sources and evaluation scores. In other words, evaluation is beginning to appear alongside traces and operational telemetry as part of the same observability foundation.

The standards remain a work in progress, and organizations will have legitimate privacy and security concerns about recording prompts, responses and retrieved context. That telemetry may contain customer information, proprietary data or other sensitive material. Platform teams will need controls governing what is collected, who can access it, how long it is retained and when it must be redacted.

Still, the direction is important. A shared semantic model gives platform teams a chance to instrument AI applications consistently without binding every development group to a different proprietary format.

Enterprises are already recognizing the requirement. A Futurum survey of 139 enterprise decision-makers, previously reported by PlatformEngineering.com, found that AI observability ranked fourth among observability purchasing priorities at 37.4%, while agent observability ranked sixth at 30.9%. Both ranked ahead of Kubernetes observability and infrastructure monitoring.

Companies may still be experimenting with agents, but they already understand that operating them will require a different level of visibility.

A New Platform Contract

Platform engineering has traditionally promised development teams a reliable way to build, deploy and operate software. That contract is now expanding.

An AI-native platform must also provide a repeatable way to measure quality, identify behavioral regressions, control cost and connect agent activity to business outcomes. It must make evaluation part of the delivery and operating process, not something performed occasionally by a specialized AI team after problems appear.

The Dynatrace-Arize deal will not settle what AI observability ultimately looks like, nor will one combined platform solve every evaluation problem. But it is a strong indication of where the market is moving. Conventional application telemetry and AI behavioral evaluation are becoming parts of the same operational system.

The next generation of platform engineering will not discard uptime, latency or error rates. It will put them in their proper place. They tell us whether the machinery is running. They do not tell us whether the agent is accomplishing the job.

An AI agent can be up, fast and available—and still be completely wrong. The platform must be able to tell the difference.