AI Agent Observability Glossary: Key Terms & Concepts

An app or software sends out alerts, which are actionable and content-rich push notifications. These alerts can be for various reasons, including messages from teammates, reporting manager...read more

An audit trail is a step-by-step sequential record that offers evidence of a financial transaction's documented history back to its source. With the use of the audit trail, an auditor can ...read more

Cost tracking is an essential feature of software that allows businesses to monitor and manage their expenses and budgets in a systematic manner. It enables companies to keep track of thei...read more

Root cause analysis is a feature commonly found in software programs that allows users to identify the underlying cause of a problem or issue. This tool is particularly useful in troublesh...read more

Performance Monitoring is a feature that allows users to closely monitor the performance of their software and applications. It is a critical aspect for businesses and organizations, as it...read more

Compatibility with OpenTelemetry, a vendor-neutral open standard for generating and shipping traces, metrics, and logs from applications. Because instrumentation follows the standard rathe...read more

Agent evaluation scores what an agent produced against a fixed set of test cases, so a prompt change, a model swap or a new action can be compared with the version already in production. ...read more

Self-hosting means the software runs inside infrastructure you control rather than the vendor, usually from an open-source build, a container image or a licensed enterprise package. For a...read more

An agent trace is the ordered record of one complete agent run: the request that started it, every model call, every retrieval, every action the agent took, what each returned, and the ans...read more

Agent simulation drives scripted users at an agent before it reaches customers, running multi-turn conversations, edge cases and adversarial inputs at volume and grading what comes back. ...read more

Online evaluation scores live production runs as they happen rather than waiting for an offline test cycle, usually with a model acting as grader against named criteria. It catches the fa...read more

Human annotation gives reviewers a queue of real agent runs to label, so a person who knows the business can mark what good looks like and feed that back into the automated graders. It is...read more

Session replay lets an engineer step back through a finished run in order, seeing what the agent saw at each point rather than reading a flat log. For agents that drive a web browser some...read more

Trajectory evaluation scores the path an agent took rather than the answer it ended on: whether it chose the right action, whether it repeated a step, whether a handoff to a sub-agent carr...read more

Span-level tracing breaks a run into individually timed units, each one carrying its own input, output, duration, token count and cost, nested to show which step triggered which. This is ...read more