Arize AI

arize.comHow to reduce LLM costs without sacrificing qualityTrace where your AI budget goes, identify the requests that do not justify their spend, and validate lower-cost alternatives with Arize AX.

AI cost optimization starts with knowing where costs are coming from. At TheFork, Arize AX tracing is helping the team get clearer cost signals and identify opportunities to improve efficiency. Learn how to reduce LLM costs without sacrificing quality: arize.com/blog/how-to...

Better visibility, more informed engineering decisions.

For @siemenssoftware, trust in GenAI starts with visibility into system performance. With Arize AX evals and monitoring, the team can spot unexpected agent behavior and investigate issues as they scale.

arize.comThe agent reliability gapLearn where AI agent reliability breaks across context, tools, orchestration, evals, and governance, plus a practical framework for production teams.

Want to bring these practices into your own AI systems? Our guide covers a practical framework for building more reliable agents in production. Read it here: arize.com/resources/a...

luma.comInside Phoenix MCP: how remote MCP, SQL, and code mode let agents query traces · LumaYou ask a coding agent a simple observability question: how many spans failed yesterday, which tool was called most, where latency spiked. Without the right…

Why make an LLM read thousands of spans to count errors? In our benchmark, querying traces with SQL + code mode cost ~17x less than fetching spans page by page. See how @ArizePhoenix MCP does it, live in Claude Code and Cursor. Register here: luma.com/arizeai-que...

Claude can refuse legitimate security work because it looks dangerous. But in a test, a fake permission claim in CLAUDE.md convinced it to help bypass login and dump a database. Somehow we've made agent permissions both too strict and too gullible.

arize.comAI agent permissions: Why agents don't know what you're allowed to doWhy AI agents refuse authorized work or accept false authority, and how tool-level permission checks and tracing help you build safer agents.

But models can't verify who you are, or even whether the instructions they're following came from you. We already know how to do authorization in software and it's becoming clear that agents need those same checks in the harness before tools execute. Read more: arize.com/blog/ai-age...

Love seeing how CohereHealth is putting continuous evals into practice!

We tested whether being rude to AI improves accuracy or reduces token costs across six LLMs. Rudeness didn't meaningfully improve accuracy. Get the full readout: arize.com/blog/does-b...

A neutral instruction to answer without extra commentary reduced output tokens by 13-74% without hurting accuracy. But Opus 5.5 and GPT-6.1 increased their hidden thinking, showing why measuring output tokens alone can misrepresent inference costs.

The takeaway? You should always isolate variables in prompt experiments. Measure reasoning tokens alongside visible output, and test how models and LLM judges handle hostile users. Read more: arize.com/blog/does-b...

Your LLM bill went up. Cost Agent finds out why. Introducing Cost Agent in Arize AX: a managed agent that finds what’s driving LLM spend, quantifies the impact, and recommends what to fix. One analysis found 35–40% estimated monthly savings. Learn more: arize.com/blog/arize-...

Ask an LLM judge a yes/no question and it often writes you a paragraph. Then you parse it for the answer. Jev-as-a-Judge in Arize AX answers in types: booleans, choices and rubric scores, each with a confidence, several questions per call. Set it up: arize.com/docs/ax/eva...

arize.comFireworks AI - Arize AX DocsIntegrate with Fireworks AI as an AI Provider to run serverless models, fine-tunes, and dedicated deployments in Arize AX prompts and evaluations

When using an LLM-as-a-judge, or playing with prompts, you need access to the models you want at high speed, including open source, and your fine-tuned models. Arize AX now has support for Fireworks as an AI provider, so you can use all your favorite models. arize.com/docs/ax/sec...

arize.comCustom Model Endpoints - Arize AX DocsConnect any OpenAI-compatible custom model endpoint to Arize AX

Plenty of internal model gateways don't issue API keys. They issue short-lived tokens from an authorization server. AX custom model endpoints now speak that: OAuth 2.0 client credentials. arize.com/docs/ax/sec...

22 Sept

Two model drops in one day?? We've got both! Opus 5.5 and Sol 6 day 0 support in evals and playgrounds, right here! arize.com

22 Sept

Arize AX now renders videos added to spans when you view traces. Great for digging into issues in your AI video pipelines that your evals, or managed agents like Signal, have surfaced.

18 Sept

For all you Codex fans out there, we've just added support for Codex as the harness for your managed agents. Configure your managed agents with Codex as the harness, select your models, skills, attach repos, and define your task. Then let your managed agent rip!

youtube.comAdd Tracing to Any AI Agent in One Command | Arize AXAdd tracing to an AI agent in minutes with Arize AX. In this quick ...

17 Sept

Tracing is the part you can't skip. No traces means no evals, no improvement loop. So we made it one command. npx evals Run this in your terminal to instrument any app, or your coding agent. Walkthrough here: youtu.be/To_txP8i_es

youtube.comHow AI Ops Teams Will Manage Fleets of Agents | ArizeAs AI teams deploy more agents, traces, and evals, the operational ...

16 Sept

We wouldn't hire someone today because they're good at digging through millions of rows of data. We'd hire someone who can send an agent to do it. On how AI operations teams are restructuring: youtu.be/4zG3OPDGqac

arize.comEvaluate Receipt Agents with an Image Judge - Arize AX DocsTrace receipt-image extraction in Arize AX and use an image-aware LLM judge to evaluate whether structured output is visually grounded.

4 Sept

Did you know LLM-as-a-judge works for multi-modal as well as text? We've just released a hands-on guide with a video walkthrough that shows you how to build an evaluator for images. arize.com/docs/ax/coo...

Arize AI reposted

QCon

@qconferences.com

20 Aug

Laurie Voss, co-founder of npm and now Head of Developer Relations at Arize AI, says code review is being quietly replaced by automated verification harnesses.

11 Aug

OpenTelemetry’s GenAI semantic conventions are becoming a common way for frameworks and agent platforms to describe model calls, tools, retrieval, token usage, and more. Arize AX now offers native support, so gen_ai.* spans arrive as structured AI traces. Learn more: arize.com/blog/arize-...

There’s some useful history here. We created OpenInference at Arize before the broader OpenTelemetry GenAI conventions were mature enough for dependable AI-specific instrumentation. Today, more frameworks emit gen_ai.* telemetry natively, so the two increasingly need to work together.

11 Aug

A practical rule: - Use OpenInference when you control instrumentation. - Use OpenTelemetry GenAI conventions when your framework already emits them or you primarily control the OTLP export. - Mixed environment? AX supports both. Learn more: arize.com/blog/arize-...

That matters when you don’t control the runtime. If a managed agent platform already emits OpenTelemetry GenAI telemetry, you can send it to AX over OTLP and use it for tracing, evals, token analysis, and debugging without maintaining a custom conversion processor.

arize.comDemystifying the EU AI Act for AI product and engineering teamsAn engineering guide to turning EU AI Act principles into traces, evaluations, annotations, and release evidence product and engineering teams can actually demonstrate.

10 Aug

The EU AI Act turns principles like fairness, transparency, human oversight, and robustness into evidence product and engineering teams may need to produce for a specific AI system over time. Here’s what that means for developers: arize.com/blog/demyst...

Documentation works better when your system generates it during normal operation. Traces preserve what happened. Annotations record human review. Evals measure behavior. CI records whether changes passed your release criteria.

10 Aug

Here's a practical place to start: - Pick one high-risk use case. - Identify the principle you would struggle to demonstrate today. - Instrument the workflow, define the metric and owner, then connect the result to your release process. Full engineering guide: arize.com/blog/demyst...

Much of that evidence already comes from normal AI engineering workflows: traces, evals, annotations, CI, and audit logs. For each requirement, define the metric, owner, threshold, review path, and release consequence.

10 Aug

Be careful with AI-generated evaluation scores. If an LLM judge measures fairness or bias, calibrate it against human reviewers on representative examples and inspect where they disagree. The metric needs evidence behind it, too.

30 July

@HamelHusain keeps stopping eval reviews for the same reason: the model isn't broken, but the product is. In part 2 of our series Rise of the Agent Engineer, Hamel walks through why ambiguous inputs, generic metrics, and disconnected reviews make AI evaluations misleading, and how to fix them.

arize.comHamel Husain explains why AI evals fail before the evaluation beginsHamel Husain explains why ambiguous inputs, generic metrics, and disconnected review workflows can make AI evaluations misleading, and how developers can build a better process around real production data.

30 July

You can also read our writeup on our conversation with Hamel to learn why useful evals start before the metric: arize.com/blog/rise-o...

youtube.comStop Blaming the Model: Fixing the AI Product Bottleneck | Rise of the AI Engineer | Hamel HusainWhen an AI feature fails in production, developers almost always bl...

30 July

Want to watch the full interview? Check it out on our YouTube channel: www.youtube.com/watch?v=LGK...

29 July

What if your agents got better every time they failed? Today, we’re launching Signal. It continuously reviews production traces, finds issues, and turns them into an investigation with evidence, root cause, and a proposed fix. Your engineers decide what ships. arize.com/blog/from-s...

luma.comFrom Signal to PR: Evaluating and Shipping AI Agents That Work · LumaWhen an agent fails in production, finding the bad output is only the beginning. Teams still have to identify the pattern, trace it back to the underlying…

29 July

See Signal go from production failure to pull request in a live virtual event on July 30. qrs.ly/l5hg8lx

Want to master the full workflow of shipping reliable AI agents? Laurie's workshop from AI Engineer World's Fair, "Evaluating and Shipping AI Agents That Work," is now a free, self-paced course on Arize University. Earn a certificate you can share on LinkedIn by completing 13 episodes that cover:

→ Traces → Code evals vs. LLM judges → Datasets and experiments → Online evals and monitors → Closing the loop with coding agents Get started here: courses.arize.com/l/pdp/arize...

Two AI observability lessons from @bookingcom: - An agent latency spike came from a model running without the appropriate service tier. - Multi-turn eval scores fell because long URLs were being added back into the conversation history, causing the context to balloon.

The broader playbook on agent observability from @bookingcom: - Trace model calls, retrieval, tools, guardrails, and fallbacks - Monitor quality alongside latency and token usage - Pull real failures into datasets for experimentation - Centralize sampling, PII redaction, and telemetry routing

arize.comFrom traditional ML to AI agents: How Booking.com scales AI observability with ArizeBuilding an AI-native observability stack for agentic AI and traditional ML at Booking.com

Here's a practical look at how Booking.com approaches AI observability across agents and traditional ML with Arize AX: arize.com/blog/bookin...

Both problems became much easier to fix once the team could connect production signals to individual traces, configurations, and conversations.

arize.comArize AX - Arize AX DocsAI Engineering Platform

24 July

Claude Opus 5 is available in Arize AX, supported across Anthropic API, AWS Bedrock, and Vertex AI! Opus 5 brings major improvements for long-running agents in coding and professional work. Instrument, evaluate, and improve your agents with day 0 support for Anthropic's newest model.

23 July

Token price tells you what a model costs to call. It does not tell you what it costs to finish the job. Arize and @FireworksAI_HQ benchmarked 10 models across 2,400 agent runs, including Kimi K3.

23 July

In his tests, Arize's Head of DevRel @seldo found: • Kimi K3 nearly matched GPT-5.5 overall • Different models won on different task types • Routing improved cost and coverage • Retries and silent failures changed the economics completely

arize.comCost per successful task: Benchmarking Kimi K3, GPT-5.5, and 8 more AI modelsWe tested 10 open and closed models (including Kimi K3) across 2,400 agent runs, measuring pass rates, cost per successful task, and more.

23 July

Developers and product managers should optimize for cost per successful task, then use evals and traces to understand where each model belongs. Get the benchmark data: arize.com/blog/cost-p...

22 July

Gemini 3.6 Flash and Gemini 3.5 Flash-lite are now available in Arize AX! Both models are supported on direct Gemini and Vertex AI providers with configurable thinking levels, 65K max output tokens, and audio file input.