TL;DR: Microsoft Foundry only delivers its full observability and evaluation feature set if you build your agents natively inside Foundry. If you're running LangGraph — or any non-Foundry framework — on Azure, you can stand up an equally powerful and more flexible pipeline yourself using OpenTelemetry, the azure.ai.evaluation library, and Grafana on top of Application Insights. Here's how it works, and why it matters for cloud cost, reliability, and hallucination risk.
Why agentic AI systems need observability
I was recently helping build an Agentic AI system for lead generation. The system uses LangGraph for control flow, Anthropic's Claude API as the underlying model, and is hosted as a Microsoft Azure Container App with logs shipping to App Insights. Pretty standard modern agentic setup.
It didn't take long before a few important questions started nagging at me. Was the model hallucinating, or was it actually producing good, reliable data? How often was it having to loop back through the Reflection pattern, Agentic AI's built-in quality control mechanism, before settling on a good answer? And what was the real token cost of a typical response across the two design patterns the system used, the Reflection pattern and the ReAct pattern?
For an enterprise running these systems in production, those aren't academic questions. They map directly to cloud spend, output reliability, and the risk of putting hallucinated data in front of a customer or a sales team. Observability is what turns those risks from invisible to manageable — and it's usually the piece teams bolt on last, after something has already gone wrong.
If you're not familiar with these two patterns, here is a quick summary. The Reflection pattern is where the agent generates an output, critiques it, and then revises it, a quality layer. The ReAct pattern is where the agent Reasons about what to do, Acts by calling a tool, observes the result, and iterates over itself until it has solved the problem. Both are powerful. Both have meaningful cost and quality implications you need to be able to see.
Those questions are what pushed me to figure out the right observability story for this system.
First stop: evaluating Microsoft Foundry for agentic AI observability
Naturally, the first place I looked at was Microsoft Foundry. On paper it looked to be exactly what I needed. It advertises continuous evaluation against a range of built-in evaluators, things like violence, sexual content, relevance, and hallucinations. It also promises log traces with span tags that capture the full request and response chain across different agent calls, plus token usage visibility. That's a solid checklist.
The reality was a different story.
What I found out is that Foundry's full feature set only works if you build your agents and control flow through Microsoft Foundry itself. Since this system is built on LangGraph, Foundry's capabilities dropped off a cliff. What you're left with is the ability to view LLM trace logs and pull data from App Insights using KQL queries, then manually run those prompt and response pairs through Foundry's evaluators. Manually. One at a time. There is no automation, no continuous evaluation loop, nothing that runs on its own.
If I wanted the real Foundry experience, I would have had to rip out LangGraph and rebuild the entire control flow using Foundry's native agent framework. That wasn't happening. So, I went in a different direction.
Getting LangGraph traces into Azure Application Insights
Before building anything custom, I needed to make sure LangGraph's trace data was actually flowing into App Insights. This turned out to be straightforward using OpenTelemetry and Azure Monitor.
LangchainInstrumentor hooks into LangGraph and LangChain and captures trace data automatically. It captures span tags, agent calls, prompts, responses, and the works. configure_azure_monitor then ships all of that to App Insights using your connection string. Two library calls and your traces are flowing. With that in place, I had the raw material to work with.
Building an automated evaluation pipeline with azure.ai.evaluation
Here is where it got interesting. What I discovered is that Microsoft ships an evaluation library as part of the Azure AI SDK:
from azure.ai.evaluation import ...
This library gives you similar evaluators that Foundry uses such as relevance, coherence, fluency, groundedness, and others, but you can run them yourself, on your own schedule, against whatever data you want. Foundry did not have to be involved at all.
So I built a custom Python application and deployed it to Azure Container Apps as a job running on a cron schedule every hour. The job does the following:
- Executes a KQL query against App Insights to pull all LLM prompts and their responses from the last hour.
- Pass those prompt and response pairs through the evaluators from azure.ai.evaluation along with a connection to an LLM API of your choice and return an evaluation score.
- Sends the evaluation scores back to App Insights as custom telemetry.
That's it. With that loop running every hour, I had a continuous, automated evaluation pipeline that did everything Foundry advertised, without touching Foundry or changing a single line of the application agent logic.
Visualizing agentic AI metrics in Azure Managed Grafana
With evaluation data landing in App Insights, I needed a good way to visualize it. Azure has a built-in visualization tool called Workbooks that lives inside App Insights and is solid for ad hoc analysis. But I ended up going with Azure Managed Grafana for the main dashboards. The visualizations are significantly better, and more importantly, Grafana supports automatic refresh on a configurable interval, so the dashboard stays live without anyone touching it.
Both tools query App Insights the same way, KQL queries, so building in either one is essentially the same exercise. Here is what I ended up with on the Grafana dashboard.
7-Day Rolling Average Evaluation Score Tracks Relevance, Coherence, and Fluency over time as a rolling average on a scale of 1 to 5. Across the first week of data, Relevance came in at a mean of 4.78, Coherence at 4.63, and Fluency at 4.0. This panel gives you a quick read on whether output quality is trending up or down over time. A sudden dip here is your early warning signal that something changed such as a prompt update, a model change, or a new category of inputs the system wasn't handling well.
Daily Token Usage breaks down input and output tokens per day. The mean input was significantly more than the output. That asymmetry is worth staring at for a moment. Input tokens are more than 100 times the output. That is the reality of agentic systems. The context window gets loaded up with instructions, examples, tool definitions, retrieved data, and prior conversations. This panel is essential for cost forecasting and spotting when your prompts are starting to bloat.
Worst-Scoring Traces is probably the most immediately useful panel day to day. It surfaces the lowest-scoring LLM responses from the last seven days as a table with the timestamp, the company the response was about, which evaluator flagged it, the score, and the rationale the evaluator generated for why. For example, one entry showed a response about Google scoring a 3 on Relevance with the note that while it provided a comprehensive company overview, it lacked specific information on key executives. That kind of actionable detail tells you exactly where to focus on prompt improvements.
Validation Retry Rate tracks how many times the Reflection pattern loop had to retry before the agent was satisfied with its output. This is one of my favorite panels because it makes visible something that was previously invisible. A spike in retry rate tells you the agent is working harder than usual. Maybe the inputs are noisier, maybe the prompt needs tightening, or maybe the model needs a clearer definition of what "good enough" looks like for that task.
Validate Average Confidence tracks the average confidence score across evaluations over time, with separate lines for each evaluator type. Values were hovering consistently between 0.80 and 0.90, which is a healthy range. This panel is less about day-to-day debugging and more about long-term drift detection. If that average starts sliding over weeks, something in the system is degrading.
Beyond those five panels, I also built charts for tool usage across the ReAct pattern loops, per-agent cost per trace, and isolated evaluation scores for traces that went through the Reflection loop specifically.
Conclusion: you don't need a platform to observe your agents
Microsoft Foundry is worth keeping an eye on as it matures, but right now it is a walled garden. If you are not building your agents through Foundry, you are not getting the experience it advertises.
The good news is that with LangchainInstrumentor, azure.ai.evaluation, and either Workbooks or Grafana on top of App Insights, you can build an observability story that is just as powerful, and arguably more flexible, because you own every layer of it. The cron-based evaluation job in particular was a surprisingly simple solution that unlocked a lot. A few dozen lines of Python running on a schedule gave me a continuous, automated evaluation that I can query and visualize however I want.
For any team building agentic systems in Azure, I would start here before going anywhere else. Get your traces into App Insights, get your evaluations running on a schedule, and build the dashboards that answer the questions your team has. You don't need a platform to do it for you.
Callibrity helps enterprise teams build and operationalize agentic AI on Azure — including the observability and evaluation layer that keeps these systems reliable, cost-predictable, and trustworthy in production.
If your team is standing up agents and wants to see what's really happening inside them, let's talk.
.svg){kind=small}
Article