# From Spikes to Stories: AI-Augmented Troubleshooting in the Network Wild

Shraddha Yeole, Cisco ThousandEyes | MLOps Community | 11:49

Source: https://www.youtube.com/watch?v=dU72QdPkNpM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/from-spikes-to-stories-ai-augmented-troubleshooting-in-the-network-wild
Published: 2025-08-27
Tags: agents, evals, observability, prompt-engineering

## TL;DR
- ThousandEyes can use an AI agent to combine application, network, and routing telemetry into a fault-domain assessment.
- The agent uses Semantic Kernel, an LLM, preprocessing, and stored chat history to turn a dashboard investigation into an explanation with follow-up prompts.
- Shraddha Yeole evaluates the agent with subject-matter experts, known outage data, human review, and metrics such as hallucination rate, answer relevance, and prompt alignment.

## Summary
Shraddha Yeole describes how Cisco ThousandEyes is moving from dashboards that present telemetry to an AI assistant that interprets it. She uses a Spotify availability problem during a Google Cloud outage as an example. An operator would usually inspect HTTP, network, and BGP routing views, then combine evidence across many metrics. The assistant instead correlates those layers and produces a fault-domain assessment, with prompts for drilling into a specific layer. The production design routes requests through Semantic Kernel to a ThousandEyes agent, which preprocesses API data before calling an LLM and stores chat history. Yeole explains the prompting work behind the system, including role prompts, step-by-step reasoning, examples, XML tags, and JSON schemas. She is direct about the hard parts: telemetry is large and complex, raw API responses create excessive token use, and the data needs baselines and enrichment. Evaluation combines expert review, outage ground truth, and LLM-based checks.

## Key ideas
### ThousandEyes traces a digital experience across systems outside the customer network
[00:31](https://www.youtube.com/watch?v=dU72QdPkNpM&t=31s)
Yeole frames troubleshooting around a Spotify stream that buffers or fails to load. The cause could be local Wi-Fi, an internet service provider, a cloud provider, or an intermediate transit network. ThousandEyes provides path traces across the application server, provider, cloud platform, and internet backbone. That visibility lets an assistant investigate environments the operator does not own as well as the customer's network.

### Dashboards leave operators to assemble the fault domain by hand
[02:03](https://www.youtube.com/watch?v=dU72QdPkNpM&t=123s)
Yeole describes a Google Cloud outage affecting Spotify availability. A ThousandEyes user would click through HTTP, network, and BGP routing layers and inspect many metrics to decide whether the application or network caused the problem. She identifies cognitive overload, lost context during investigation, and difficulty interpreting the views for people without deep network operations experience.

### The assistant correlates layered telemetry into an explanation
[03:51](https://www.youtube.com/watch?v=dU72QdPkNpM&t=231s)
Clicking an explain button starts an investigation across the available layers. In the example, the assistant identifies an HTTP-level problem, lower availability than the baseline, and impact across multiple agent locations. It also connects the affected phase with different error codes. Users can open follow-up prompts to ask about a particular layer. The intended change is from presenting data to interpreting it and isolating a fault faster.

### The production agent preprocesses data before calling the language model
[05:38](https://www.youtube.com/watch?v=dU72QdPkNpM&t=338s)
The request from the AI assistant passes through Semantic Kernel, an open-source framework from Microsoft, and then reaches the ThousandEyes agent. The agent calls API endpoints, performs preprocessing before sending information to the LLM, receives the response, and displays it. Chat history is stored to support faster explanations. The architecture keeps the model call behind an application layer that prepares the telemetry.

### Prompt design combines network expertise with controlled output structure
[06:42](https://www.youtube.com/watch?v=dU72QdPkNpM&t=402s)
The team uses role prompting that asks the model to act as a network engineer. Yeole says they worked with solution analysts and subject-matter experts to develop step-by-step reasoning prompts. Few-shot examples show the expected input and output, including links that users can open quickly. XML tags for instructions, examples, and output format, together with a JSON schema in the system prompt, give the response a structure the product can consume.

### Evaluation uses outage evidence and several response checks
[08:11](https://www.youtube.com/watch?v=dU72QdPkNpM&t=491s)
The team began with manual evaluation by subject-matter experts. For the Spotify case, a known outage and its published analysis provided reference material for comparing agent responses. The team also uses an LLM to assess hallucination rate, answer relevance, prompt alignment, and whether the supplied data is appropriate. Human review remains part of the evaluation loop rather than treating model scoring as the only check.

### Telemetry needs baselines and enrichment before it is useful to an LLM
[09:41](https://www.youtube.com/watch?v=dU72QdPkNpM&t=581s)
Application and network telemetry creates both volume and complexity. Yeole says the team preprocesses and enriches the data, including HTTP timing and latency measures, and uses baselining or predefined machine-learning strategies before sending it to the model. Passing raw API data directly produced high token counts and poor correlations. The model needs a more focused representation of the underlying measurements.

### The next version may learn from network-specific outage data
[10:46](https://www.youtube.com/watch?v=dU72QdPkNpM&t=646s)
Yeole says the team is adding continuous learning and a human feedback loop built around outage ground truth and subject-matter review. The current system uses an Anthropic model. Future work includes fine-tuning a model on network data, adding more training examples, and hosting that model within the company's environment.

## Notable quotes
- "Instead of manually piecing together clues, imagine asking, "What's wrong here?" and receiving clear, contextual insights." (01:14)
- "So this is where with the AI agent in production, we are trying to shift from data presentation to the intelligent interpretation mode." (05:16)
- "If you provide the raw API data as is to LLM, there were few issues leading to high token counts and poor correlations." (10:23)
- "We have a human in feedback loop to evaluate the responses." (08:35)

## Tools & references mentioned
- Cisco ThousandEyes
- Spotify
- Google Cloud
- Semantic Kernel
- Microsoft
- Anthropic Claude 3.7
- LLM
- XML
- JSON schema
- FastAPI
- Pydantic

## Who should watch
- You operate network or application observability systems and want an assistant to connect evidence across several telemetry layers.
- Your team is building an LLM feature over complex operational data and needs practical guidance on preprocessing, prompt structure, and evaluation.
- You need to explain an outage to engineers who may not have deep network operations experience.

## Related talks

- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [Logs Are All You Need: Rethinking Observability with AI Agents](https://mlopstalks.com/talks/logs-are-all-you-need-rethinking-observability-with-ai-agents) (Sherwood Callaway, Sazabi, 46:40)
- [How AI Agents Are Transforming Data Analytics](https://mlopstalks.com/talks/how-ai-agents-are-transforming-data-analytics) (Ines Chami, Numbers Station, 24:26)
- [The Science of Improving AI Agents](https://mlopstalks.com/talks/the-science-of-improving-ai-agents) (, 26:43)
- [Evaluating AI Agents: Why It Matters and How We Do It](https://mlopstalks.com/talks/evaluating-ai-agents-why-it-matters-and-how-we-do-it) (Annie Condon & Jeff Groom, Acre Security, 13:27)