LangSmith adds alerts for errors, latency and LLM quality issues

LangChain has launched Alerts in LangSmith, giving developers a way to monitor LLM applications and agents for production problems in real time. The feature can flag errors, latency changes and drops in feedback scores.
- LangSmith Alerts monitor error count and rate, average latency and average feedback score.
- Developers can filter alerts by model, tool call or run type.
- Alerts support 5 or 15 minute aggregation windows and configurable thresholds.
- Notifications can be sent through PagerDuty or a custom webhook.
- LangChain plans alerts for run count, token usage, relative changes and custom time windows.
Alerts target reliability and output quality
LangChain says LangSmith Alerts are designed to help developers catch failures before they affect users. The feature is available for applications that send production traces to LangSmith, while developers new to tracing can begin with tracing in LangSmith.
Alerts can be based on three groups of metrics: error count and rate, average latency, and average feedback score. These metrics address both operational failures and changes in the quality of an LLM application's output.
Filters and time windows narrow the signal
Developers can apply filters to focus alerts on particular subsets of runs. LangChain gives model, tool call and run type as examples of filtering options.
Each alert can use an aggregation window of 5 or 15 minutes, together with a threshold that controls its sensitivity. This allows teams to tune alerts around the parts of an application they want to monitor, rather than treating every run as the same.
LangChain also supports sending alerts to PagerDuty or connecting them to a custom webhook. The company gives sending notifications directly to a Slack channel as an example of how a webhook could be used.
Why monitoring is difficult for LLM applications
According to LangChain, agentic applications depend on external services including model providers, APIs, web search services and databases. Outages, rate limits and increased latency affecting those dependencies can reduce the user experience, making early detection important.
Quality problems create another challenge. Changes to prompts, models or inputs can produce unexpected results, and prompts that work well in controlled evaluations can regress in real-world use because user interactions differ. LangChain says feedback scores from user input or online evaluations can provide an early warning of these quality drops.
More alert types are planned
LangChain says it plans to add alerts for run count and LLM token usage. It also plans change alerts, which would allow developers to set a relative value, such as an alert when latency spikes 25%, as well as alerts covering custom time windows.
For users of LangSmith, the launch adds a monitoring layer that can connect application metrics with existing incident workflows. The planned additions suggest the company intends to expand beyond fixed thresholds and the current error, latency and feedback measures.
LangChain: Catch production failures early with LangSmith Alerts


