Grafana 11's New Provisioning Order Causes Folder UID Collisions
A recent change in Grafana version 11 has introduced a critical bug that breaks dashboard folder provisioning when alerts are configured. The issue stems from Grafana now provisioning alerts before dashboards. This shift in order leads to a conflict where Grafana creates a folder with a generated UID for alerts first, and then fails when it attempts to provision a dashboard folder with the same name and a pre-defined UID, resulting in an error: "a folder with the same name already exists in the current location".
The problem was first observed by users who were attempting to pin their dashboard folder UIDs to prevent deep links from breaking across redeploys. This is a common practice for ensuring dashboard links remain stable even after infrastructure changes or container restarts. However, upon restarting their Grafana containers after implementing this configuration, they encountered the aforementioned error, making their dashboards inaccessible through expected links.
The core of the issue lies in how Grafana handles UIDs for both alert rule folders and dashboard folders. Traditionally, when provisioning dashboards, a user might specify a folder name and a corresponding UID. This UID is crucial for stable linking. However, Grafana 11's new alert provisioning mechanism, which now runs earlier in the startup sequence, creates its own folder for alerts, often using a generated UID that can collide with a user-defined UID for a dashboard folder if the names are similar or identical. The error message, "a folder with the same name already exists," is misleading because the user has not manually created such a folder; Grafana itself created the conflicting entry during the alert provisioning phase.
Understanding the Collision Mechanism
To illustrate, consider a scenario where a user defines a dashboard folder named 'jo4' with a specific UID. In Grafana 11, if alert rules are also configured to be provisioned, Grafana might create an alert folder, also named 'jo4' (or a similarly named folder that triggers the conflict), before the dashboard provisioning process even begins. When the dashboard provisioning process then attempts to create its 'jo4' folder with the user-defined UID, it finds that a folder with the name 'jo4' already exists, albeit one created by the alert provisioning system, and throws the error.
This behavior is counterintuitive because users expect their dashboard provisioning configurations to take precedence or at least not conflict with system-generated folders for unrelated features like alerts. The problem is exacerbated by the fact that the error message does not clearly indicate that the conflict originates from the alert provisioning system.
The Impact on Users and Workflows
For organizations relying on Grafana for monitoring and observability, this bug can be a significant roadblock. Stable deep links to dashboards are essential for incident response, reporting, and general operational visibility. When these links break, it disrupts workflows, wastes valuable time during critical events, and can lead to confusion. Developers and operations teams may find themselves unable to access the dashboards they need, forcing them to manually recreate folders or reconfigure their Grafana setup, which is a time-consuming and error-prone process.
The reliance on provisioning is a best practice for managing Grafana configurations in a declarative, infrastructure-as-code manner. This bug directly undermines that practice, introducing manual intervention where automation is desired. It suggests a lack of coordination between the alert provisioning and dashboard provisioning modules within Grafana, or at least an oversight in how their respective UIDs and naming conventions interact during the startup sequence.
Potential Solutions and Workarounds
The most immediate workaround involves adjusting the naming conventions for either alert folders or dashboard folders to ensure they do not share names. For instance, users could prefix their dashboard folder names with something unique, like 'dashboards_jo4', while alert folders might be named 'alerts_jo4'. This separation prevents the direct name collision.
Another approach is to delay or disable alert provisioning temporarily until a fix is available. However, this is not a sustainable solution for teams that depend on automated alert rule management. Users could also consider reverting to a previous Grafana version that does not exhibit this behavior, though this means foregoing the latest features and security patches.
The Grafana team is aware of the issue and is working on a resolution. A potential fix involves reordering the provisioning steps or ensuring that generated UIDs for alert folders do not collide with user-defined UIDs for dashboard folders. Until a patch is officially released and deployed, users must carefully manage their naming conventions or accept the risk of broken links and manual intervention.
The Unanswered Question: Future-Proofing Provisioning
What remains unclear is whether this incident signals a broader architectural shift in Grafana's provisioning strategy. As Grafana adds more features that require their own provisioning mechanisms (like recording rules, data source provisioning, etc.), the potential for similar UID or naming conflicts increases. It raises the question of how Grafana will ensure robust, conflict-free provisioning across all its modules in the future. A more sophisticated approach to UID generation and management, perhaps a centralized provisioning orchestrator, might be necessary to prevent such issues from recurring.