The SRE Roadblock: Good Plans, No Action
In the autumn of 2019, a mid-size financial services firm hired its first dedicated Site Reliability Engineer (SRE). Within six weeks, this practitioner had produced a comprehensive SRE adoption roadmap. It included detailed proposals for error budget policies, Service Level Objective (SLO) frameworks, on-call rotation designs, postmortem processes, and toil reduction programs. The documents were meticulously researched and fully aligned with established Google SRE practices. The SRE presented this plan to the VP of Engineering in a forty-minute meeting.
The VP acknowledged the effort, and the roadmap was filed away. Nothing happened. The proposed changes, despite their thoroughness and alignment with best practices, failed to gain traction. This scenario is not unique; it highlights a common challenge: organizational resistance to adopting new reliability paradigms, even when presented by a dedicated expert.
The Production Incident Catalyst
Eighteen months later, a significant production incident struck the firm. This incident, critically, had been predicted three hours in advance by the very observability stack the SRE had been developing. Following the incident, the SRE authored the postmortem, facilitated the discussion, and diligently tracked all action items to completion. It was only after this event, which underscored the SRE's foresight and the value of their work, that the same VP of Engineering approached the SRE. The question posed was simple but profound: "Can you help us do this for all of our services?"
The stark contrast between the initial polite dismissal of the roadmap and the subsequent urgent request for widespread SRE adoption reveals a critical insight: technical solutions, however well-designed, often falter without the right organizational context and timing. The incident acted as an undeniable, high-stakes demonstration of the SRE's value, forcing a re-evaluation of the organization's priorities and risk tolerance.
Understanding Organizational Inertia
Why does such a disconnect occur? Several factors contribute to organizational inertia when it comes to adopting SRE principles:
- Fear of Change: Introducing SRE often means fundamentally altering established workflows, team structures, and responsibilities. Developers accustomed to a certain release cadence might resist processes that introduce more stringent reliability checks. Operations teams might feel threatened by the shift in ownership.
- Lack of Perceived Urgency: Without a visible, impactful event that directly links current practices to negative business outcomes, the need for SRE can seem abstract or theoretical. The financial services firm, prior to its major incident, likely viewed its existing systems as "good enough."
- Misunderstanding of SRE Value: SRE is often perceived as an added cost or a bureaucratic hurdle rather than a strategic investment in stability, scalability, and ultimately, business continuity. The initial presentation likely didn't sufficiently bridge the gap between technical SRE concepts and tangible business impact in the VP's eyes.
- Cultural Mismatch: SRE thrives in cultures that embrace blameless postmortems, data-driven decision-making, and shared ownership of reliability. Organizations with a culture of finger-pointing or resistance to transparency will struggle to adopt SRE effectively.
- Resource Constraints: Even with a solid plan, implementing SRE requires dedicated time, tooling, and personnel. If teams are already stretched thin with feature development or firefighting, carving out the necessary resources for reliability initiatives becomes a challenge.
A Phased Influence Playbook for Practitioners
The SRE practitioner in this case study, though initially unsuccessful in broad adoption, demonstrated resilience. Their experience suggests a phased approach is often more effective than a single, comprehensive presentation. This playbook draws on that observation:
Phase 1: Build Trust and Demonstrate Value (The "Quiet" Phase)
Instead of presenting a grand roadmap upfront, focus on small, impactful wins within your immediate sphere of influence. This involves:
- Targeted Observability: Deploy monitoring and alerting for a critical service or application. Use this to proactively identify and fix issues before they impact users.
- Blameless Postmortems: When incidents inevitably occur (even small ones), meticulously facilitate blameless postmortems. Focus on systemic issues and actionable improvements, not individual blame. Document these thoroughly.
- Toil Reduction Pilots: Identify a repetitive, manual task that consumes significant engineer time. Automate it and document the time saved and the reduction in error.
- Data Collection: Quietly gather data on system performance, incident frequency, MTTR (Mean Time To Recover), and the cost of downtime. This data will be crucial later.
During this phase, your goal is to become known as the person who solves problems and improves stability, not the person who dictates new processes. Build relationships with key engineers and team leads. Show, don't just tell.
Phase 2: Seed the Concepts (The "Whisper" Phase)
Once a track record of success is established, begin to gently introduce SRE concepts in relevant contexts:
- Integrate into Existing Meetings: Bring up SLOs when discussing feature success metrics. Mention error budgets when planning sprints. Discuss toil reduction as a way to free up developer time for innovation.
- Share Success Stories: Casually share the time saved by automation, or how a proactive fix prevented a potential outage. Frame these in business terms (e.g., "We saved X hours this week by automating Y").
- Informal Education: Offer to run short, informal brown-bag sessions on specific SRE topics (e.g., "What is an SLO and why should we care?", "Introduction to Blameless Postmortems"). Keep them practical and relatable.
- Identify Champions: Look for engineers or managers who are receptive to these ideas and encourage them to experiment or adopt specific practices within their teams.
The key here is to make SRE concepts seem like natural extensions of good engineering practice, not a radical overhaul.
Phase 3: Formalize and Scale (The "Advocate" Phase)
This phase is often triggered by a significant event, like the production incident in the case study, or when a critical mass of internal support has been built. When the demand arises, you are ready:
- Leverage the Incident: If an incident prompted the demand, use its data and impact as the primary justification. Show how SRE practices could have prevented or mitigated it.
- Present the Phased Roadmap: Now, present the comprehensive roadmap, but frame it as the logical next step based on demonstrated value and growing internal interest. Highlight the successes from Phase 1 and 2.
- Develop Policies and Standards: Work with leadership to formalize SLOs, error budgets, and incident response procedures as organizational standards.
- Secure Resources: Advocate for the necessary tooling, training, and headcount to support a wider SRE rollout.
- Continuous Improvement: Establish mechanisms for ongoing feedback and iteration on SRE processes.
This structured approach, moving from quiet demonstration to active advocacy, acknowledges that organizational change is a process, not an event. It requires patience, strategic communication, and a deep understanding of how to influence without explicit authority, especially in the early stages.
Referenced Sources
- verified
