TL;DR — Key Takeaways
-
Shared platforms need clear accountability. Every service should have one accountable owner, even when multiple teams support its operation.
-
Incident leadership shouldn’t depend on finding the culprit. A designated incident commander can coordinate recovery while specialists investigate the root cause.
-
Recovery isn’t the end of the job. Postmortems, updated runbooks and response drills are essential to prevent confusion during future outages.
When a developer platform fails, infrastructure, DevOps and site reliability engineering teams may each own part of the affected system. That division can bring specialists into an investigation quickly. It can also leave developers waiting while teams debate who should respond, which service failed first and who has authority to coordinate recovery.
The challenge is to make shared delivery compatible with clear accountability. Platform teams need an ownership model that identifies both who supports individual services and who directs the response when an outage crosses those boundaries.
“Lack of clear ownership often boils down to situations where the responsibilities are either unclear and overlapping, or they are just not documented anywhere,” says Mykhailo Voitovych, head of engineering at Sombra.
He said those weaknesses become harder to address when organizations neither test incident response procedures nor conduct postmortems after major failures. Responsibilities can appear settled until an outage exposes conflicting assumptions about who does what.
Give Each Service an Accountable Owner
A platform may depend on several teams without requiring developers to understand its entire organizational structure to get help. The support process should make the first point of contact clear, even when resolving the problem requires several specialists.
“Every service needs to have a single accountable owner, even when it is being operated by more than one team,” Voitovych says.
He recommended service catalogs, ownership matrices and well-defined escalation paths to make those responsibilities visible. Together, they should help developers identify where to report a failure and help responders determine whom to involve next.
On-call procedures should identify who receives the initial incident report and when to involve other teams. The service owner should coordinate those handoffs so developers can report a problem without having to assemble the response team themselves.
Separate Response Leadership From Root Cause
When the cause of an outage remains uncertain, assigning leadership to whichever team initially looks responsible can make coordination depend on an unproven diagnosis.
Voitovych instead recommended an assigned incident commander to manage the response while technical teams investigate. The commander establishes priorities and coordinates decisions without requiring responders to settle responsibility for the underlying failure first.
“The incident commander communicates priorities and defines decisions while the technical team investigates the root cause in parallel,” he says.
This gives a cross-team investigation a common direction. Specialists can examine their respective systems while the response has an identifiable leader responsible for coordinating their work.
Shared ownership is useful in complex outages because networking, application and observability expertise can become available quickly, Voitovych says. But bringing more people into the response does not resolve uncertainty about decision-making authority.
Without that authority, he warned, teams can duplicate investigations and delay action because nobody has clear ownership or responsibility for deciding how to mitigate the failure.
Test the Ownership Model After Recovery
Restoring service does not establish that the coordination process worked. A retrospective needs to examine where responsibilities were unclear, where communication broke down and which procedures slowed the response.
“A documented retrospective needs to be prepared to uncover gaps in the assignment of responsibilities, as well as communication failures and process issues for every major incident that occurs,” Voitovych says.
Those findings should lead to changes in runbooks, escalation protocols and testing practices, he adds Otherwise, the same ambiguity can survive into the next outage despite a documented explanation of what happened.
The final step is to exercise the revised process. A clearer ownership matrix has limited value if responders still cannot use it to coordinate an investigation under pressure.
“Every one of these measures must be verified through drills, particularly before the next incident occurs,” Voitovych says.