Platform Engineering and the Challenge of Tribal Knowledge

The most dangerous dependency in a platform may be the engineer nobody can operate without.

Critical knowledge about production quirks, service relationships, configuration exceptions and recovery procedures often remains concentrated among a few senior engineers. When one of those engineers is unavailable, an otherwise manageable incident can become a prolonged outage while the team reconstructs information the organization never formally captured.

“Sometimes, all it took was for the one guy who knew how something worked to be sick on the same day an unfortunate issue occurred,” Dyna Software CEO and co-founder Ron Browning says. “What could have been an unseen and unfelt outage suddenly had a big bright light shone on it, a noticeable $100,000 price tag attached to it, and every executive talking about it.”

Where Tribal Knowledge Becomes an Operational Risk

The problem is particularly acute during incidents, production changes, vulnerability remediation and compliance work. Engineers need more than technical instructions in those situations; they need to understand why systems were configured a certain way and what else a change could affect.

“Tribal knowledge creates the greatest operational risk where speed, consistency, and accountability matter,” says Perforce Puppet senior technical marketer and evangelist Robin Tatam.

He cautions that if only a few experienced engineers understand why a configuration exists, which dependencies a change might affect, or how to restore a service quickly and safely, those individuals become part of the platform’s critical infrastructure.”

The growing use of AI in development and infrastructure creation may increase the risk by accelerating the volume and pace of changes. Platform teams that already struggle to document existing dependencies could find that their environments evolve faster than engineers can transfer the knowledge needed to operate them.

Infrastructure as code does not eliminate the problem if the deployed environment has drifted from the repository or contains manual changes that were never recorded.

“The greatest operational risk within platform teams is in the gap between what the code says and what is running in the cloud,” says env zero CEO Steve Corndell. “The IaC repo describes intent. The cloud contains emergency fixes, manual changes, and resources built by engineers who have since left.”

Hidden ownership creates another point of failure: If teams must ask around to determine who is responsible for a resource, the organization does not have a dependable system of record.

“If the only way to find out who owns a resource is to ask a person, tribal knowledge is load-bearing infrastructure,” Corndell says.

More Documentation Is Not the Answer

Organizations have attempted to address tribal knowledge through wikis and documentation repositories for decades. These initiatives often begin with strong participation but deteriorate when ownership is unclear and updates are disconnected from daily work.

“You stand up a wiki, everybody contributes to it for a while, and before long nobody knows what is still accurate,” Browning says. “Eventually, people stop trusting it and go back to asking the one person who knows.”

Instead of launching a separate documentation initiative, platform leaders should capture expertise when it is being applied. Incidents, architectural decisions, peer reviews, exceptions and recurring service requests all provide opportunities to record why a decision was made and convert it into something reusable.

“Teams should capture knowledge as close as possible to the point where it is applied,” Tatam says.

He adds that while traditional documentation remains useful for explaining intent, architecture, and unusual decisions, information determining how systems are managed should be expressed through version-controlled templates and translated into policy as code, automated workflows, and validation of outcomes.

This separates information that machines can continuously derive from the human reasoning that must be recorded. Ownership, dependencies, configuration state and change history should come from systems rather than manually maintained documents.

“The rule I would offer: Anything a system can derive, never write down,” Corndell says. “Write down only what a machine cannot infer, which is reasoning.”

AI may help identify stale information, extract knowledge from incidents and propose updates, but it does not remove the need for ownership and human review. Browning says knowledge-management efforts work only when they are integrated with project controls, operational validation and employees’ existing responsibilities.

“The mechanisms have to be practical and properly woven into process and expectations,” Browning says. “If you attempt to do a specific dedicated process or effort of activities on the side that essentially creates added and perceived ‘new’ work, it will fail.”

Turn Incident Knowledge into Platform Capabilities

Service catalogs, runbooks, incident reviews and automation each contribute to knowledge sharing, but simply creating those artifacts does not ensure engineers will use or trust them.

Service catalogs can establish ownership and make approved capabilities discoverable. Runbooks provide responses to known conditions, while incident reviews reveal undocumented assumptions and dependencies. Automation turns repeatable knowledge into controls and workflows that do not depend on an expert’s availability.

“If an incident review identifies a recurring configuration problem, the outcome should not be limited to a new paragraph in a runbook,” Tatam says. “The team should ask whether the platform can detect the condition, prevent the unsafe change, or execute a verified remediation automatically.”

Incident reviews therefore need to produce operational changes, not merely action-item lists that disappear after the meeting.

“The review should produce a catalog entry, a policy, or an automation,” Corndell says. “Something that outlives the meeting.”

Runbooks still have value, particularly for less predictable procedures that require human judgment. However, repeatedly following the same manual instructions suggests that the platform team has identified an opportunity for automation without acting on it.

“A runbook you’re still following by hand a year later isn’t knowledge capture, it’s a failure to automate something that should have been,” Corndell says.

Measure Whether the Dependency Is Declining

Platform leaders should avoid measuring knowledge-sharing programs solely through the number of documents created or training sessions completed. Those activities do not demonstrate that employees can safely operate a system without the original expert.

More useful indicators include escalation concentration, repeat incidents, first-contact resolution, recovery times and the frequency with which work stalls while waiting for a particular engineer. Teams should also examine whether incident resolution times vary significantly depending on who is available.

“Recurring incidents are especially revealing,” Tatam says. “If the same engineers repeatedly diagnose or remediate the same type of problem, the organization may be resolving symptoms without translating the learnings into a shared resource.