The 'Golden Path' That Led to a Dead End: Platform Engineering War Story

We spent three months building the perfect CI/CD “golden path,” a textbook example of Platform Engineering: Building Golden Paths for Developer Productivity. Six months later, we discovered our top internal user was the platform team itself, force-merging updates while everyone else had quietly forked our code and run for the hills. This is the story of how our elegant solution became a dead end.

TL;DR: Our centrally managed CI/CD templates were technically sound but a developer experience nightmare, leading to near-zero adoption. This post details how we diagnosed the friction points causing this failure and pivoted from monolithic templates to a composable, self-service model. You’ll learn how to spot a failing “golden path” and fix it before it becomes a dead end.

What you’ll walk away with:

• A checklist to audit the adoption of your own internal platforms.
• The exact jq command we used to parse CI logs and find developer pain points.
• A “before and after” YAML diff showing the move from monolithic to composable pipelines.
• A mental model for treating your internal developer platform (IDP) as a product with real customers.

What Was Our ‘Golden Path’ Supposed to Do?

Our goal was to provide a fully managed CI/CD solution as part of our new Internal Developer Platform (IDP), a self-service toolkit meant to abstract away infrastructure complexity. We created a central repository with GitLab CI templates that handled everything: linting, unit tests, SAST scans, container builds, and deployments to Kubernetes. A developer only needed to include one file and set a few variables.

The initial implementation looked clean. A service’s .gitlab-ci.yml was just five lines:

include:
  - project: 'our-org/platform/ci-templates'
    ref: v1.2.3
    file: '/pipelines/node-service.yml'

variables:
  APP_PORT: 8080

On paper, it was perfect. We enforced security scanning, standardized deployments, and reduced cognitive load. We were building the foundation for what we hoped would become an AI-driven future of developer experience.

The process was rigid by design, enforcing compliance and best practices from a single source of truth.

graph TD
    Dev["Developer pushes code"]
    Dev --> CentralRepo["include: central-repo/template.yml"]
    CentralRepo --> Pipeline{Monolithic Template Runs}
    Pipeline -->|Pass| Deploy["Deploys to Staging"]
    Pipeline -->|Fail| Block["Blocks Merge"]

We launched it, onboarded the first few teams, and celebrated our new, standardized world.

The goal of a golden path isn’t just to enforce standards; it’s to make the right way the easiest way. If it’s not the path of least resistance, developers will pave their own.

Why Did Developers Stop Using Our CI Templates?

They stopped using them because debugging a failure was an exercise in frustration that killed all productivity. The abstraction that made the “happy path” so easy made the “unhappy path” completely opaque. When a pipeline failed, developers had no idea if it was their code, their configuration, or a bug in our centralized template.

We noticed the problem in our adoption metrics. After an initial spike, new service onboarding flatlined. Worse, we saw teams pinning to ancient versions of our templates or, more often, copying the entire YAML into their own repo just to make a small change.

Here are the warning signs we missed:

• A growing number of support requests asking “what does this CI error mean?”
• Teams pinning their include: ref to a version that is months old.
• Developers copying and pasting entire template files into their own repository.
• A low ratio of “feature” teams to “platform” teams opening PRs against the central template repository.
• An increase in CI pipeline duration as our monolithic template grew larger and ran steps services didn’t need.

The core issue was a lack of control. A team building one of the new flagship AI engineering projects needed to install a specific Python dependency for their build step. Our template didn’t allow it. Their only recourse was to eject from the system entirely.

If developers are forking your “don’t repeat yourself” platform code, it’s a critical sign that your abstractions are failing them.

How Did We Diagnose the Real Developer Experience Problem?

We diagnosed the problem by treating it like a production incident: we went straight to the logs. We used the GitLab API to pull the last 100 failed pipeline logs from across the organization and used a jq one-liner to find the most common failure reasons.

Here’s the command we ran to aggregate the failure stages:

# Assumes you've saved API output to gl_pipelines.json
cat gl_pipelines.json | jq -r '.[].failure_reason' | sort | uniq -c | sort -nr

The output was a gut punch.

     87 SCRIPT_FAILURE
     11 JOB_EXECUTION_TIMEOUT
      2 BRIDGE_UNREACHABLE

The SCRIPT_FAILURE was generic, but digging into the raw logs revealed the true culprit. The error messages were completely unhelpful because they were coming from deep inside our abstracted template logic. A developer would see something like this and have no context.

Click to see the full, painful error log

Running with gitlab-runner 16.5.0 (17825862)
  on green-runner-xyz-123
Resolving secrets
Preparing the environment
00:01
Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/our-org/some-service/.git/
...
Executing script...
$ /bin/bash /scripts/run_build.sh
ERROR: Missing required environment variable: CI_ARTIFACT_NAME
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

The developer’s code was fine. Their local tests passed. The problem was that our v1.2.0 template update added a new mandatory variable, CI_ARTIFACT_NAME, but the developer was still using a configuration for v1.1.0. The abstraction hid the breaking change, and the error message gave them nothing to work with.

Your platform’s error messages are a core part of its user interface. If they don’t point to a fix, they’re just noise.

What Did the Successful Pivot Actually Look Like?

We pivoted from a monolithic, inherited template to a set of small, composable, opt-in components. Instead of one giant node-service.yml, we offered test.yml, sast.yml, and build.yml. Developers could include only what they needed and, crucially, could override any part of the job.

Here is the before-and-after diff for a typical service’s .gitlab-ci.yml.

--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,7 +1,14 @@
-include:
-  - project: 'our-org/platform/ci-templates'
-    ref: v1.2.3
-    file: '/pipelines/node-service.yml'
+include:
+  - project: 'our-org/platform/ci-templates'
+    ref: v2.0.0
+    file: '/jobs/test.yml' # Opt-in to unit tests
+  - project: 'our-org/platform/ci-templates'
+    ref: v2.0.0
+    file: '/jobs/build.yml' # Opt-in to container builds
 
 variables:
   APP_PORT: 8080
+
+# Users can now easily override steps
+unit-tests:
+  script:
+    - echo "Running custom test command first!"
+    - !reference [.unit-tests, script]

This approach shifted control back to the developer. We still provided the “golden path” components, but they were no longer a black box. The platform team’s job changed from enforcing a single process to curating a library of high-quality, reusable CI jobs.

The new workflow emphasized choice and transparency.

graph TD
    subgraph Developer's Repo
        direction LR
        DevCI[".gitlab-ci.yml"]
    end

    subgraph Platform Repo
        direction LR
        Test["test.yml"]
        Build["build.yml"]
        Deploy["deploy.yml"]
    end

    DevCI -- "include:" --> Test
    DevCI -- "include:" --> Build
    DevCI -- "include:" --> Deploy

    Test & Build & Deploy --> Pipeline["Pipeline Runs Composed Jobs"]

Default to composable, opt-in components over monolithic, inherited templates for your internal platform. Let developers build their own path from your golden bricks.

Bottom Line

An Internal Developer Platform is a product, and your developers are its customers. Success isn’t measured by the elegance of your solution or the number of standards you enforce. It’s measured by voluntary adoption. If you have to force teams to use your platform, you’ve already failed.

Stop building rigid, top-down “golden paths.” Instead, provide a toolkit of well-documented, reliable, and composable components that make developers’ lives genuinely easier. When the path of least resistance is also the path of best practice, you’ll know you’ve succeeded.

FAQ

What is the difference between a golden path and a paved road?

A “golden path” is a single, opinionated, and often mandatory route for a specific task, like deploying a service. A “paved road” is a supported and smoothed-out path, but it’s one of several options; developers can go “off-road” if they need to, accepting the trade-offs. We pivoted from a golden path to a paved road with well-maintained escape hatches.

How do you measure the success of an Internal Developer Platform?

Measure success with product metrics, not compliance checklists. Track voluntary adoption rate, time-to-first-pipeline-green, developer satisfaction scores (NPS), and the number of support tickets related to debugging your tooling. A successful IDP reduces support load and increases deployment frequency.

What are the first components to build in an IDP?

Start with the highest-friction part of your current developer workflow. For most organizations, this is CI/CD and provisioning a new “hello world” service. Focus on a starter template (cookiecutter, etc.) and a simple, composable pipeline for testing and deploying it.

Is Backstage a full Internal Developer Platform?

No, Backstage is a service catalog and a framework for building an IDP, but it’s not an IDP out of the box. You must integrate your own tooling (CI/CD, observability, security scanners) into it. It provides the “frontend” or portal, but you still need to build the “backend” platform capabilities.

How do you get feedback from developers on an internal platform?

Don’t just rely on surveys. Embed yourself with a feature team for a week. Hold office hours where developers can bring their specific problems. Most importantly, analyze the “negative signals”: what tools are they not using, what policies are they bypassing, and what internal libraries are they forking?

Further Reading

• https://platformengineering.org/blog/what-is-platform-engineering
• https://backstage.io/
• https://cloud.google.com/blog/products/devops-sre/a-guide-to-internal-developer-platforms

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.