“Harness” is still a new and loosely defined term. Some people use it to describe the system around a coding agent; others use it for the tools and checks around a repository. Here is how I define harness in two distinct ways:

• A coding-agent harness is the deterministic logic around the LLM: the planning loop, tool calls, and memory architecture in systems like Claude Code and Codex CLI.

• A repo harness is the collection of tools and automated checks that verifies a repository’s correctness—including its behavior, architecture, contracts, and engineering standards.

This article is about the repo harness.

In Beyond 2x AI productivity boost: Architecture is the missing piece, I described an architecture organized across system, application, and module boundaries. A repo harness makes that architecture enforceable: it turns those boundaries, public interfaces, contracts, and other rules into checks that reject violations before the change is accepted.

How a repo harness verifies correctness

A repo harness is there to verify repository correctness. So what does correctness mean? At a high level, it has two dimensions: behavioral correctness and repository conformance.

Behavioral correctness asks whether the software does what it is supposed to do. Teams build confidence in behavioral correctness through the familiar automated testing pyramid: many unit tests, fewer service and integration tests, and a small number of end-to-end tests. Testing strategy is not a new topic, and it is well covered elsewhere, so I will not revisit it here. The important point for this discussion is that the automated test suite is an important part of the repo harness.

Repository conformance—or repo conformance—asks whether a change preserves the intended shape and constraints of the repository: module boundaries, dependency direction, ownership, public interfaces, contracts, type safety, security requirements, and project-specific rules. Conformance violations can leave the software behaving correctly today while making the codebase harder and less safe to change tomorrow. This is the less familiar part of the harness—and where this post will focus.

AI is really good at patterns. Give it a good pattern and it will create more of it. Give it just one bad pattern and it will proliferate through the codebase. This is why it’s so important to ensure repo conformance upfront.

The checks vary, but reliable verification requires the same basic ingredients.

A precise rule. “Keep the code clean” is guidance. “Only this module may access these tables” is enforceable.

An observable violation. The repo harness needs evidence it can inspect. A behavioral test may compare an output or durable state with the expected result. A structural check may detect an illegal import, dependency cycle, foreign table access, or drifted contract.

An automated checker. Enforcement cannot depend on someone remembering the rule during review.

One command and a deterministic consequence. The same command should run for a developer, a coding agent, and remote CI. A violation should fail clearly, identify the broken rule, and point toward the fix.

Tests for custom enforcement code. Custom checkers and generators are code, so they need automated tests like any other code. A checker test should include a known violation and prove that the checker rejects it. A clean repository passing only proves that the current repository passes; it does not prove that the checker works.

A harness combines 3rd-party tools with repository-owned checks

A repo harness is not one product, and most of it does not need to be built from scratch. It usually combines general-purpose 3rd-party tools with tests, configuration, and custom checks owned by the repository.

3rd-party tools provide the general-purpose capabilities. Test runners, linters, type checkers, dependency analyzers, security scanners, migration tools, and dead-code detectors already solve common verification problems. Use an existing tool when it can reliably enforce the property you care about.

Repository-owned verification provides the local rules and fills the gaps. Automated tests encode the application’s expected behavior. Tool configuration, custom checkers, and generators encode repo-specific constraints such as dependency direction, module boundaries, table ownership, public-interface restrictions, and generated contract consistency. When an existing tool cannot express a constraint reliably, write a narrow custom checker or generator.

The harness is language-specific

Part of a repo harness exists to enforce guarantees that a language does not provide on its own. The same architectural rule can require different enforcement depending on the stack.

This matters especially with AI agents. An agent will often take the shortest valid path to complete a task. It does not treat a convention as a hard boundary simply because the convention is documented. If the language permits a shortcut, an agent will eventually use it.

Consider module privacy. In Python, a leading underscore marks a name as internal, but that is only a convention. Nothing prevents another module from importing it directly. If that boundary matters, the harness has to enforce it.

Java provides stronger access control. Private and package-private members are enforced by the compiler, and the Java module system can restrict which packages are exported. When the boundary is represented correctly, ordinary code outside it cannot reach those internals. In that case, the compiler and module system are already enforcing part of the harness.

Dependency cycles provide another example. Go rejects cyclic package imports during compilation. Python and TypeScript do not enforce an intended acyclic module architecture, so repositories using them may need additional dependency-analysis tooling.

There is no universal implementation of a repo harness. Start with the guarantees the language, compiler, and type system already provide. Then use linters, dependency analyzers, tests, or custom checks to close the gaps that matter to the repository.

The harness is never complete

A strong baseline is mandatory. Every repository needs the basic checks appropriate to its stack: automated tests, linting, type checking, security scanning, dependency hygiene, and build validation. These checks catch broad classes of problems and should be present from the beginning. But they are only a starting point.

Repo-specific checks are equally important. A payments application may need to enforce tenant and ledger isolation. A data platform may need schema-compatibility checks. General-purpose tooling cannot know which properties matter to a particular system.

Higher development speed produces feedback faster. When that feedback exposes a gap in the architecture, the code and harness should evolve together.

A postmortem should therefore ask not only, “What code allowed this failure?” but also, “What gap in the harness allowed this change to be accepted?” The fix may include a code change, a new conformance rule, a checker and tests for that checker, or clearer documentation of the repository’s constraints.

That does not mean mechanizing every preference. A noisy or ambiguous check teaches people to ignore the harness. Add enforcement when the property is important, the violation is observable, and the signal can be reliable.

Some drift is visible only over time

Per-change checks catch violations of rules the repository already knows. But some forms of degradation accumulate slowly. No individual change is clearly wrong and the pattern becomes visible only across many changes.

Module dependencies may all be legal while the graph becomes increasingly dense. Public interfaces, suppression allowlists, test runtimes, and artifact sizes may grow a little at a time. Domain concepts may acquire duplicate names and representations. External conditions also change: a dependency can gain a vulnerability without anyone touching the repository.

These properties are better evaluated by recurring checks. Deterministic checks can run as scheduled jobs and fail against an explicit threshold or baseline. More subjective concerns, such as domain model quality, can produce a report for periodic architectural review.

Recurring checks should not become a dumping ground for noisy analysis. If a check is precise, reliable, and cheap enough to run on every change, it belongs in the normal harness. If a recurring review reveals a repeatable failure mode, the next step is to turn that lesson into a deterministic rule wherever possible.

This is really it—a repo harness is not a complicated concept: establish a strong baseline, add the conformance checks your system needs, run them consistently, use recurring checks to catch slower drift, and evolve the harness as the codebase teaches you.

To make it all tangible, the appendix below catalogs representative checks generalized from my real-world experience, along with a few recent ideas.

Appendix: Sample harness checks

This is not a complete list of harness checks. These examples focus on less obvious forms of repo conformance: checks that prevent architectural drift and repository-specific failures ordinary tooling will not catch.

System level

The repository has an explicit directory structure

Problem. Developers and coding agents create new top-level folders or one-off directory shapes whenever the existing structure is not immediately obvious. Over time, the repository stops communicating where code belongs.

Rule. Tracked files must fit a declared directory structure. Adding a new structural concept requires an explicit change to that definition.

How.

• Describe the permitted tree in a machine-readable manifest or the repository’s existing build and workspace configuration.

• Compare tracked paths with that structure and fail on unknown structural paths.

• If this requires a custom checker, test it with a known invalid path.

Cross-app contracts cannot drift

Problem. A contract is often represented independently by several applications or languages. A producer can change while a consumer, generated artifact, or hand-written representation silently remains on the old shape.

Rule. Every cross-app contract must stay synchronized across its producer, consumers, and committed schema. This includes REST APIs, queue and event payloads, WebSocket or SSE messages, webhook bodies, shared file formats, and custom cross-language protocols.

How.

• In a code-first design, generate a canonical OpenAPI, AsyncAPI, Protobuf, JSON Schema, or equivalent artifact from the producer and commit it.

• Regenerate that artifact in CI and fail if it differs from the committed version.

• Generate each consumer’s internal types from the artifact and check those generated results for drift as well.

• In a schema-first design, make the committed schema authoritative and generate both sides from it. If a representation must remain handwritten, compare it with the schema or test the contract directly.

• OpenAPI Generator supports Java, Python, and TypeScript, while Buf can check Protobuf compatibility. Independently deployed consumers may need compatibility checks in addition to drift checks.

App level

Every public entry point declares its cross-cutting policy

This is one of the most important examples in this appendix. It demonstrates how you can enforce an important goal through complementary architecture & harness implementations. Here architecture reduces a complex cross-cutting requirement to a single declaration mechanism. And the harness verifies that this declaration mechanism is used on every entry point.

Problem. Authentication, authorization and permissions, auditing, and observability are easy to apply inconsistently when every endpoint or module operation handles them independently. A new entry point can omit a concern entirely or enforce it differently from the rest of the application.

Rule. Every endpoint and public module operation must use the application’s standard entry-point declaration. The declaration explicitly states the policies that vary by operation, while the runtime automatically applies the concerns that are universal.

For example:

@endpoint(
    url="/cart",
    authn=SignedIn,
    permissions=[ViewCart, ViewPaymentMethods],
    audit=False,
)

How.

• Provide one standard declaration mechanism: a decorator, annotation, attribute, or typed registration object, depending on the language and framework.

• Require explicit values for policies such as authentication mode, permissions, and auditing. Omission cannot silently mean “unprotected.”

• Route invocations through shared runtime infrastructure that interprets the declaration and applies authentication, authorization, auditing, observability, and other cross-cutting behavior.

• Have the harness inspect every registered endpoint and declared module interface and fail if the required policy metadata is missing.

• Test both pieces: checker tests prove that missing metadata fails the harness, while behavioral tests prove that the shared runtime infrastructure enforces each declared policy.

The harness guarantees that every entry point declares a policy and that the declared policy is enforced. Choosing the correct permission remains a policy-design and testing problem.

Test-only code cannot enter production

Problem. Reset endpoints, seed helpers, fake controls, test imports, or environment overrides can become reachable in a production artifact even though the test suite itself works correctly.

Rule. Test support may exist in a controlled test topology but must be absent from production dependencies, routes, contracts, and build outputs.

How.

• Isolate test support with mechanisms such as Java test source sets, dedicated Python test packages, or separate TypeScript entry points.

• Prevent production code from depending on that source layer and exclude it from production builds.

• Inspect the production artifact’s routes, imports, bundled files, or generated contracts for test-only capabilities.

Module dependencies and public interfaces follow the architecture

Problem. As an application grows, modules begin depending on modules they should not know about, form cycles, or bypass a module’s public interface by importing its internals.

Rule. Cross-module dependencies must follow permitted directions, remain acyclic when required by the architecture, and target only interfaces deliberately exposed by the owning module.

How.

• Prefer native module and package visibility when the language provides the required guarantees.

• Otherwise, use dependency analysis to reject forbidden edges, cycles, and imports of internal files or symbols.

• Examples include Tach for Python, ArchUnit for Java, and dependency-cruiser for TypeScript and JavaScript.

• In dynamic languages, keep the public surface statically discoverable so enforcement tools do not need to execute application code.

• Account for gaps such as dynamic imports and reflection with a custom check or an explicit restriction.

Database models and migrations cannot drift

Problem. Application models can change without a migration, migrations can diverge from the models, or a migration history can fail when applied from an empty database.

Rule. The migration history must produce the database schema expected by the current application.

How.

• Apply the complete migration history to a disposable database.

• Derive the expected schema from the application’s current model or schema definition.

• Compare the two schemas and fail on meaningful differences.

• If applications run migrations during startup, separately test concurrent migration attempts.

Inline suppressions must be explicitly allowlisted

Problem. Inline directives such as “noqa,” “type: ignore,” “eslint-disable,” “ts-ignore,” “nolint,” or “SuppressWarnings” are sometimes necessary, especially in modules doing technically unusual work. But an untracked suppression can also bypass an important check and then be copied without its original context.

Rule. Every inline suppression must appear in a small, committed allowlist. An unlisted suppression fails the harness.

How.

• Keep the approved suppressions in a simple text file, identified by their source location and directive.

• Use a repository script to scan the source tree, normalize the suppressions it finds, and compare them with the allowlist.

• Fail on both new suppressions and stale allowlist entries.

• Keep the list small—likely dozens of entries concentrated in a few unusual modules—so every addition receives explicit review.

• Test the script with an unlisted suppression and a stale allowlist entry.

Module level

Persistence models stay inside the module that owns them

Problem. When ORM entities or persistence records cross a module boundary, callers can become coupled to database columns, relationships, lazy-loading behavior, or session lifecycle. The module no longer owns its persistence implementation in practice.

Rule. When persistence is intended to be a module implementation detail, public interfaces expose domain values or DTOs rather than ORM entities.

How.

• Use native visibility where the language can keep persistence types inside their owning module.

• Otherwise, use architecture tests, import rules, or public-signature analysis to detect references to known persistence types.

• Tailor the check to the ORM and type system rather than assuming one technique works everywhere.

• If persistence models are deliberately shared contracts, encode that architecture instead of prohibiting them.

Module interfaces do not expose module-owned mutable state

Problem. Exporting a registry, cache, ambient context object, mutable collection, or singleton accessor lets callers read or change state outside the behavior owned by the module.

Rule. A module exposes behavior and immutable values, not direct access to the mutable state it owns.

How.

• Use access modifiers, immutable interfaces, or ownership types where the language provides them.

• Otherwise, inspect exported bindings and public return types with architecture tests or a custom check.

• Target concrete ways state can escape—such as exported instances, mutable collections, or singleton accessors—rather than treating every class instance or collection as invalid.

Every database table has one owning module

Problem. A module can bypass another module’s interface by importing its persistence model or issuing SQL directly against its tables. The code may still work while ownership quietly disappears.

Rule. If the architecture assigns table ownership, only the owning module may access that table directly.

How.

• Derive or declare which module owns each table.

• Use dependency rules to prevent modules from importing another owner’s ORM models.

• Where reliable, analyze literal SQL for references to tables owned by another module.

• Enforce only the query forms the checker can inspect accurately, and centralize infrastructure exceptions such as migrations.

Most teams can get a meaningful productivity boost from AI coding tools without changing their architecture. Agents write boilerplate, explain unfamiliar code, and accelerate local implementations. That is the 2x version: coding tasks get faster. The 5x version is not just faster code generation; it is about safely delegating much larger units of work.

I use 2x and 5x to describe the difference in leverage. They are not precise benchmarks, although they are directionally consistent with informal industry reports.

That requires moving up an abstraction layer. Instead of spending most of the day manipulating code, a developer describes goals, constraints, architecture, and tradeoffs while the agent handles more implementation. This sounds like a tooling change. It is actually an architecture problem.

A developer can compensate for a confusing codebase by learning its hidden conventions. An agent starts with none of that institutional memory. If the architecture is implicit, it must reconstruct it—or guess. Agents are great at repeating the patterns they find: in a clean codebase that is desirable; in a legacy codebase it produces legacy code faster.

Architecture is a system of complementary patterns

What is architecture? My working definition is pretty simple. Architecture is a system of complementary patterns.

Both system and complementary are important. A codebase can contain dependency injection, repositories, services, events, tests, and clean interfaces without those patterns composing into coherent architecture.

Good architecture answers the same questions consistently:

• Where does this behavior belong?

• Who owns this data?

• How may another part of the system use it?

• What context is required to change it?

• How do we know the change is safe?

• Which rules require judgment, and which can be enforced?

The answers create architectural seams: the places where parts of the system meet—and where we can enforce rules. A module interface is a seam. So is an app boundary, API contract, database ownership rule, or the line between production code and system tests.

Those seams also become the language for discussing change. “Create a returns domain module and add a get_returnable_items operation to the orders module’s public interface” tells an engineer or agent where behavior belongs and which boundary may change. That is far more useful than “add return functionality.”

AI is excellent at repeating patterns, which makes consistency unusually valuable. Humans must own and approve the seams, although agents can increasingly propose them. Without explicit human ownership, an agent may make local architectural choices without the global purview needed for system-level design.

The goals

Before getting into the repository shape, here are the four goals of this architecture.

Human-directed and agent-legible. Humans own goals, constraints, boundaries, and judgment. The repository makes those decisions explicit enough for an agent to act without reconstructing hidden conventions.

Locally understandable and changeable. Most changes should require understanding one bounded area and its explicit dependencies, not the whole system.

Clear ownership, automatically enforced. Every behavior, contract, and piece of state has one owner. And CI rejects violations of critical architecture rules.

Safe to change and govern. Stable seams provide consistent hooks for cross-cutting policy and allow parts of the system to evolve independently where their boundaries permit.

These are not AI-only goals. They also reduce human onboarding time, review scope, accidental coupling, and knowledge trapped in senior engineers’ heads.

Historically, maintaining this level of consistency required substantial communication and oversight, so it was most common in large engineering organizations. AI agents and executable harnesses now make the same discipline accessible to much smaller teams.

Architecture, disassembled

Show me the map

repo/
├── apps/
│   ├── app1/
│   │   ├── app/
│   │   │   ├── core/
│   │   │   │   ├── module1/
│   │   │   │   └── module2/
│   │   │   └── domain/
│   │   │       ├── module1/
│   │   │       └── module2/
│   │   ├── bin/
│   │   ├── docs/
│   │   ├── harness/
│   │   ├── db_migrations/
│   │   └── test_support/
│   └── app2/
├── shared/
│   └── <language>/
│       ├── shared/core/
│       ├── bin/
│       ├── docs/
│       └── harness/
├── e2e/
│   ├── tests/
│   ├── test_services/
│   ├── bin/
│   └── docs/
├── bin/
├── docs/
├── harness/
├── AGENTS.md
└── README.md

The directory tree is not the architecture itself, but it is a useful projection of it. At a glance, a human or agent can infer whether code is system-wide, app-owned, reusable, business logic, technical infrastructure, or test-only. The path communicates likely ownership, dependencies, documentation, tests, and governing policy.

One pattern repeats across the major scopes: bin, docs, and harness.

• bin is the stable command surface for humans, agents, and CI.

• docs explains the architecture and workflows owned by that scope.

• harness implements the checks, generators, and dependency analysis.

The trio appears at repository, shared-language, and app scopes. Modules are leaf scopes: each contains its implementation, tests, and docs, but does not duplicate the command and harness platforms.

Everything has a place, and the place communicates meaning.

Apps, services, and modules are different boundaries

These terms are often blurred, leading to architecture driven by deployment topology rather than software cohesion. Code that belongs together may be separated simply because it runs in different processes.

• A service is an independently deployable runtime boundary. It may run on infrastructure you operate—such as an API, worker, or scheduler—or in a client environment, such as a separately deployed SPA.

• An app is a durable code, build, and data-ownership boundary. It has its own build target and CI scope and, when it stores data, usually its own database and migration lifecycle.

• A module is a bounded package of production code for one capability. It belongs to an app or a shared-language scope and exposes a public module interface. In practice, it owns one job—billing, reviews, authentication, etc.

One app may produce an API, worker, scheduler, and event consumer services. They can scale and deploy differently while reusing the same modules and data ownership.

Splitting out a new app is a big commitment. If the boundary is not obvious, keep the capability as a module. You can still run it as a separate service when it needs independent scaling or failure isolation. A well-structured module is easy to extract later; prematurely split apps are painful to stitch back together.

The same distinction applies on the frontend. A feature area within an SPA is usually a module. An independently built and deployed SPA is a service boundary. Create a separate frontend app only when it represents a durable product, code, and ownership boundary—not merely a new persona, screen, or route. Micro-frontends are an operational choice, not the default module boundary.

One repository

As you may have guessed already, this is a monorepo of independently bounded apps, reusable same-language core modules, and system-level support.

This is not limited to server code. Web, mobile, ML, and other production applications belong in the same monorepo when they meet the app boundary, even when they use different languages and toolchains.

The monorepo is the envelope, not the core idea. It gives an agent one system map, allows atomic cross-app contract changes, and gives shared enforcement one home.

The monorepo provides visibility. The app boundaries provide independence.

Modern build and CI/CD tools provide first-class monorepo support, making builds and deployments much easier to manage than they were a decade ago.

Shared is not a miscellaneous drawer

Code reused by several apps is grouped by language under shared.

Shared is not a home for generic utils or helpers; catch-all modules obscure ownership and become dumping grounds.

Shared contains reusable core capabilities, not app-specific workflows or domain modules. Similar business concepts may remain separate when their rules and ownership differ.

Each app consumes only the shared modules it needs through explicit local dependencies. Shared modules are linked directly during each app’s build; the team does not publish them as internal packages.

The language-specific mechanism can vary. The invariants do not:

• The dependency is explicit.

• The module keeps a stable namespace and owner.

• The same boundary checks apply.

Module dependencies form an acyclic graph, with the repo harness rejecting any circular dependency.

Modules are the primary unit of understanding

Inside an app, modules are the primary architectural boundary. A module owns one cohesive capability, its public interface, implementation, tests, docs, and any tables it declares.

A domain module owns a business capability: its rules, workflows, and state. Its boundary also gives the module’s business language one consistent meaning—the domain-driven design (DDD) idea of a bounded context.

A core module owns a technical capability such as authentication, observability, database mechanics, messaging, auditing, etc.

Core does not have to mean domain-blind. Some core modules are domain-aware. Observability may describe tickets, accounts, runs, and actors. Auditing may record that a user changed an account.

A domain-aware core module may accept small typed domain values or references, as long as doing so does not make core import or depend on the domain module. But it does not own a business workflow or make a business decision.

Domain-aware core uses business vocabulary to perform technical work. Domain owns business behavior.

The appendix includes concrete examples of core and domain modules.

Modules meet the system through explicit seams

A seam is any boundary where a module or app exposes something others depend on, or interacts with something it does not own. Within an app, common seams include public code interfaces, endpoints, table definitions, and adapters. At app boundaries, APIs, events, queues, and other owned contracts are the seams. Apps do not import one another’s implementation or access one another’s databases.

Each seam has one owner and an explicit contract. Other modules use public interfaces rather than internal implementation; clients use module-owned endpoints; integrations pass through owned messages and adapters. Because seams have standard locations, the repo harness can enforce ownership, detect contract changes, attach policy, and identify affected consumers.

Every table has exactly one owning module. Other modules use that module’s public interface rather than querying its tables directly.

Modules own their schemas and table access; the app coordinates them through one physical database and migration stream. This produces a modular monolith: in-process collaboration and one database where useful, without shared global state.

Cross-module composition still needs an owner

Explicit ownership creates an immediate question: if business logic lives in domain modules, where does behavior spanning several modules belong? Composition is not ownerless—the module accountable for the outcome coordinates the work through public interfaces.

A use case spanning several modules therefore belongs to the module that owns its outcome. For example, returns can retrieve order items through the orders interface, apply return-eligibility rules, and expose the assembled result.

Screen-specific combinations can be assembled by the frontend or by a thin client-facing query module when they need a dedicated server-side contract. Clients may assemble and present data, but authoritative business rules remain with the modules that own them. If a query becomes expensive or historical, its owner may maintain an event-fed read model.

Composition must use public interfaces—never another module’s tables or a generic aggregation module that collects unrelated use cases.

Architecture inside a module

Do modules need their own clean, hexagonal, or layered architecture? Sometimes, but not automatically.

Most modules should remain simple.

In the online store, catalog might remain a straightforward set of product operations over persistence. orders, with a multi-step lifecycle such as placement, payment, fulfillment, and cancellation, may benefit from separating state transitions and business rules from persistence and external integrations.

For a complex module, separating business logic from databases and external dependencies is usually the highest-value internal seam. If it still needs many layers, first ask whether it contains several capabilities and should become several modules. Add deeper architecture only when the remaining complexity is cohesive.

Architecture should go down to the level where local patterns are clear. Below that, the agent can figure out the implementation.

Documentation and testing follow ownership

Docs are colocated with the scope they explain. Root docs own system behavior, app docs own cross-module flows, and module docs own the module’s boundary and invariants. Parent docs link downward instead of repeating facts. One fact lives in one place.

Context discovery follows the same structure. For a changed path, it returns the smallest complete doc set. Root context is included for system work, not every local change; irrelevant context can be as harmful as missing context.

Testing follows ownership too:

• Module tests verify behavior inside one module.

• App-level test support verifies flows crossing modules through public interfaces.

• Top-level E2E tests verify the assembled system through the browser, HTTP, and events.

• Purpose-built fake external services—such as a fake payment gateway or email provider—live with the E2E topology. This is why the e2e directory is top-level rather than another app: it owns verification of the assembled system.

The harness makes the map true

The repo harness turns the architectural map from guidance into an enforced system.

In my earlier framework for AI dev tooling, I separated the repo harness from the coding agent harness. A coding agent harness—such as Claude Code—is the loop around the model: planning, tool use, memory, and execution. The repo harness is specific to the repository and makes any coding agent effective inside it.

The repo harness has two sides.

Deterministic. Scripts and hard checks enforce dependency direction, cycles, public-interface use, table ownership, tests, generated contracts, and other invariants. Violations fail before merge.

Non-deterministic. Docs and AI skills are explicit, deterministic artifacts, but how an agent interprets and applies them is not. They help it load the right context, follow the intended workflow, and apply human-defined judgment where hard checks cannot.

The repo harness should not become a second, drifting architecture description. It treats the tree and build files as sources of truth:

• Convention establishes structure.

• The harness derives structural facts—including module inventories, dependencies, context, contracts, and affected consumers—from the tree and build files.

• Configuration is used only for choices that cannot be inferred, such as review requirements or risk classifications.

The same dependency graph also supports CI/CD: changes can build and test the affected apps and consumers, while contract and migration changes receive additional verification. Deployment ownership, observability, rollback, and failure isolation attach to the same app and service boundaries.

Seams are policy hooks

Stable seams support more than dependency checks.

A module interface is a natural attachment point for tracing, audit logging, authorization, validation, and other cross-cutting policy. If all cross-module operations use instrumented entry points, every incident can at least be narrowed to a module.

The same paths support human ownership, reviews, alerts, dashboards, runbooks, reliability targets, and security responsibility.

They also support AI governance. A good software factory exposes hooks where teams decide how much autonomy an agent receives, and periodically checks whether the architecture still matches its intended shape. Recurring drift checks catch cumulative problems that no single change exposes—dependency creep, blurred ownership, stale docs, or missing enforcement. Their findings become refactoring work and, where possible, new deterministic checks. CODEOWNERS-style policy can govern authority, not just who does the code review.

For example:

• An internal module change with no interface change may proceed autonomously.

• A module interface change or new dependency may require human review.

• Contracts and migrations trigger extra verification.

• Security or harness changes remain human-gated.

This is better than reviewing every AI change manually or allowing unrestricted autonomy. The goal is high autonomy inside low-risk scopes, with escalation when a sensitive boundary or contract changes.

Why this compounds

Taken together, these patterns change the size and number of tasks that can be delegated safely.

Cohesive modules reduce context. The agent reasons about one capability and its explicit dependencies.

Standard shapes make patterns reliable. Nearby examples teach the right lesson.

Explicit ownership bounds the blast radius. Interfaces and table ownership make impact clear.

Executable rules enable autonomy. Humans need not supervise every import, table access, contract update, or documentation obligation.

Context discovery reduces setup. The right docs and skills arrive with the task.

The dependency graph makes verification complete. Shared changes trigger every affected consumer without testing unrelated code.

Stable seams make governance cheap. Reviews, permissions, observability, auditing, and authorization use the same model.

These benefits compound: a local task is easier to implement, a standard change is easier to review, a bounded change is safer to automate, and targeted verification makes that automation faster.

Humans get the same map. New engineers navigate faster, reviewers focus on real impact, teams own modules without owning microservices, and incident responders narrow failures to stable boundaries.

This is not architecture optimized for AI at humans’ expense. It is good architecture made explicit enough that AI can participate and that human devs equally benefit from as well.

The architecture has to evolve at AI speed

Some boundaries will be wrong, and modules will sometimes grow beyond what remains locally understandable. Faster implementation makes continuous architectural correction more important because structural problems can spread just as quickly as useful patterns.

When AI lets a team move 5x faster, refactoring also has to move 5x faster so architectural feedback keeps pace. I would expect 10–20% of development time to go toward architecture, the repo harness, and refactoring as a practical rule of thumb. When an agent makes a mistake a check could catch, add the check; when a rule repeatedly needs explanation, improve the docs or skill; when a module becomes confusing, redesign it.

Without equally fast architectural feedback, faster implementation creates a faster-growing legacy system. The promise of AI development is not more lines of code but moving human attention toward product intent, boundaries, tradeoffs, and judgment. Great architecture makes the correct next action locally discoverable and incorrect actions cheap to reject.

As models improve, the delegation ceiling rises

As models improve, the same explicit modules and seams let agents safely own progressively larger units of work. Today that may be a bounded implementation task; over time it can grow to designing a module, coordinating changes across modules, evolving contracts, or planning migrations. The structure does not need to loosen as model capability rises.

Humans can also grant agents more architectural authority over time. Agents may propose boundaries and dependencies while the harness verifies mechanical rules and sensitive changes still require review. Trust expands by scope instead of becoming unrestricted.

This architecture therefore does not depend on the limitations of today’s models. Better models raise the ceiling of work that can be delegated, while explicit ownership, contracts, and verification keep ambiguity and blast radius bounded.

Appendix

The appendix turns the preceding patterns into concrete repository shapes. It includes a complete generic tree and a small online-store example showing how core and domain modules divide responsibility.

Complete repository tree

This generic tree shows the complete repository shape without tying it to a particular product domain.

repo/
├── apps/
│   ├── app1/
│   │   ├── app/
│   │   │   ├── core/
│   │   │   │   ├── module1/
│   │   │   │   │   ├── docs/
│   │   │   │   │   └── test/
│   │   │   │   └── moduleN/
│   │   │   └── domain/
│   │   │       ├── module1/
│   │   │       │   ├── docs/
│   │   │       │   └── test/
│   │   │       └── moduleN/
│   │   ├── bin/
│   │   ├── db_migrations/
│   │   ├── docs/
│   │   ├── harness/
│   │   ├── contracts/
│   │   └── test_support/
│   │       ├── docs/
│   │       └── test/
│   ├── app2/
│   └── app3/
├── shared/
│   ├── language1/
│   │   ├── bin/
│   │   ├── docs/
│   │   ├── harness/
│   │   └── shared/
│   │       └── core/
│   │           ├── module1/
│   │           │   ├── docs/
│   │           │   └── test/
│   │           └── moduleN/
│   └── language2/
├── e2e/
│   ├── bin/
│   ├── docs/
│   ├── tests/
│   └── test_services/
│       ├── service1/
│       │   ├── docs/
│       │   └── test/
│       └── serviceN/
├── bin/
├── docker/
├── docs/
├── harness/
├── AGENTS.md
└── README.md

Example modules

Here is how a small online store might apply the module structure:

apps/store/app/
├── core/
│   ├── auth/
│   ├── observability/
│   ├── database/
│   └── web/
└── domain/
    ├── catalog/
    ├── orders/
    └── returns/

Core modules

• core/auth owns authentication and exposes login mechanics once for the entire app.

• core/observability attaches baseline telemetry at module seams so domain code does not recreate tracing and logging.

• core/database owns connections, transactions, and migration mechanics without owning domain tables.

• core/web owns HTTP server setup, shared middleware, request and response mechanics, and module route registration. Domain modules still own their uniquely prefixed endpoints.

Domain modules

• domain/catalog owns products, their descriptive information, tables, public operations, and /catalog endpoints.

• domain/orders owns the order lifecycle, order tables, public operations, and /orders endpoints.

• domain/returns owns return eligibility and workflows and publishes endpoints under /returns. It uses the orders module’s public interface rather than querying order tables.

In my recent posts I laid out the framework for thinking about software factories and reviewed sixteen of them. I ended up with a question: what should an ideal software factory look like? And will it scale for the next 3-5 years?

My opinion is based on research and on my experience building a small internal software factory. That is where my conviction comes from, but it is likely also the source of some bias.

Please note that software factory is a newly emerging term for this product category. It’s not universally adopted across the industry, but I like it and I’ll use it instead of my previous orchestrator term.

The developer flow

I want to start with the developer experience that a software factory should enable.

It starts in the inner loop, where a developer shapes the requirements, spec, and architecture with a local agent. Once that is solid, the factory takes over: planning, implementation, audits, and the PR at the end. Then the work often returns to the inner loop for small fixes or final polish—typically on larger or UI-heavy tickets (small tickets can merge right in).

However, not every task starts locally. A bug may arrive from Slack, Jira/Linear, or an alert. Some factory tasks come from recurring jobs (eg, check observability for errors, look for dependencies to upgrade, etc).

Every ticket ends in one of two ways. If the factory clears its audits and gates, it opens a PR—or merges it when policy allows. If it is uncertain, needs missing input, or wants to change protected code, it pauses and contacts the developer through Slack, email, or the factory UI with the exact question or approval it needs. Once the developer responds, the run resumes from the same state.

Capabilities a factory must support

Here is a practical breakdown of the core capabilities.

Take work from anywhere. A local spec, a Slack thread, Jira/Linear, or a monitor firing from logs and traces. Whatever the source, the factory converts the raw input into the first formal requirements artifact before the pipeline proceeds.

Run recurring work. Security drift, performance regressions, user-feedback triage, etc are just some examples of workflows triggered by a schedule instead of a ticket.

Keep the full task context together. Requirements, plans, review findings, and human decisions should be readable in one place and loadable into the inner loop.

Make execution explicit and durable. Stages, state, gates, retries, and resumptions should be visible. The LLM work is non-deterministic, but the workflow around it should not be mysterious.

Escalate by risk and uncertainty. Protected code, low confidence, incomplete requirements, and custom policies can all justify human-in-the-loop (HITL) intervention.

Make disagreement cheap. When a developer rejects an LLM architecture choice or review finding, the fix should be an quick human-factory exchange followed by the factory resuming the ticket run.

Send work backward. If implementation exposes a requirements gap, the factory should return the task upstream with a reason, repair the artifact, and continue from that stage.

Learn from history. Dismissed findings, disputed decisions, and stalled runs should produce concrete suggestions for improving skills and repository instructions: “this review skill over-produces findings here,” or “the repository guidance no longer matches the architecture.”

Move cleanly between inner and outer loops. A developer should be able to hand work off and pull it back without rebuilding context.

Together, these requirements point toward a spec-first pipeline: durable artifacts crossing explicit boundaries, with configurable HITL between them. Here’s what that architecture looks like.

The architecture

Linear by default, limited DAG when justified

Mature factories are converging on explicit pipeline stages with LLM calls inside them. Not one agent wandering freely from intent to PR.

Most of these pipelines should be linear. Requirements feed architecture, architecture feeds a plan, and a plan feeds implementation. A general-purpose DAG adds authoring and debugging complexity that most teams will not use. Specialized software should be opinionated; exposing every topology primitive pushes complexity onto the adopter.

That said, a sophisticated campaign loop (running multi-week projects) can call for a limited DAG:

• Fan-out and fan-in: test several root-cause hypotheses or architecture options, or apply the same migration pipeline across many files.

• Work dependencies: sequence related workstreams inside a large project plan. This is typically more of a tree where tasks depend on each other.

The factory should support those cases without turning into Airflow.

Spec-first pipeline stages

The factory should map the team’s spec-first workflow directly into pipeline stages, with each spec stage becoming one pipeline stage.

Any sensible spec-first flow can work. I use this one and will use it as a reference for this post.

requirements → architecture → plan → implement → code review

Each stage produces a durable artifact for the next. Spec Kit uses four stages, Kiro three, Microsoft seven. The exact count matters less than the contract between them.

The spec guides the run. After merge, the code becomes canonical and the spec becomes a historical record of how the code got there.

Inside a stage: audit, fix, escalate

However, stages are a bit more sophisticated than they seem. Every stage can be configured to run a loop - run the skill (plan, implement, etc), audit the output, fix what the agent can resolve confidently, and escalate the rest.

Complex artifacts may deserve several reviewers or audit passes, just as a design document may go through several human reviews. The factory should make the number of passes, prompts, roles, and models configurable.

HITL should support four forms:

• Uncertainty-driven: low confidence or incomplete requirements.

• Fixed gate: a stage always pauses for review.

• Hook-driven: adopter-written checks can block progress.

• Declarative policy: paths or owners require approval.

Declarative protected-code rules are especially useful in structured repositories. They are similar to GitHub’s CODEOWNERS. They can be implemented through hooks, but I want to bring them to the forefront because they are a practical way to manage risk. For example, authentication, security, and database migrations may stay human-gated indefinitely while other paths are auto-merged.

Escalation can also send work backward across stages. If implementation finds a requirements gap, it returns the task to requirements with an explanation. The artifact is revised and the pipeline runs forward again through the same gates. That is better than improvising or failing the run, and because the transition is explicit, it is fully auditable.

Stage contract: formal artifacts only

So how do the stages communicate with each other? Only the completed artifact—whether a document, code change, branch, or PR—crosses a stage boundary. Not the full conversation, partial drafts, or failed audit attempts.

That keeps context bounded because intermittent state is discarded between stages. It also makes inner/outer-loop handoff portable, allows the pipeline to resume from the last completed stage, and gives humans something readable and diffable. A person can reject one artifact without replaying the entire conversation that produced it.

Spec Kit’s constitution.md → plan.md → tasks.md and Kiro’s requirements.md → design.md → tasks.md are file-level versions of the same contract.

Skills implement stages

A stage typically contains at least two skills: a main skill that creates the artifact and a review skill that audits it. In the plan stage, those might be create plan and review plan. In the implementation stage, they are implement and code review.

The artifact can be a document, code change, branch, or PR. The rule is the same: the main skill produces it, the review skill evaluates it, and the stage emits one completed artifact across the boundary.

Inside those skills, anything goes: parallel agents, adversarial verification, multi-model consensus, or deterministic post-processing. A code review skill, for example, might run parallel security, quality, and architecture reviewers, followed by an adversarial pass and synthesis. Nine agents inside, one findings artifact outside. Stage-internal complexity is normal; it just should not leak into the outer workflow unnecessarily.

This also means that a developer can run the same pipeline locally by invoking each skill manually.

The factory owns its ticket

Every task gets a factory ticket, regardless of where it came from. Requirements, plans, findings, audit entries, and HITL decisions attach to it.

Many valid tasks—the Slack bug, an alert, an ad-hoc cleanup—have no external ticket and should not be forced to create one. When an external Jira or Linear ticket does exist, the factory treats it as input and uses its context to produce stage artifacts on the factory ticket. This avoids mirroring every intermediate artifact back into the external system.

The factory should expose ticket context through MCP, an API, or its own CLI. The mechanism is secondary. The requirement is that a developer—or another agent—can load the complete task outside the factory.

The PR is the other important artifact. CI failures return the task to implementation, PR comments become new factory input, and merges remain human-controlled where required.

This design produces the audit trail almost automatically: every artifact, approval, dismissal, and timestamp is attached to the same ticket.

A proprietary harness is fine. A closed one isn’t.

I started this research convinced the factory had to run the same agent harness developers use locally (eg, claude code, codex, etc). I was wrong.

A proprietary harness may be better in the outer loop. The vendor can optimize context retrieval, model routing, retries, audit, and long-running execution as one system. The problem is not a proprietary harness. It is a closed one.

The minimum bar is portable skills and artifacts, and complete context accessible outside the product. A developer must be able to write a spec in the inner loop, hand it to the factory, then pull back the branch and every relevant decision. Using Claude Code or Codex is useful when available, but not mandatory.

Price changed my view too. I expected proprietary orchestration to carry a huge premium. From the public pricing I could find, a factory with BYOK lands near raw agent costs at team scale—not an order of magnitude above them.

Why this scales as models improve

This is the million dollar question. Will this scale as LLMs get better and better? Or will advanced LLMs make this scaffolding obsolete?

I believe that this will scale as models get better. Model-dependent pieces remain replaceable: models themselves, skills, and state. The pipeline does not encode or rely on any specific model behavior.

The pipeline encodes organizational thinking: trust boundaries, audit requirements, ownership, and acceptable blast radius. Better models do not change those.

However, better models will produce better output, handle larger tasks, and move through the same architecture with fewer audit failures, escalations, and send-backs.

Even a perfect implementation model does not remove the need to approve a high-stakes spec, retain an audit trail, or control who merges protected code. Those requirements come from an accountability framework, not model capability.

So with better LLM model, this will scale by being able to tackle larger tasks with higher quality.

I don’t expect every factory to look exactly like this, but I do think these are the factory aspects that teams should consider.

The software factory market is still nascent. Even the term itself is not universal, and the products on the market are quite different from one another. I hope this gives you a concrete way to evaluate them and choose the best fit for your needs.

After this post went live I’ve learned that there’s a new term emerging - Software Factories. I like this term - it’s exactly what I meant by coding orchestrator.

At some point in every org’s AI adoption curve, gaining further effectiveness requires a cloud-based coding-agent orchestrator. The work that benefits is the kind that should run without a developer driving — “fix this bug”, “investigate this log spike”, or “ship this feature I’ve already specced”. Solo developers benefit too. But the inherent standardization that an orchestrator brings is more beneficial for larger orgs.

In this post I look at how Stripe, Coinbase, Uber, Meta, and Shopify built their internal AI coding tools. I also look at how they compare to on-the-market AI coding tools. I was surprised to find out that there’s a clear division between internal vs. on-the-market tools.

The technical language used in this article — models vs agent harnesses vs repo harnesses, inner vs outer loops — is something I laid out in a previous post. This post is a survey of sixteen orchestrators (all that I could find) analyzed along these axis.

What counts as an orchestrator

There are several categories that got swept up in my initial broad search. These are all excluded from my analysis:

• Coding agents (Claude Code, Codex CLI) — also called the agent harness. They aren’t orchestrators themselves.

• Point-solution tools offer a single specialized capability — code-review tools (CodeRabbit, Greptile), bug detectors (Cursor Bugbot). One scope, one job.

What’s left is the layer that takes an intent — “fix this bug”, “investigate this log spike”, or “ship this feature I’ve already specced” — and runs it through one or more coding agents to completion. Sixteen of them have public details good enough to analyze.

• On-the-market (8): Conductor, Sculptor, Nimbalyst, OpenHands, Warp Oz, Augment Code, Factory Droid, Devin Desktop.

• Internal (8): Stripe Minions, Block Builderbot, Coinbase Mux + NodeSmith, Shopify Roast, Uber LangFX, Meta Confucius, Google’s migration pipeline, Airbnb’s batch migration pipeline.

Findings, by axis

Major findings

Two patterns dominate the dataset - the workflow model and the agent harness support.

Two workflow models

All orchestrators break into two workflow models:

• Handoff orchestrators. Non-deterministic. The orchestrator hands work off to the LLM via a single prompt or skill; the LLM controls 100% of the flow from there on.

• Pipeline orchestrators. Deterministic. The devs design the pipeline as explicit steps, interleaving calls to a coding agent and HITL (Human-In-The-Loop) gates between deterministic execution. The orchestrator runs the pipeline as a state machine.

Here’s the interesting part. The handoff / pipeline split is almost perfectly the build/buy split.

Internal platforms cluster on the pipeline side. Stripe Minions, Shopify Roast, Uber LangFX, Meta Confucius, Coinbase NodeSmith, Google’s migration pipeline, Airbnb’s migration pipeline — all of them. Stripe’s Blueprints alternate deterministic and agentic nodes; Shopify’s Roast is a Ruby DSL with named cogs; Meta’s Confucius is a DAG with HITL gates between stages. The pipeline languages vary — Ruby (Roast), LangGraph/Python (LangFX), bespoke DSL (Minions), state-machine-in-code (Airbnb) — but the shape is uniform: deterministic steps with LLM and HITL nodes interleaved. The rest fit the same shape — see appendix for the per-tool teardown. Block’s Builderbot and Coinbase’s Mux look like handoff at first glance but sit on top of internal pipeline infrastructure; even where internal orgs ship handoff dispatch, what’s underneath is pipeline.

On-the-market orchestrators cluster on the handoff side. Conductor, Sculptor, Nimbalyst, Warp Oz, OpenHands, Devin Desktop. Each dispatches an agent and lets it figure out the steps.

Two on-the-market products break the handoff pattern. Factory Droid ships Custom Droids you author; Augment Code ships a coordinator + specialist + verifier architecture with explicit scope-approval HITL. Both target enterprise structured workloads. Both expose real pipeline design.

Agent harnesses

How does the orchestrator relate to the coding agent it dispatches? Three patterns:

• Proprietary agent harness. The orchestrator ships its own. Block built Goose, Meta built Confucius, Stripe forked Goose into Minions. On-the-market, Factory Droid runs Droid and Augment Code runs Auggie underneath. The org owns and maintains its own agent harness and uses raw LLM models from LLM vendors.

• BYO direct. The orchestrator dispatches a third-party agent harness untouched — native execution, no interception. Internally, Airbnb runs Claude Code as-is and Coinbase mixes harnesses through Mux. On-the-market, Conductor, Sculptor, Nimbalyst, OpenHands, and Devin Desktop all pass the chosen harness through; ACP (the Agent Coordination Protocol) is the standard underpinning the latter two.

• BYO wrapped. The orchestrator runs the chosen agent harness inside its own context/memory layer. Warp Oz adds “Agent Memory” and “Skills” on top of execution; Augment Code layers a structured context including system prompts and workspace metadata. The wrapper can degrade the inner harness’s context — Augment Code’s own docs explicitly disclose the added context. Verify before depending on it.

Internally, six of the eight platforms in the dataset have consolidated on one agent harness across the org: Goose at Stripe and Block, Claude Code at Anthropic and Airbnb, Confucius at Meta, LangFX at Uber. Polyglot agent harnesses are rare inside.

On-the-market, the 8 products split into 5 BYO direct, 2 BYO wrapped, and 1 Proprietary (Factory Droid).

Supporting patterns

Smaller findings that reinforce or color the major split.

HITL surfaces

How do humans interact with the orchestrator? That’s mostly settled - Slack, Linear, etc. The chat-in-dashboard anti pattern is essentially dead.

When do humans interact with the orchestrator? That’s a deeper question with three approaches in the wild:

• Confidence-driven. The agent grades itself, a verifier sub-agent grades (Meta KernelEvolve, Augment Code’s verifier role, Devin’s auto-review), or the user sets sensitivity dials (Cursor Bugbot effort levels).

• Single-gate. One HITL event, otherwise autonomous. Two variants worth distinguishing: upfront plan-approval (Cursor’s long-running agents, Augment Code scope approval, Devin’s preflight checklist) and exit-only PR review (Copilot Coding Agent, Stripe Minions, OpenHands headless). Upfront catches direction errors before compute spend; exit reuses existing review process.

• Hook-driven. The adopter writes scripts or config that fires at named execution milestones — pre-plan, pre-tool-call, post-tool-call, pre-commit. The script can halt the agent (exit 1), modify state, or run guardrails. OpenHands’ hooks.json is the canonical declarative-registration version; Claude Code ships its own hook system. Hooks are the lowest-level, most flexible HITL surface — you write code and it runs.

The three approaches aren’t equivalent in coverage. Confidence-driven is opaque to the adopter — you tune indirectly by enriching context, not by setting thresholds. Single-gate is binary — one ask, then nothing until the PR. Hook-driven is fully programmable but pushes the work onto the user and offers no built-in human-prompt UI.

Loop types (and the new campaign loop)

All sixteen orchestrators target outer-loop work — the developer hands off and checks back. Inner-vs-outer isn’t an interesting axis in this dataset; orchestrators exist for outer-loop work by definition. What IS interesting is a third loop the binary missed: the campaign loop.

Campaign-loop work is autonomous multi-day-to-multi-week execution against a goal. The developer doesn’t check back in any meaningful cadence; the orchestrator owns long-running state.

• Uber Shepherd runs multi-day migration campaigns across services.

• Meta REA has a “hibernate-and-wake” pattern that bridges multi-day training waits inside one logical run.

• Factory Missions have a median around two hours, with 14% running over 24 hours and the longest recorded at 16 days.

• Google’s migration pipeline has run multi-month migrations across 149 teams.

These aren’t outer-loop with a slow review cadence. The orchestrator is driving goal-with-stopping-criteria rather than a task-with-handoff. What makes the loop distinct is the durable state — these orchestrators persist work across host / service boundaries because no single machine outlives the campaign.

Most orgs aren’t close to needing campaign-loop tooling, so I won’t focus on it for the rest of this post.

What else stood out

• Repo harness. Every orchestrator in the dataset respects the standard repo-harness files — AGENTS.md, CLAUDE.md, .cursorrules, and friends all coexist. No tool earns the “imposes” verdict.

• Audience. Individual-developer tools (Conductor, Sculptor, Nimbalyst, Warp Oz, Devin Desktop) all sit on the handoff side. Org-wide tools (Factory Droid, Augment Code, all internal platforms) all sit on the pipeline side.

• BYO harness orchestrators are a real 2026 category. Most on-the-market products now let the user swap agent harnesses. ACP and broad MCP adoption made the swap technically straightforward by mid-2026 — with the caveat that wrapping can silently degrade the inner harness.

The synthesis

• The market is bifurcated along the build/buy boundary. On-the-market: handoff dispatch with BYO harnesses (7 of 8 products). Internal: pipelines with Proprietary harnesses (6 of 8 platforms).

• Internal engineering teams are using this framework’s vocabulary. Coinbase explicitly segments inner loop vs. outer loop in its own engineering docs. Anthropic and Shopify use the same distinction internally. Three independent organizations adopted this mental model before my framework post was published.

• Factory Droid & Augment Code are the only on-the-market pipeline orchestrators. But neither supports Claude Code or Codex CLI. Factory Droid runs Droid — its own agent harness (you can use any LLM model). Augment Code runs Auggie underneath, also Augment’s own agent harness with its own model routing.

This leaves us with some questions…

The insights from this research left me with some new questions:

• Why does the handoff/pipeline bifurcation track the build/buy boundary so closely? Is it just headcount — bigger orgs build internal platforms — or is something else pulling the architectures apart?

• Similarly is Proprietary agent harness vs. BYO here to stay? Are Proprietary harnesses beneficial? Will they survive?

• What would be an ideal orchestrator that will scale for the next 3-5 years? I don’t think any of the on-the-market orchestrators are it. So what would it take?

More on this in a follow up soon!

Addendum

The matrix and vendor blurbs below is the raw data behind the synthesis. And all tools are linked to their primary source.

How to read the matrix

• Build/buy — On-the-market (commercial or OSS) product vs. internal platform. The build/buy split is the central organizing distinction in the post.

• Workflow model — Handoff = one prompt, agent controls the flow. Pipeline = explicit deterministic + LLM steps wired by the adopter.

• Agent harness — Proprietary = built and owned by the org/vendor. BYO direct = third-party harness run as-is. BYO wrapped = third-party harness inside a vendor context/memory layer.

• Model — Open = any LLM, BYOK. Fixed = vendor picks.

• HITL — Three approaches from the post. Confidence = agent grades itself. Single-gate = one HITL event, otherwise autonomous; variants noted as (upfront) plan-approval or (exit) PR review. Hook = adopter-authored scripts at named milestones.

• Audience — Individual = solo dev installs. Org = deployed as a platform. Both = packaged for either.

The matrix

On-the-market tools

Conductor (commercial)

Free macOS desktop app (Apple Silicon only) from a YC S24 company that raised a $22M Series A. Spawns parallel Claude Code / Codex CLI / Cursor sessions in isolated git worktrees, each with its own branch, terminal, and Diff Viewer. Adds no agent logic of its own — thin orchestration shell over the underlying harnesses; users bring their own subscriptions. The canonical “outer loop defined by attention, not execution” example.

• conductor.build

• conductor.build/docs/concepts/workflow

Sculptor (OSS)

Open-source Mac/Linux desktop app (Electron) from Imbue. Wraps Claude Code and Codex in Docker containers (vs Conductor’s worktrees) for tighter isolation, with a Pairing Mode that bidirectionally syncs container files into the developer’s IDE for live co-editing. Explicitly labeled “experimental research preview”; MCP and Sculptor-layer CLAUDE.md handling still roadmap.

• imbue.com/sculptor

• github.com/imbue-ai/sculptor

Nimbalyst (OSS)

Stravu’s MIT-licensed successor to Crystal (deprecated Feb 2026). Cross-platform desktop (Mac/Win/Linux + iOS companion) for parallel Claude Code / Codex / OpenCode / Aider sessions in isolated worktrees. Adds a session kanban, inline red/green diff review, and an extension SDK with ACP alpha for Copilot. Very young (open-sourced April 2026); team tier not yet GA.

• nimbalyst.com

• github.com/nimbalyst/nimbalyst

OpenHands (OSS)

All Hands AI’s MIT-licensed agent platform, 78K+ GitHub stars. Docker-sandboxed CodeAct agent with LiteLLM model abstraction (100+ providers). Five surfaces: Agent Canvas GUI, headless CLI, GitHub Action triggered by fix-me labels, Slack, and ACP for IDE embedding. The cleanest external zero-chat outer-loop path; .openhands/hooks.json is the canonical declarative hook-driven HITL primitive in the dataset.

• openhands.dev

• docs.openhands.dev/openhands/usage/run-openhands/github-action

• docs.openhands.dev/openhands/usage/customization/repository

Warp Oz (commercial)

The first multi-harness cloud-agent control plane: runs Warp Agent, Claude Code, and Codex side-by-side under unified governance and credit billing. Triggers via Slack @Oz, Linear issue assignment, GitHub Actions, cron, or webhooks; HITL surfaces are native to the triggering tool. Internal stat: 60% of Warp’s own PRs by Oz agents. Warp terminal open-sourced May 2026.

• warp.dev/cloud

• warp.dev/blog/multi-harness-cloud-agent-orchestration

• warp.dev/blog/oz-orchestration-platform-cloud-agents

Augment Code (commercial)

Three-surface platform — Cosmos (IDE plugin), Auggie CLI, Intent (macOS desktop orchestrator) — anchored on a proprietary Context Engine indexing 400K+-file monorepos. Intent’s BYOA wraps Claude Code / Codex / OpenCode in a coordinator + specialist + verifier architecture with explicit scope-approval HITL. Warning: BYOA agents get degraded Context Engine access vs first-party Auggie, Augment’s own docs disclose it.

• augmentcode.com

• augmentcode.com/blog/intent-a-workspace-for-agent-orchestration

• docs.augmentcode.com/context-services/mcp/overview

Factory Droid (commercial)

Multi-specialist outer-loop platform on a proprietary harness (Droid). A Delegator decomposes work and dispatches to Code / Review / Knowledge / Reliability / Product / Tutorial Droid sub-agents. Flagship “Missions” feature is the only external product with campaign-loop duration: median ~2hr, 14% over 24hr, longest recorded 16 days. $1.5B valuation April 2026; reads AGENTS.md only (Factory co-authored the standard with OpenAI).

• factory.ai

• factory.ai/news/missions

• docs.factory.ai/cli/configuration/agents-md

Devin Desktop (commercial)

Cognition’s June 2026 rebrand of the Windsurf acquisition. Agent Command Center kanban manages local + cloud + third-party (ACP-hosted Codex / Claude / Gemini / OpenCode / Junie) agents from one surface. The fleet-management piece of Cognition’s product line, separate from Devin Cloud (excluded as a coding agent, not an orchestrator). Devin 2.1’s confidence scoring (🟢🟡🔴) is the canonical confidence-driven HITL.

• devin.ai

• cognition.com/blog/introducing-devin-desktop

• cognition.com/blog/devin-2-1

Internal

Stripe Minions

The prime empirical example of the post’s thesis. Engineer tags a Slack bot → pre-warmed AWS devbox containing Stripe’s 30M-line monorepo spins up in under 10 seconds → Toolshed (Stripe’s ~500-tool MCP server) provides context → a Blueprint alternates deterministic steps (git, lint, CI) with agentic LLM nodes → PR opens, CI green, no human in the loop. 1,300+ PRs/week merged. Built on a fork of Block’s Goose with interruptibility and confirmation prompts deliberately removed.

• stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents

• stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2

Block Builderbot

Block’s internal orchestration layer on Goose (which Block built and donated to the Linux Foundation Dec 2025). Vertically integrated stack: Anthropic Claude → Block Goose → Anthropic MCP → Block Builderbot. Slack-native @builderbot dispatch + Linear/Jira pickup; live thread updates while the agent works across the monorepo (Cash App, Square, all subsidiaries). ~1,500 PRs/week merged, ~15% of all production code changes.

• block.xyz/inside/block-rolls-out-builderbot

• github.com/block/goose

• github.com/block/builderbot

Coinbase Mux + NodeSmith

Coinbase explicitly segments inner vs outer loop in its own engineering docs — direct framework-vocabulary match. Forge is the Slack/Linear/GitHub-native task agent (1,000+ engineers, 5% of merged PRs, 150h→15h cycle time). Mux layers parallel-agent orchestration on Forge (600+ users, 5,068 merged PRs across 461 repos in one month). NodeSmith is a two-phase pipeline for blockchain node upgrades — LLM Triage Agent + Upgrade Orchestrator with 5 specialized sub-agents blending LLM reasoning with deterministic Python. OpenAI-compatible model router used daily by 1,500+ engineers; explicitly multi-vendor.

• coinbase.com/blog/coding-had-a-concurrency-problem-how-mux-helped-solve-it

• coinbase.com/blog/NodeSmith-AI-Driven-Automation-for-Blockchain-Node-Upgrades

• coinbase.com/blog/Tools-for-Developer-Productivity-at-Coinbase

Shopify Roast

Two-layer stack. Genie is an internal LLM proxy that sits under every AI tool Shopify uses (Claude Code, Copilot, Cursor, Codex) — PII masking, prompt-injection detection, cost analytics, model switching, plus 24+ internal MCP servers exposing Vault, GSD, data warehouse, Salesforce, Slack, Google Workspace, GitHub, Figma. Roast is the Ruby DSL workflow framework on top: named cogs (chat, agent, ruby, cmd, map, repeat) interleave deterministic code with LLM calls. Open-sourced June 2025. Tobi Lütke’s April 2025 memo: “reflexive AI usage is now a baseline expectation.”

• shopify.engineering/introducing-roast

• github.com/Shopify/roast

• bvp.com/atlas/inside-shopifys-ai-first-engineering-playbook

Uber LangFX

Full-stack internal AI dev suite for ~5,000 engineers on a multi-hundred-million-line monorepo. Three layers: end-user tools (uReview, AutoCover, Minion, Shepherd, Validator), agent framework (LangFX — opinionated LangGraph wrapper integrated with Michelangelo), infra (AIFX CLI, MCP Gateway exposing 10,000+ internal services as MCP tools, Agent Builder). 60K agent tasks/week. 92% of devs use agents monthly; 11% of PRs by agents. Multi-vendor model strategy benchmarked per task (Claude-4-Sonnet + o4-mini-high for uReview was optimal).

• uber.com/blog/ureview/

• newsletter.pragmaticengineer.com/p/how-uber-uses-ai-for-development

• aaif.io/blog/how-uber-runs-60000-ai-agent-tasks-per-week-with-mcp/

Meta Confucius

Specialist-per-domain pattern on a shared internal harness (Confucius — ~2 years old, 60+ applications, DAG workflows + HITL gates). REA owns the ML experimentation loop for ads ranking (hibernate-and-wake across multi-day training waits). KernelEvolve owns GPU/MTIA/AMD kernel optimization (60%+ inference throughput on Andromeda; ISCA 2026 paper). Tribal Knowledge Agents produce ~59 concise context files per codebase (40% fewer tool calls per task downstream). Capacity Efficiency owns perf regression detection + fix generation.

• engineering.fb.com/2026/03/17/developer-tools/ranking-engineer-agent-rea

• engineering.fb.com/2026/04/02/developer-tools/kernelevolve

• arxiv.org/abs/2512.23236

Google migration pipeline

Fully vertically integrated stack: Cider V (VS Code fork on Borg, indexes all of Google3 monorepo) + Gemini fine-tuned via DIDACT + Kythe code index + Critique review tool as the HITL surface. The migration pipeline uses Kythe BFS for deterministic site discovery, fine-tuned Gemini for the diff, deterministic validation cascade (whitespace → AST → build → tests), then per-CL human review in Critique with OWNERS-routing (effectively protected-code HITL via Google’s existing ownership graph). Example campaigns: int32→int64 proto migration (595 CLs, 149 teams, 12 months); JUnit3→JUnit4 (5,359 files, ~149K LOC, 3 months). Pichai April 2026: 75% of new Google code AI-generated.

• arxiv.org/abs/2504.09691

• arxiv.org/abs/2601.19964

• blog.google/company-news/inside-google/message-ceo/alphabet-earnings-q1-2026/

Airbnb migration pipeline

The “rent the agent harness, build the repo-harness layer” pattern. Off-the-shelf Claude Code as the harness, custom internal MCP servers as the context layer — distinct from Stripe (fork) and Block (own). Two surfaces: engineers running 3–5 parallel Claude Code CLI sessions on the monorepo, plus a standalone batch migration pipeline (per-file state machine: refactor → fix jest → fix lint → fix tsc → complete; LLM invoked only on gate failure) that migrated ~3,500 Enzyme test files to React Testing Library in 6 weeks (97% automated, 3% manual long-tail). Chesky Q1 2026: 60% of new code AI-written.

• airbnb.tech/infrastructure/accelerating-large-scale-test-migration-with-llms/

• dpe.org/sessions/szczepan-faber-mike-nakhimovich/agentic-coding-at-airbnb

• techcrunch.com/2026/05/08/airbnb-says-ai-now-writes-60-of-its-new-code/

I feel that there’s a pattern forming in how cutting-edge engineering orgs use AI tooling to speed up the dev process.

Two years ago the debate was about models. While models are still important, the LLM harness (this is what Claude Code and Codex CLIs are — a harness around LLMs) is becoming an important differentiator. Most cutting edge companies have settled on Claude Code or Codex CLI as their primary LLM harness.

Some quick definitions before we move on.

• Coding agent harness — the deterministic logic around the LLM itself (Claude Code, Codex CLI). The planning loop, the tool calls, the memory architecture.

• Repo harness — CLAUDE.md / AGENTS.md (with other repo documentation), custom slash commands & skills, linters, automated tests, MCP servers.

Looking at the AI tooling market today, newer products are clearly converging on the repo harness + outer-loop orchestration pattern. Older products — and in the age of AI, “older” means over a year — are more piecemeal, built as standalone vertical features. Walking through the segments:

• Code review segment - Code Rabbit, Qodo, Baz, etc. First-generation products, each with their own LLM, prompting, and harness. My read is that this category is gradually being absorbed into the outer-loop implementation tools below.

• Implementation segment - Devin, GitHub Copilot Coding Agent, OpenHands, and others. This is where the convergence is most visible — these tools have grown meaningful repo harness support (AGENTS.md and similar) and outer-loop orchestration primitives, though each still ships its own agent harness. A few tools let you delegate to Claude Code or Codex (Conductor, OpenRig), but they’re aimed at individuals running the tool on their laptop. OpenHands recently added the option to run frontier coding agents via ACP.

• Fix code review findings segment - many CI tools, Linear, etc are all jumping into “press a button and fix your findings”. This is the one segment where I think the bet is wrong — piecemeal features running in proprietary agents, disconnected from the dev’s primary loop. Not what developers actually need.

The bigger story, though, is what the companies that have figured this out are building themselves. Stripe, Shopify, Uber, Airbnb have all talked publicly about their internal AI dev platforms. These orgs are not bolting on piecemeal vendor features — they’re building full outer-loop orchestration that integrates one agent harness across local dev, cloud agents, and review. Smaller cutting-edge teams I’ve talked to are doing the same (but they don’t post this online). The reason is always the same: integrate the agent into your process, don’t import someone else’s.

A couple more definitions for the AI development process. The loops are best defined by how often the developer has to check in with the AI. Note that where the loop runs (on your laptop, in the cloud) is not important - it’s the interaction model between the builder and AI that is the defining factor.

• Inner loop — conversational and real-time. Check-ins happen in seconds to minutes. Even when you ask the agent to do research or a small implementation and wait a few minutes for it to come back, that’s still part of the conversation. Typically this is your laptop (but not necessarily): ticket/task definition & architecture, exploration, smaller implementations.

• Outer loop — not conversational. Check-ins happen in 30-minute to multi-hour increments. The agent runs, hits a gate, asks for input, and runs again. Longer implementation tasks, PR reviews & fixing findings, investigating production incidents & error spikes, recurring tasks (security/architecture drifts, analyzing new CVEs, etc).

• Campaign loop — also HITL-based, but the goals span days to weeks. Think: “migrate this service off the old framework,” “harden auth across the platform.” This is a much more advanced concept and the industry is barely starting to figure it out. I won’t focus on it here, but it’s worth naming because it’s coming.

The core principle: same agent harness, same repo harness, and same context across both loops. And every handoff between inner and outer should be warm-context.

There’s one more axis worth getting right: the interaction model for outer-loop agents. Platforms like OpenHands offer both — workflow-style entry points (GitHub labels, Slack mentions, CLI) and a conversational dashboard you can sit in front of. The principle I lean toward: conversation is what the inner loop is for. If I’m going to iterate with an agent in natural language, I’d rather do it on my laptop, where the full repo harness is loaded and the feedback loop is tight.

The outer loop calls for something different: deterministic workflows that incorporate AI as a step, with explicit HITL gates at the surfaces where work already happens — GitHub PRs, Linear tickets, Slack threads. The market is moving this way — Devin Playbooks, GitHub Copilot’s .github/agents/ profiles, OpenHands hooks and skills are all takes on this.

An outer-loop code review workflow could look like this:

• On PR open, AI runs the initial review and posts findings as inline GitHub comments.

• AI responds to developer questions and “skip this” instructions through the GitHub UI — same surface, no new tool.

• AI confirms “this is fixed” comments and does incremental reviews on new commits.

• AI auto-approves the PR when conditional gates pass: all findings addressed, risk below threshold, no sensitive modules touched.

One honest caveat on the outer loop: the framework is sound, but the devil is in two related problems. First, HITL design — you can’t just let these agents run unattended, but where exactly do you build in the gates? When should the agent stop and ask vs. push through? The market is actively exploring this — confidence-driven (Devin’s self-grading), single-gate (Copilot’s PR-only review), hook-driven (OpenHands milestones) — each with different trade-offs. Second, context — how does context carry across HITL moments and across the agent’s run so it doesn’t lose the thread between gates? Both topics are meaty enough that I’ll dig into them in a follow-up article.

To summarize - there are three concepts to take away:

• LLMs ≠ agent harness ≠ repo harness. The model is the engine. The agent harness (Claude Code, Codex CLI) is the loop around it. The repo harness (CLAUDE.md, skills, MCP servers, tests, linters, etc) is the project context that makes either useful.

• Three loops, defined by check-in frequency. Inner loop is conversational, minute-scale — your laptop. Outer loop is HITL-based, 30-min to multi-hour — cloud agents on bounded tasks. Campaign loop is HITL-based, days to weeks — advanced territory, not the focus here. The agent harness, repo harness, and MCPs should be identical across inner and outer.

• Conversation in the inner loop. Workflows in the outer loop. Chatting with an agent on your laptop is the right inner-loop UX. Cloud agents should run deterministic workflows with AI incorporated & HITL gates at the surfaces where work already happens.

The market is still organizing around these distinctions. If you’re building or evaluating AI dev tooling, these three concepts are useful axes — both for thinking about your own stack and for the conversations you have with your vendors.

The second day after I joined a new company one of my new reports burst into tears at a seemingly innocuous “How is it going?” at the beginning of our first one-on-one. Though I was vaguely aware (it was my second day) that there were some bad vibes on the team, this emotional reaction caught me completely by surprise.

The story

I was not prepared for what had happened at the one-on-one and I almost panicked in the moment. Luckily, I remembered a management article that talked about similar situations and how it’s important to just listen. So I focused on listening and empathizing without trying to jump to solutions. It took me a few more meetings to better understand the whole picture.

This person was an engineer with about four years of experience, who had to deal with very direct and matter-of-fact feedback from one of the more senior engineers. In addition, she frequently felt talked over, disrespected, and never listened to in general. I’d like to add (as a side note), that this is a somewhat common problem in engineering where juniors (who are naturally less confident in their tech skills) feel disrespected when faced with a lot of direct feedback. And being a female in tech is also a factor to be mindful of.

After chatting with the senior engineer, I came away with the feeling that he didn’t mean to be harsh or disrespectful, but at the same time, it was obvious that he could improve on how he delivers his feedback. Over the next couple of months, I worked with him on adding empathy and understanding to his communication style.

This helped some and at the same time, I started to coach the junior on how to receive critical feedback and on some of the work processes that she could leverage to overcome conflict. For example, instead of waiting for feedback during a code review (which may ask the author to rewrite significant portions of the code), I pushed the junior developer to seek design feedback earlier (design reviews, etc).

Over time I also introduced several other process changes like:

• Everyone gets a chance to speak during group discussions (as opposed to only the loudest voices in the room).

• Ideas are debated around their pros/cons and trade-offs instead of around being good or bad.

• If a debate gets too heated (we’re all human after all), take a break and come back to it in a few hours or the next day. It has the additional benefit of being able to sleep on the ideas and get a better or a new understanding of them.

Our regular coaching sessions and my willingness to listen helped the junior engineer improve her skills, but more importantly, gain trust and confidence in herself and the team. She became more comfortable with professional conflict and debating various engineering decisions. She also learned to be less and less stressed while doing so.

There was an occasion that I’d like to highlight where I basically had to pull rank. I and another very senior engineer had to override the junior’s technical idea on a fairly complex topic after a long and heated debate. I had a follow-up chat with her and we chatted about how this made her feel like a junior. Towards the end of the conversation (and after listening), I pointed out that the very senior dev and I have 40 years of experience combined vs. her five years. And while I don’t like using this as an argument, sometimes it’s very hard to explain the totality of your experience and why it necessitates certain decisions (but you should try anyway). That conversation (including my remark) ended well and I attribute that to the trust I’ve built up to that point.

Starting with our initial one-on-one, the junior engineer made steady progress and over time became one of the star performers on the team who wasn’t shy of getting into debates and pushing for her opinion (in a productive kind of way). I was proud of her accomplishments and, at a later date, she went on to work for a major Silicon Valley company.

What I’ve learned

Writing this blog, it feels like some of these lessons are obvious and maybe even trite. And yet, in the moment, it can be hard to remember them and apply them in practice. I hope that this little summary is at least a helpful reminder.

• Listen. You don’t have to try and fix “it” now. Frequently the important part is to listen and empathize. Which helps to build trust in the longer term.

• Do problem solve. But at a later time and day, after you’ve listened, established some trust, and gathered feedback and context from multiple sources.

• Delivering feedback well can be learned. Like all other soft/people/EQ skills it’s not the easiest to learn but it’s possible. You can be both direct and empathetic at the same time (radical candor ftw). Invest time in teaching developers how to do it.

• Set good (equitable) team practices. Everyone needs a chance to speak, decisions need to be transparent, etc. This is helpful for everyone, not just juniors.

• Pulling rank is ok. But only very occasionally. And only after you discuss the decision first and explain why you chose to pull rank in the end. And that assumes that you’ve already built some trust that you can rely on.

• Personal change takes time. This whole story took place over a two year period. As a manager you need to be patient and supportive throughout.

Like many people in the tech space, I am biased to action. It is a generally a great trait that helps to move forward and get things done. And it just feels good! Being a manager I also feel the extra need for speed since projects inevitably slow down when more and more people get involved.

It’s a trait that I was naturally imbued with and that I further developed through my career. It’s a resume builder and something that’s typically sought after in managers.

what is minimum timeboxing?

Yet, there are situations and problems where deliberately slowing down is necessary. It is necessary to ensure you can evaluate all the ideas. It is necessary so you can “sleep on it” and let it brew in your head. It is necessary so you can reflect on the problem itself (in the words of my manager “to live in the problem space”). It is necessary so others who are involved can reflect on it on their own timeline.

So how do you do this without compromising your core bias to action? How do you reconcile the two? For someone with a bias to action it is hard to be still and not act. I know it is for me.

Minimum timeboxing is a technique I came up with recently to address this. Timeboxing, a well established personal productivity and project management technique, helps to move things along. It helps to be more biased to action by limiting how much you spend on a decision.

Minimum timeboxing, takes the same approach but defines the minimum time you must spend on a decision. In fact, you can combine the two and specify both the minimum and the maximum time.

where and how do you use it?

There isn’t more to this technique — you can start using it right away. But I want to dive deeper into why and when it is important to sometimes not move fast, especially in times when you feel the extra push.

A great place to start is with the classic decision matrix below.

The decisions that are high impact and hard to undo should definitely have a minimum timebox. The ones in the upper left and lower right quadrants should be seriously considered as well, while the low impact / easy to undo are good candidates for moving fast.

Decision matrix aside sometimes there is urgency around problems but before you jump into action take some time to think where this urgency comes from. While some decisions are truly urgent, you should ask yourself is the urgency real? Is it self-generated due to your stress / bias to action / subconscious want to just-be-done-with-it?

If you are a problem solver like me and like to move fast, there is a good chance that you are feeling these urges. On the other hand, could they be coming from someone else in the organization who is dealing with similar emotions?

Using minimum timeboxing is a way to give yourself a chance to get immersed in the problem and to ensure that you don’t jump to a rushed decision.

P.S. I’m sure someone somewhere already thought of this idea but I personally haven’t heard of it.

If you’re an engineer, check out this article by Rich Hickey (creator of Clojure) that is on a very much related topic — Step Away from the Computer or Hammock-driven Development.

Let me start by saying just how much I enjoyed the entire trip. The city of Denver. The local startups. The hosts. The itinerary. And my fellow participants.

The goal of the Denver Startup Week is to promote the local startup community as well as the city of Denver at the national level. It’s been running for seven years straight now. The Ambassador program, however, is only two years old and was created to promote Denver as a high tech hub and a viable alternative to the Valley both for work and high quality living. By partnering with Denver startups and more established companies (thank you Southwest Airlines) the program brings 50 Ambassadors from across the country to experience Denver and network with the local startup community.

Let me start with sharing my impressions of Denver. Despite having been to the Denver Airport only about 50 times (I’m an avid skier and midwest just does not cut it), I have never managed to visit Denver itself. I was always focused on maximizing my slope time and I was excited to finally roam around Denver. And I liked what I saw. Really liked! Living downtown Chicago I am very much a city person and Denver certainly catered to my taste. It’s a city, it’s got a city vibe, it’s got graffiti (the artistic kind), it’s got coffee shops, restaurants, rental bicycles and scooters (first time I saw those), and it’s got people.

It’s more laid back and more friendly than Chicago, it shuts down earlier, and it’s just noticeably smaller. But it’s got all the right parts and it’s growing! About 10,000 people move to Denver monthly. It’s also the sunniest city in the States with 300 sun days a year and, contrary to what you might think, it’s warm with the average winter temp way above Chicago’s.

And Denver has mountains. Mountains with some of the best snow on this planet.

I was also surprised by how developed the startup scene is in Denver. We had the opportunity to have at length meetups with several startups. We heard presentations and talked with CEOs and other executives, techies, marketers — you name it. Given an opportunity, I would love to work with any of these people. And it’s not just the number of startups, it’s the scale, and the professional experience that I was impressed with.

One thing I would like to call out specifically is the gender diversity that stood out to me. It felt to be more mixed than Chicago and that’s something that I hope we in Chicago can catch up with soon.

While Denver might not be for everyone, it’s great to see that Valley’s monopoly on tech and tech startups is beginning to wane.

Thank you Denver Startup Week for the wonderful experience!

When we plan a roadmap several quarters or a whole year ahead what are our expectations (and those of the leadership team)? Frequently the implicit and subconscious expectation is that everything will get done. After all if we use good scoping techniques we should be able to hit every deadline. Right?

Not quite. We might choose to extend the scope of some projects. We might remove some projects and add others, with different scoping, all due to market demand. And what about that quick marketing change that’s so needed but wasn’t planned for or an internal tools improvement that will shave off 20% of operational costs from your account management team? If our plans are too rigid we won’t be able to react to the outside changes and might miss good business opportunities.

Another reason why it’s typical to underdeliver on the roadmap is the one-sided variability in planned dates. A team that’s already pushing hard will almost never deliver projects ahead of time (but will sometimes run over the original plan). So whatever variability there is will add up and will eventually mean that a project or two won’t get done.

A good way to look at long term roadmapping is the same way as any other goal setting exercise. OKRs and EOS, two popular organizational methodologies, are both heavy on settings goals and both say that you should aim at about 80% achievement. Because if you hit 100% of your goals they weren’t ambitious enough and you didn’t push yourself (or your org) hard enough.

However, let’s not walk away from this thinking that roadmap planning is useless. On the contrary, we should deliberately plan roadmaps that are a (reasonable) stretch and push ourselves to deliver as much as possible. Like we do with all other goals.

“What if the customer clicks on this button, sees the popup, types some text, then cancels, and later comes back to the popup — do we preserve the previously typed text?”

This is a typical conversation that happens during product / design / technical discussion on software engineering teams. Inadvertently, the answer is an enthusiastic “yes”. Everyone wants to deliver an awesome product that’s thought out through and through. And therein lies the danger — we get carried away and blow up the scope of work.

According to Mary Poppendieck’s “Leading Lean Software Development”, 60% of released features are never used by end users. Extensive user research, user interviewing, and such helps to narrow things down, however it’s an imprecise science. Users will typically ask for everything (because why wouldn’t you), but they can’t tell us what’s a must have vs. a nice to have. They may genuinely believe that they need a certain feature only to never use it when it comes out.

For many businesses it’s more important to deliver a core set of functionality and move on than to spend time on fluffy features and questionable edge cases that might never be used. It’s also good to remember that if we end up missing some needed feature we can always add it later.

In project management there is the holy trinity of scope, time, and quality and we can only choose two out of the three. Cutting scope is an underused technique that needs to be talked about more. Cutting quality is not a good idea (modern users expect top notch products) and so is making the team work consistent overtime (lest we want to lose team productivity and potentially the team itself).

It frequently seems impossible to cut scope but if we really put our mind to it, we can find a few minor features or edge-cases that will shave 5–15% off of scope. Done consistently this will increase the average team productivity and allow us to kick out more projects. Moreover, we can timebox our projects, which will let us deliver all (or almost all) of the important, high visibility features in our yearly roadmap.

project structuring

Given the unpredictable nature of software engineering, how do we know when and what scope to cut? Maybe we can fit everything into the given project’s timeline? The truth is that we do not know upfront.

As we start working on a project, we need to first deliver the must have features. As we get closer to the end of the timebox, we can discuss the remaining features and make an informed decision on which ones we are able and want to finish and which ones we can drop.

There is a wealth of information on this technique available online, known as vertical slicing in Agile circles.

“I am completely satisfied with how fast our engineering team delivers and how well they hit the deadlines,” said no CEO ever.

We all want to deliver faster and hit those upfront estimates or deadlines. But is that a goal that we want to impose on engineering teams? Is that a goal that benefits the business as a whole? To understand this issue we need to look at it in terms of cost / benefit analysis and we need to delineate the difference between productivity and predictability.

Productivity vs. predictability

Productivity is simply how fast can a team deliver. Predictability is how accurately the team can estimate project timelines. While productivity is largely a factor of team members individual skills and the overall team cohesion (and team size), predictability more so depends on external factors. Software engineering is inherently unpredictable due to a multitude of possible unforeseen technical issues and requirement gaps discovered in the middle of project implementation. Sales, marketing, contract negotiation, and other business functions are similarly unpredictable — there are too many external factors that are impossible to account for upfront. If we really think about it, life in general is quite unpredictable.

But of course a business cannot operate without some degree of predictability. Instead of looking at it as a binary — can we hit deadlines or not — we need to look at it as a range of how much we are willing to pay for an extra degree of certainty.

So what is the cost of predictability? Unfortunately the cost of predictability is productivity. For a project to have more predictable timeline it needs to be discussed, designed, prototyped, and researched extensively before an estimate can be produced. That work takes away from the current project being worked on. It is tempting to think that all this upfront work needs to be done for the new project anyway and as such goes towards its eventual implementation, however this is only partially true. While some of this work is reusable, some inadvertently is not. The biggest hit, however, comes from context switching the engineering team, which happens twice. Once, as the team switches away from the current project and another time switching back to the current project. Moreover, by the time the new project starts many findings from research have faded from memory and need to be re-learned causing more wasted time.

The good news is that if we graphed time investment (i.e., productivity) vs. predictability it would look something like the graph on the left. What that means in practice is that we can get a good level of predictability for a moderate time investment, however if we want precise dates it will require a larger time investment and a larger hit to productivity.

Roadmapping & project management

Now that we understand the inverse relationship between productivity and predictability we can make informed business decisions. Some projects (and some business domains) require more predictability (e.g., contractual obligations, date related business opportunities, etc), however for most e-commerce businesses a week or two of difference on individual project delivery don’t matter. What matters is how quickly on average projects are delivered (e.g., how many projects per year a team can complete).

What we typically want to optimize for is quicker average delivery with acceptable predictability. A mature software team can predict projects with 50% accuracy with minimal time investment while still focusing on the project at hand. For those few projects that do require hard dates, we can make exceptions ahead of time and plan on doing more research and design.

With this approach in mind, the engineering team can move fast with the acceptable level of predictability. Some projects are on time and some take a week or two longer — not a big deal for each project— but over time the weeks accumulate and we soon realize that there is no way we will get to the last 2–3 projects on our yearly roadmap. And of course demanding more predictability will only backfire.

So what do we do? Take a look at another article titled “scope & on time delivery”.

In Part 1 of this series, I covered why it’s important to be aware of your product’s health and how you can do so. I covered the user experience and technical types of metrics and how to get started with collecting them. In this part of the series I will walk you through how to use all these metrics to maximize the user experience and product uptime.

Metrics, Metrics Everywhere

There are no silver bullets here and you have to be aware continuously of your product’s health and how it fares against user expectations. It surely takes time and effort, but it’s an essential part of building a good product. Imagine if chair designers didn’t care for how people use their creations; those chairs would not be comfortable to sit in. Similarly, understanding user needs is a paramount part for any software engineer.

Once you’ve established the basic understanding of your application patterns, you are not done. Every release is an opportunity to introduce more latency or failures and the baseline is ever-shifting. It is easy to significantly affect the latency and as easy to break an edge case and drive user failures up by a few percent with any release. The only way to ensure high-quality user experience is to be aware of your product’s health and your users’ expectations at all times.

This is the case where less is more and users prefer fewer features that work and are responsive. Try it out and see what your users say!

User Experience Monitoring

The first part of a good user experience is low error rates. For web products, these include all http error responses as well as all application errors returned to the user. For APIs and GUIs, this includes application failures (and http errors for REST APIs). One recurring comment I hear is that users submit invalid input, which invariably causes error responses and so it should not count. It is certainly useful to be able to differentiate hard failures from bad user input, but the line is more blurred than you think. For example, a new release might change input expectations and previously valid requests can start erroring out. Is that bad user input or did you just break your product?

In either case, if the failure numbers are high, users won’t be happy and you need to deal with this problem. The solution here can be non-technical: Maybe you need better documentation or user training, or maybe the API or UX design is cumbersome and drives misunderstanding and misuse. Whatever the case is, it must be addressed. Users are as likely to walk away from a cumbersome product as from a product that doesn’t work.

As with everything else, I suggest that you identify a few high-level and most important parts of the product and keep track of those metrics. If you do notice an issue, then you have the more detailed metrics on hand for a deep dive.

The other part of good user experience is performance, and the approach here is no different. Typically you need to track the latency of the same user actions as you’re tracking for failures. Performance tracking is where you really need to understand what’s acceptable and what’s not for your users and that depends on the domain and your users’ expectations. Two seconds to open a reservation on a public travel site seems like a long time, but two seconds to pull up a loan record for an internal user at a financial company is not bad.

An important fact to remember is that you will always have some failures in the system and that’s normal. No matter how great your product is, how clear its documentation, and how in tune you are with your users, there always will be some failures caused by bad user interactions. At some point, once you feel good about your product overall, you will have to assume that the given error rate is your baseline. Then you need to pay attention to variation over time and revisit critically every once in a while.

Predictive Monitoring

In addition to user experience metrics, there are key technical metrics that are good to watch, as they can point to upcoming problems. Some of the common ones are CPU and memory utilization, thread and database pool sizes, VM heap size and garbage collection times. Just like with user metrics, you need to know the operational signature of your application. But in addition, you need to understand how changes in these metrics impact your users, if at all.

A growing thread pool might mean that response times are slowing down or it could simply mean that more users are accessing your service. A growing email queue size means that emails aren’t going out as fast, but is it bad? Only knowing your domain and users will give you the answer. If the CPU is hovering at 80 percent everything is fine-for now, but you need to scale up quickly before a tiny addition of traffic kicks your whole app over.

When tracking hardware metrics everyone pays close attention to overutilization because that’s what causes outages. But what about underutilization? Performance fluctuates up and down and at times you can find yourself underutilizing resources. Paying attention to this is a good way to stay frugal and keep costs in check.

Alerting

How do you actually know when something breaks? You can’t stare at the graphs all day long. Once you get a feel for the normal characteristics of your system, you’re ready to create automated monitors and alerts. One of the most important things about automated alerting is that it always must be trustworthy. The moment you and your team start getting false alerts, everyone will learn to disregard some of them and the the value of the system will plummet. Thus I recommend you start with auto-alerting only on several most critical metrics. Run the whole system for a few weeks and slowly add more metrics over time.

Another aspect to be aware of when setting up automated alerting is metric variations over time, the most common one being day vs. night. Nobody wants to get paged because traffic is too low at 3 a.m. At the same time, if your metrics skyrocket or drop suddenly, it can be a cause for concern. Are you being DDoSed or scraped? Is there a network failure that’s preventing your service from being reached?

Tooling to the rescue, DataDog has built-in capability to create alarms, called Monitors. They provide a few different ways to ensure no false positives. For example, instead of setting absolute thresholds, you can create a change alert that fires only on deltas. And adding a new monitor only takes a few minutes.

Finally, someone needs to get paged out when an alert goes off. DataDog can do email notifications but nobody checks that at night and even during core hours it takes time to notice a new email. For critical issues like yours, someone needs to get notified immediately.

Services like PagerDuty and VictorOps have made it trivial to do this. As expected, DataDog integrates with both of them natively. Both services support on-call rotation scheduling with an escalation policy, calling out to phones, text messaging, emails, and more.

Production Troubleshooting

So you’ve been alarmed and need to quickly figure out what’s going on and what to do. Your next stop is typically logs, but looking through log files one server at a time is a sure way to waste quite a bit of time.

Log management tools and services have matured over time and there are good alternatives to Splunk. Loggly and LogEntries cover more than just the basics and provide indexing and search capabilities, graphing of log occurrence, understanding of JSON, and out-of-the-box integration with just about all logging frameworks out there. At their core they provide a quick way to look through all logs without having to go to individual servers. Moreover, you can monitor various aspects of logging à la DataDog and alarm (via VictorOps or PagerDuty) when metrics are off.

The above approach already solves most needs, but I’d like to touch on a more advanced problem, especially for a microservice architecture. When multiple services are involved, finding the failure at the entry point is easy but figuring out which back-end service failed in the call chain while processing the user request can be quite difficult.

An architectural pattern that I call log stitching solves this issue. With log stitching you can pull up all logs across your entire architecture for a given user request. This allows you to reconstruct the complete call chain and quickly find the root cause of failures. Conceptually this technique is straightforward: You generate a unique request ID at the entry point, pass it down the chain for every call, and log it. Now you can pull up all logs with that request ID by searching for it. In practice it can get complicated because you need to: a) ensure consistency across all calls, b) account for all entry points including batch jobs, and c) preserve the request ID internally in multithreaded services.

Log stitching is not something I recommend from the get-go; however, it’s definitely something to consider as your system matures. If you are going down the microservices path, this really becomes a must since tracing call chains get progressively more difficult as the number of services grows.

Putting it All Together

There is not much development work in putting it all together, but nevertheless you will need to put in some time. I’d like to emphasize how important it is to work in iterations. Set up one to three alarms, connect your DataDog to VictorOps or PageDuty, study them for a couple weeks and then move on. Keep in mind that you will not get it right at first; your thresholds will be off and you will misinterpret or misunderstand system behavior.

Operational pattern analysis is not trivial, but don’t spend too much time analyzing the hell out of everything up front. Configure some alarms, get them out there, learn and adjust as you go. When you find an alarm that went off erroneously, take some time to understand why. Specifically, you need to focus on why you thought it was the correct alarm setup vs. why you now know it is not. You will learn by leaps and bounds if you take this systematic approach.

As I mentioned above, configuring Loggly or LogEntries is super easy and you should do that right away. Log stitching, however, is an involved endeavor that requires planning. Due to its nature it only becomes useful when adopted by all, or most, of your systems. Thus, you should only start it when you feel the need and when you have the buy-in from the rest of your development organization.

What’s Next?

Part 3 will cover the topics of continuous integration and continuous delivery, breaking them down into multiple stages of complexity so you can pick and choose what makes the most sense for you from the cost/benefit point of view.

When most people think of DevOps, what comes to mind first are continuous integration (CI) and continuous development (CD) pipelines. While they are arguably the most important parts of DevOps, they are not all of it. So, then, what is DevOps? It is a philosophy of how to build and operate software products and systems to provide the best user experience. It’s a way of thinking that blends the previously independent disciplines of development and operations so that software products can both evolve functionally and be stable operationally. In practical terms, it means development teams are involved in operations and, in many instances, fully own the operation of their products.

The top companies are already there because they otherwise couldn’t deliver their massive and feature-rich products with 99.9 percent uptime. But there are still too many companies and developers who aren’t “doing it.” And it’s typically reflected in product quality.

It seems DevOps has been stigmatized as difficult, requiring expensive tools and generally classified as needed if you’re big. In this blog I’ll walk you through different aspects of DevOps and share the business value behind each, the common implementation techniques and the free (or inexpensive) tools that will let you jump-start your DevOps in a matter of weeks.

Product Health: The User Experience

Our goal as engineers is to build products that will be used by others. Typically we call them users and sometimes we have a strenuous relationship with them. But if users don’t like our products and aren’t using them, we have failed at the very core of our job. So what do users want-besides features, that is?

Users want products that are reasonably fast and work when they need them. Your site going down is an obvious example of why it’s important to be aware of your product’s health. But there are more nuanced examples. Sometimes a new release breaks an edge case and isn’t caught by regression testing. Sometimes a service becomes slow-it still works and nobody is complaining, but it’s dangerously close to being unusable. You need to be aware of your users’ experience so you can be proactive in fixing problems and delivering quality products.

Thus, the first set of metrics you need to measure are user experience metrics. These typically include success/failure and latency of all external endpoints: REST, web pages, etc. If the users are external, you want to measure the experience from their point of view, which includes the network latency and HTML/CSS/JS execution time for web pages. If this was a car, these metrics would be your low coolant red light. If the light is on, you must stop as soon as possible and add coolant before your engine block cracks from overheating.

To measure and monitor customer experience for web pages, Google Analytics is by far the most popular choice. It’s easy to set up and gives you exactly what you’re looking for: error stats per page, page load times, including individual page components and even browser parsing time for HTML. You can slice and dice the data by geographic region, browser type and so on to zero in on improvement or problem areas. Google Analytics is a great tool for tracking other product aspects as well, and you can kill many birds with one stone by adopting it.

However, Google Analytics will provide data only as long as there are users and your site works. But what if the site breaks, maybe partially or maybe only for a certain geographic region? And what do you do for public APIs that aren’t instrumentable with Google Analytics? You’ll need a synthetic monitoring service. These services emulate clients by calling your APIs or opening your web pages in an automated fashion. They are typically scriptable and run from multiple places around the world. They can tell you when your services are up or down and can measure (emulated) user latencies by geographic region. Pingdom is one of the newer generation of such services that delivers what you need for a modest price. It also integrates with a graphing and alarming tool called Datadog, which I’ll talk about later.

Product Health: Technical Innards

Now you have a solid understanding of your user experience and can detect user problems in real time. But how do you find the root causes? And can you do even better by detecting user problems before they happen? What you need to do is go a level deeper and monitor key technical aspects of your apps so you can correlate them with user problems. The technical metrics are important to watch because they can be early signs of trouble, which can be averted if you act quickly.

This can be a dangerous territory as the first desire is to monitor everything. Please keep your cool here, lest you drown yourself in hundreds of metrics that nobody understands and miss the important ones. In software, failure typically happens at integration points, so that’s what you should focus on: database and cache interactions, calls to other services, disk operations, queue sizes (for pub/sub architectures) and so on. Typically you don’t need to instrument business logic, as it either works or doesn’t and problems are caught in testing. This is where it is easy to create many metrics in a blink of an eye. A notable (and rare) exception is load-intensive parts of code that you want to monitor for performance reasons.

If you see latency in database calls creeping up, cache hit ration going down or thread pool or queue size growing, you can frequently identify and fix the problem before it affects the users. Going even deeper, you want to keep an eye on the virtual machine (VM) and server performance. If your garbage collection times are increasing, if your thread count grows or if your server is showing signs of strain, it’s time to pay extra attention. If this was a car, these would be your check engine yellow light. If the light is on, you need to check it out soon but can keep on driving for a little bit longer.

The most popular way to measure application metrics is with code instrumentation. Presently, Coda Hale (a.k.a. Dropwizard) Metrics framework and its ports to most popular languages are the way to go. Given the abundance of documentation and integration with all sorts of systems, this isn’t difficult. There are a plethora of coding patterns or external libraries you can use to avoid writing boilerplate code. Note that you want to include as much as possible in measuring requests (serializing, logging, etc.) so you need to put this logic as high up the filter chain or middleware as you can.

The last piece of the puzzle is VM and server monitoring. Traditionally, this has been the territory of specialized ops tools, but Datadog, a service that I will talk about in the next section, provides this out of the box. Its agents run on each machine or VM and collect server metrics, JVM/JMX data, IIS info (yep, it supports even Windows) and more.

Processing and Displaying the Data

Now your code is instrumented and generates all sorts of interesting data. What do you do with it? And where does the data go in the first place? All this data is useless if it just sits there. You want to be able to look at the trends, correlate them and get a basic set of statistical information such as averages, means, standard deviations and so on. You also want to see pretty graphs (who doesn’t?) and be able to create dashboards. This is the cool part, where all the previous work comes to fruition.

Datadog is a great way to start. It provides everything you need and more. All you have to do is install its agent on each box and configure the Metrics framework to send data to Datadog. Most Metrics ports already include Datadog adapters so there is no extra coding required.

Now you have everything and can slice and dice the data whichever way you want. You can export data for further analysis. You can check it periodically for problems. You can even display dashboards on big screens around the office (looks cool, eh?).

Putting it All Together

I have seen all of this put together in a couple of weeks. However, if this whole area is new to you it might take longer, as you’ll be learning the concepts along with the tools. Sometimes the deadlines are tight and even a few weeks is a luxury. However, launching a product without any insight into user experience or product health is not a good idea. If I had limited time and had to prioritize, I’d start with instrumenting user experience metrics and hooking it up to Datadog. Even for web pages you can start with only monitoring the REST services that power them, or the controller layer, if you’re on model-view-controller. Soon you will realize how nice it is to have hard metrics and will want to add Google Analytics and deeper code monitoring to your product.

All of this might seem like a lot, but you’re only dealing with four tools here: Pingdom, Google Analytics, Metrics framework, and Datadog. If your product is an API then you don’t need Google Analytics and if your product is internal, you don’t need Pingdom.

As I mentioned earlier, Metrics framework is ported to multiple languages already. But even if you have specific requirements that aren’t supported out of the box, it’s easy to extend it. For example, at Guaranteed Rate we wrote and open-sourced the Metrics/DataDog adapter for .NET and a library for code instrumentation and DataDog integration for Clojure.

.NET is particularly known to be behind on DevOps. Until recently one of the reasons was the lack of tools; however, that’s not the case anymore. All tools mentioned here, as well as Part 2 and 3, work in .NET and have adapters and ports where needed. With .NET entering the new open-source age, there are no more excuses to stay behind.

Give yourself two weeks, read the docs, go one step at a time, and you’ll be surprised how far along you can get!

What’s Next?

Part 2 will cover what you can do to maximize product uptime based on the data you now have. It also will discuss making sense of the data, automated alerting and troubleshooting when the inevitable production problems happen.