# MTHDS

> Official documentation for the MTHDS open standard: to give AI methods to AI agents.

# Home

## Overview

# MTHDS: the language of executable AI methods

**TLDR**

MTHDS is a declarative language for defining **AI methods**: discrete, reusable units of cognitive work like extraction, analysis, synthesis, generation. It is built on TOML and introduces two primitives: **concepts** (semantically typed data named after real domain things) and **pipes** (deterministic orchestration steps with explicit typed inputs and outputs). Pipes can invoke LLMs, VLMs, OCR, and image generation models, with built-in *structured generation*.

The typing is **conceptual**: `NonCompeteClause` is a refinement of `ContractClause`, meaning pipes compose wherever types are compatible, without manual glue code, because the types carry *business meaning* rather than just structure.

Methods are **executable and composable** like Unix tools: a method can be saved as a CLI command, combined with others using standard Unix pipes, or invoked directly by Claude Code. Methods can also be published as packages, used as templates to customize, or called as components from other methods. Teams can share some methods openly and keep their secret sauce private.

The standard ships with a **Claude Code plugin** that lets Claude write, modify, and compose methods on your behalf. This makes MTHDS **agent-first by design**: a domain expert who can describe what they need in plain language can have Claude author the method, which then runs consistently, is testable, and lives in version control.

Where agent skills handle open-ended tasks, methods handle the parts that benefit from being **explicit, versioned, and validated**. And unlike skills, methods are *executable outside the scope of an agent entirely*.

- Hub: [mthds.sh](https://mthds.sh)
- Spec: [github.com/mthds-ai/mthds](https://github.com/mthds-ai/mthds)
- Reference implementation: [github.com/Pipelex/pipelex](https://github.com/Pipelex/pipelex)
- Agent plugins: [github.com/mthds-ai/mthds-plugins](https://github.com/mthds-ai/mthds-plugins)
- VS Code extension: [go.pipelex.com/vscode](https://go.pipelex.com/vscode)

<div class="grid cards" markdown>

-   **Learn the Language**

    Concepts, pipes, domains — everything you need to write `.mthds` files.

    [:octicons-arrow-right-24: The Language](language/bundles.md)

-   **Read the Specification**

    The normative reference for file formats, validation rules, and resolution algorithms.

    [:octicons-arrow-right-24: Specification](spec/mthds-format.md)

-   **Get Started**

    Set up your editor and write your first method in a few steps.

    [:octicons-arrow-right-24: Write Your First Method](getting-started/first-method.md)

</div>


## What is MTHDS?

# What is MTHDS?

MTHDS (pronounced "methods") is an open standard for AI methods. It defines a typed language for describing what an AI should do — the data it works with, the transformations it performs, and how those transformations compose together — in plain text files that humans and machines can read.

An AI method in MTHDS is not code in the traditional sense. It is a declaration: "given this kind of input, produce that kind of output, using this approach." An agent or runner decides how to execute it. The method author decides what it means.

## Why a New Standard?

Current approaches to AI workflows each solve part of the problem but leave significant gaps:

**Code** (Python, frameworks) provides full control, typed outputs, and testability. But the business logic that matters — the actual extraction, analysis, or reasoning steps — is buried in infrastructure code. Domain experts cannot read or iterate on it.

**Natural-language instructions** (prompts, skills) are the opposite: easy to write and human-readable. But they carry no typed outputs, no validation, and no guaranteed output structure. The agent reinterprets them at every invocation.

**Automation platforms** (Zapier, Make, n8n) connect APIs through visual, GUI-based editors designed for deterministic workflows — not for AI. AI capabilities were bolted on after the fact. Their interfaces are built for human point-and-click editing, not for agents to read or compose programmatically. And when workflows require multi-step cognitive work — extraction, analysis, synthesis — these platforms lack the conceptual typing system to validate what flows between steps.

MTHDS provides the missing combination: a declarative language that is typed enough to validate, structured enough to compose, and readable enough for domain experts to understand. The method separates *what* a workflow does from *how* it is executed, analogous to how SQL separates data queries from storage engines.

## Agents as First-Class Participants

MTHDS gives agents the ability to discover, compose, and execute structured AI methods. But agents are not limited to execution. Because `.mthds` files are plain text with typed structure, agents can also *build* new methods, *modify* existing ones, and *extend* the ecosystem. An agent can search the Know-How Graph for methods by typed signature ("I have a `Document`, I need a `NonCompeteClause`"), compose them into multi-step workflows, create new methods that fill gaps in the graph, and execute them with validated data flow. Methods are artifacts agents can create, not just consume.

The language reads close to natural language and is designed to transcribe business logic. ***Concepts*** like `ContractClause`, `CandidateProfile`, or `Joke` carry business meaning directly — they are not programming abstractions but representations of real domain knowledge. A ***Pipe*** that declares `inputs = { doc = "ContractClause" }` and `output = "NonCompeteClause"` reads as a business statement, not as code. Domain experts can read and understand `.mthds` files without programming skills, making methods a shared artifact between technical and non-technical teams.

## The Two Pillars

MTHDS has two complementary halves, designed so you can start with one and add the other when you need it.

### Pillar 1 — The Language

The `.mthds` file format. Everything you need to define typed data and AI transformations in a single file.

A `.mthds` file is a valid [TOML](https://toml.io/) document with structure and meaning layered on top. If you know TOML, you already know the syntax. Inside a file, you define:

- **Concepts** — typed data declarations. A concept is a named type that describes a kind of data: a `ContractClause`, a `CandidateProfile`, a `Joke`. Concepts can have internal structure (fields with types like `text`, `integer`, `boolean`, `list`, `number`, `date`, `dict`, and `concept`) or they can be simple semantic labels. Concepts can refine other concepts — `NonCompeteClause` refines `ContractClause`, meaning it can be used anywhere a `ContractClause` is expected.

- **Pipes** — typed transformations. A pipe declares its inputs (concepts), its output (a concept), and its type — what kind of work it does. MTHDS defines **operators** (PipeLLM for language model generation, PipeStructure for turning text into a structured concept, PipeFunc for Python functions, PipeImgGen for image generation, PipeExtract for document extraction, PipeSearch for web search, PipeCompose for templating and assembly) and **controllers** (PipeSequence for sequential steps, PipeParallel for concurrent branches, PipeCondition for conditional routing, PipeBatch for mapping over lists).

- **Domains** — namespaces that organize concepts and pipes. A domain like `legal.contracts` tells you what a bundle is about and prevents naming collisions between unrelated definitions.

A single `.mthds` file — called a **bundle** — works on its own. No manifest, no package, no configuration. This is the starting point for learning and prototyping.

[:octicons-arrow-right-24: Learn the Language](../language/bundles.md)

### Pillar 2 — The Package System

The infrastructure for distributing and composing methods at scale.

When a standalone bundle is not enough — when you want to share methods, depend on other people's work, or control which methods are public — you add a `METHODS.toml` manifest. This turns a directory of bundles into a **package**: a distributable unit with a globally unique address, semantic versioning, declared dependencies, and explicit exports.

Packages are stored in Git repositories. The package address (e.g., `github.com/acme/legal-tools`) doubles as the fetch location — no upload step, no proprietary hosting. A lock file (`methods.lock`) pins exact versions with SHA-256 integrity hashes for reproducible builds.

Cross-package references use the `->` syntax: `scoring_lib->scoring.compute_weighted_score` reads as "from the `scoring_lib` dependency, get `compute_weighted_score` in the `scoring` domain." The separator was chosen for readability by non-technical audiences — arrows are intuitive, visually distinct from dots, and universally understood.

[:octicons-arrow-right-24: The Package System](../packages/structure.md)

## Core Concepts at a Glance

| Term | What it is | Analogy |
|------|-----------|---------|
| **Concept** | A typed data declaration — the kinds of data that flow through pipes. | A form with typed fields. |
| **Pipe** | A typed transformation — declares inputs, output, and what kind of work it does. | A processing step in a workflow. |
| **Domain** | A namespace that groups related concepts and pipes. | A folder that organizes related definitions. |
| **Bundle** | A single `.mthds` file. The authoring unit. | A source file. |
| **Package** | A directory with a `METHODS.toml` manifest and one or more bundles. The distribution unit. | A versioned library. |

## A Concrete Example

Here is a complete, working `.mthds` file:

```toml
domain      = "joke_generation"
description = "Generating one-liner jokes from topics"
main_pipe   = "generate_jokes_from_topics"

[concept.Topic]
description = "A subject or theme that can be used as the basis for a joke."
refines     = "Text"

[concept.Joke]
description = "A humorous one-liner intended to make people laugh."
refines     = "Text"

[pipe.generate_jokes_from_topics]
type        = "PipeSequence"
description = "Generate 3 joke topics and create a joke for each"
output      = "Joke[]"
steps = [
    { pipe = "generate_topics", result = "topics" },
    { pipe = "batch_generate_jokes", result = "jokes" },
]

[pipe.generate_topics]
type        = "PipeLLM"
description = "Generate 3 distinct topics suitable for jokes"
output      = "Topic[3]"
prompt      = "Generate 3 distinct and varied topics for crafting one-liner jokes."

[pipe.batch_generate_jokes]
type             = "PipeBatch"
description      = "Generate a joke for each topic"
inputs           = { topics = "Topic[]" }
output           = "Joke[]"
branch_pipe_code = "generate_joke"
input_list_name  = "topics"
input_item_name  = "topic"

[pipe.generate_joke]
type        = "PipeLLM"
description = "Write a clever one-liner joke about the given topic"
inputs      = { topic = "Topic" }
output      = "Joke"
prompt      = "Write a clever one-liner joke about $topic. Be concise and witty."
```

This file defines concepts (`Topic` and `Joke`, both refining the built-in `Text` type) and pipes: a sequence that generates topics and then batch-processes them into jokes. It works as a standalone file — save it, point a runner at it, and it runs. (A runner is any server implementing the [MTHDS HTTP Runner Protocol](../spec/protocol.md) — five routes; anyone can build one.)

## Progressive Enhancement

MTHDS is designed so you can start simple and add complexity only when you need it:

1. **Single file** — a `.mthds` bundle works on its own. No configuration, no manifest, no dependencies. Define concepts and pipes, and run them.

2. **Package** — add a `METHODS.toml` manifest to get a globally unique identity, version number, and visibility controls. Pipes become private by default; you choose what to export.

3. **Dependencies** — add a `[dependencies]` section to compose with other packages. Reference their concepts and pipes using the `->` syntax.

4. **Ecosystem** — publish packages to Git repositories. Registry indexes crawl and index them, enabling search by domain, by concept, or by typed pipe signature. The **Know-How Graph** — a typed network of AI methods — lets you ask "I have a `Document`, I need a `NonCompeteClause`" and find the pipes (or chains of pipes) that get you there.

Each layer builds on the previous one without breaking it. A standalone bundle that works today continues to work unchanged inside a package.

## What Makes MTHDS Different

MTHDS differs from other approaches to describing AI capabilities in three ways:

- **Typed signatures.** Every pipe declares the concepts it accepts and produces. This enables semantic discovery ("I have X, I need Y") and compile-time validation of data flow — something text-based descriptions cannot provide.

- **Composition built in.** Controllers (sequence, parallel, condition, batch) are part of the language, not an external orchestration layer. Multi-step methods are defined in the same file as the individual steps.

- **A real package system.** Versioned dependencies, lock files, visibility controls, cross-package references — the same infrastructure that makes code ecosystems work, applied to AI methods.

## Where to Go Next

- **Method authors**: Start with [The Language](../language/bundles.md) to learn bundles, concepts, pipes, and domains. Then move to [The Package System](../packages/structure.md) when you are ready to distribute.

- **Runner implementers**: Start with the [Specification](../spec/mthds-format.md) for the normative reference on file formats, validation rules, and resolution algorithms.

- **Everyone**: [Write Your First Method](../getting-started/first-method.md) walks you through creating a working `.mthds` file step by step.


## Design Philosophy

# Design Philosophy

MTHDS was designed with a specific set of principles that inform every decision in the standard. Understanding these principles helps explain why the standard works the way it does.

## Declarative by Design

MTHDS separates *what* a method does from *how* it is executed. The method author declares intent — "given this input concept, produce that output concept, using this approach" — and the runtime decides how to fulfill it. This is analogous to how SQL separates data queries from storage engines: the query author describes the result they want, not the access path to get there. A method may specify which model to use, but it does not specify how to manage context windows, how to retry on failure, or how to allocate compute resources. Those are runtime concerns.

## Filesystem as Interface

MTHDS packages are directories of text files. `.mthds` bundles are TOML. `METHODS.toml` is TOML. `methods.lock` is TOML. There are no binary formats, no databases, no proprietary encodings.

This means:

- **Version control works natively.** Every change to a method is a diff. Merge conflicts are resolvable by humans.
- **Agents can read and write methods.** AI agents that work with text files can create, modify, and validate MTHDS files without special tooling.
- **No vendor lock-in.** Any tool that reads TOML can read MTHDS files. The standard does not require any specific runtime, editor, or platform.

## Progressive Enhancement

MTHDS is designed so that each layer of functionality is opt-in:

1. **A single `.mthds` file works on its own.** No manifest, no package, no configuration. This is the entry point for learning and prototyping.
2. **Add a `METHODS.toml` to get packaging.** A globally unique address, version, and visibility controls. No behavior changes for the bundles themselves.
3. **Add `[dependencies]` to compose with others.** Cross-package references become available. Existing bundles continue to work unchanged.
4. **Publish to the ecosystem.** Registry indexes crawl your package. The Know-How Graph discovers your methods. No changes to your files are required.

Each layer builds on the previous one without breaking it. A standalone bundle that works today continues to work unchanged inside a package.

## Type-Driven Composability

Every pipe in MTHDS declares a typed signature: the concepts it accepts and the concept it produces. This is not just documentation — it is the foundation of the system.

Typed signatures enable:

- **Compile-time validation.** A runtime can verify that the output of one pipe is compatible with the input of the next before executing anything.
- **Semantic discovery.** The Know-How Graph answers "I have a `Document`, I need a `NonCompeteClause`" by traversing typed signatures and refinement hierarchies.
- **Auto-composition.** When no single pipe transforms X to Y, the graph can discover multi-step chains through intermediate concepts.

This contrasts with text-based approaches where capabilities are described in natural language. Text descriptions enable keyword search but not type-safe composition.

## One Artifact, Three Audiences

A `.mthds` file serves as both specification and executable artifact. A domain expert reads the business logic — concepts named after real-world entities, pipes that declare intent in plain language. An engineer reads something testable and deployable — typed signatures, validated data flow, version-controlled definitions. An agent reads something it can build, modify, and execute — structured text with machine-readable types and composable transformations. No separate documentation layer, no translation step between what the method describes and what the method does.

## Federated Distribution

MTHDS follows a federated model: decentralized storage with centralized discovery.

- **Storage is decentralized.** Packages live in Git repositories owned by their authors. There is no central package host. The package address (e.g., `github.com/acme/legal-tools`) IS the fetch location.
- **Discovery is centralized.** Registry indexes crawl and index packages without owning them. Multiple registries can coexist, each serving different communities.

This mirrors how the web works: content is hosted anywhere, search engines index it. No single entity controls the ecosystem.

## Packages Own Namespaces, Domains Carry Meaning

Domains are semantic labels that carry meaning about what a bundle is about — `legal.contracts`, `scoring`, `recruitment`. But domains do not merge across packages. Two packages declaring `domain = "recruitment"` have completely independent namespaces.

The package is the isolation boundary. Cross-package references are always explicit (`alias->domain.name`). There is no implicit coupling through shared domain names.

This is a deliberate design choice. Merging domains across packages would create fragile implicit coupling: any package declaring a domain could inject concepts into your namespace. Instead, cross-package composition is explicit — through dependencies and typed references.

The domain name remains valuable for discovery. Searching the Know-How Graph for "all packages in the recruitment domain" is meaningful. But discovery is not namespace merging.


## Comparison with Agent Skills

# Comparison with Agent Skills

Both MTHDS and [Agent Skills](https://agentskills.io/) address the problem of defining and discovering AI capabilities. They take fundamentally different approaches, reflecting different design goals.

## Scope Comparison

| Dimension | Agent Skills | MTHDS |
|-----------|-------------|-------|
| **Format** | JSON or YAML manifest describing a skill | TOML-based language with concepts, pipes, domains |
| **Type system** | Text descriptions for inputs/outputs | Typed signatures with concept refinement |
| **Composition** | No built-in composition model | Controllers (sequence, parallel, condition, batch) |
| **Package system** | No dependencies or versioning | Full package system with manifest, lock file, dependencies |
| **Discovery** | Text-based search (name, description, tags) | Typed search ("I have X, I need Y") + text search |
| **Distribution** | Hosted registry or skill files | Git-native, federated (decentralized storage, centralized discovery) |
| **CLI** | No CLI | Full `mthds` CLI with package management |

## What Agent Skills Does Well

Agent Skills is deliberately minimal. A skill is a manifest file that describes what an AI capability does in natural language. This makes it:

- **Simple to adopt.** Writing a skill manifest requires no new syntax — it is standard JSON/YAML.
- **Runtime-agnostic.** Any AI framework can consume a skill manifest.
- **Easy to discover.** Text descriptions are searchable by keywords, tags, and categories.

The simplicity is a feature. Agent Skills serves the use case of "tell me what capabilities exist" without prescribing how they are implemented or composed.

## What MTHDS Adds

MTHDS targets a different use case: defining, composing, and distributing AI methods with type safety.

- **Typed signatures** enable semantic discovery that text descriptions cannot support. "Find pipes that accept `Document` and produce `NonCompeteClause`" is a precise query with a precise answer.
- **Built-in composition** means multi-step methods are defined in the same file as the individual steps. A PipeSequence that extracts, analyzes, and summarizes is a single method, not an external orchestration.
- **A real package system** with versioned dependencies, lock files, and visibility controls makes methods reusable across teams and organizations.

## Design Parallels

Despite different approaches, the two standards share design principles:

- **Progressive disclosure.** Agent Skills' tiered skill hosting (built-in → user-created → community) parallels MTHDS's progressive enhancement (single file → package → ecosystem).
- **Skills as files.** Both standards treat capabilities as human-readable text files, not database entries or API registrations.
- **Federated distribution.** Both favor decentralized storage with centralized discovery.

## When to Use Which

- Use **Agent Skills** when you need a lightweight manifest that describes what an AI capability does, for use with frameworks that support the Agent Skills standard.
- Use **MTHDS** when you need typed composition, versioned dependencies, and type-safe discovery across packages.

The two standards are not mutually exclusive. A package's `main_pipe` could be exposed as an Agent Skill for frameworks that consume that format.


## The Know-How Graph

# The Know-How Graph

When packages export typed pipes and concepts into a shared ecosystem, something emerges: the **Know-How Graph** — a typed, searchable network of AI methods that spans packages. Instead of searching for methods by keyword or description, you can ask "I have a `ContractDocument`, I need a `NonCompeteClause`" and the graph finds the methods — or chains of methods — that get you there.

## Pipes as Typed Nodes

Every exported pipe has a typed signature — the concepts it accepts and the concept it produces:

```
extract_clause:          (ContractDocument) → NonCompeteClause
classify_document:       (Document)         → ClassifiedDocument
summarize_findings:      (Text)             → ExecutiveSummary
```

These signatures, combined with the concept refinement hierarchy, form a directed graph:

- **Nodes** are pipe signatures (typed transformations).
- **Edges** are data flow connections — the output concept of one pipe type-matches the input concept of another.
- **Refinement edges** connect concept hierarchies (e.g., `NonCompeteClause` refines `ContractClause` refines `Text`).

## Type-Compatible Discovery

The type system enables queries that text-based discovery cannot support:

| Query | Example |
|-------|---------|
| "I have X, I need Y" | "I have a `Document`, I need a `NonCompeteClause`" — finds all pipes or chains that produce it. |
| "What can I do with X?" | "What pipes accept `ContractDocument` as input?" — shows downstream possibilities. |
| Compatibility check | Before installing a package, verify its pipes are type-compatible with yours. |

Because MTHDS concepts have a refinement hierarchy, type-compatible search understands that a pipe accepting `Text` also accepts `NonCompeteClause` (since `NonCompeteClause` refines `Text` through the refinement chain).

## Auto-Composition

When no single pipe transforms X into Y, the Know-How Graph can find a **chain** through intermediate concepts:

```
Document → [extract_pages] → Page[] → [analyze_content] → AnalysisResult
```

This is auto-composition — discovering multi-step pipelines by traversing the graph. The `mthds pkg graph` command supports this with the `--from` and `--to` options.

## Cross-Package Concept Refinement

Packages can extend another package's vocabulary through concept refinement:

```toml
# In your package, depending on acme_legal
[concept.EmploymentNDA]
description = "A non-disclosure agreement specific to employment contexts"
refines     = "acme_legal->legal.contracts.NonDisclosureAgreement"
```

This builds on `NonDisclosureAgreement` from the `acme_legal` dependency without merging namespaces. The refinement relationship enriches the Know-How Graph: any pipe that accepts `NonDisclosureAgreement` now also accepts `EmploymentNDA`.

## From Packages to Knowledge

The Know-How Graph emerges naturally from the package system:

1. Each package exports pipes with typed signatures.
2. Concepts define a shared vocabulary with refinement hierarchies.
3. Dependencies connect packages, enabling cross-package references.
4. Registry indexes crawl this information and make it searchable.

The result is a federated network of composable, discoverable, type-safe AI methods — where finding the right method is as precise as asking "I have X, I need Y."

The Know-How Graph is infrastructure that agents can navigate. An agent can discover methods by typed signature, compose multi-step chains through intermediate concepts, and build new methods that extend the graph. Each method an agent creates or refines becomes a node that other agents — or humans — can discover and reuse. The graph grows as the ecosystem grows.

The ecosystem follows an open-commons model. Common tasks — contract extraction, document classification, expense processing — get solved once and shared as public packages. Organization-specific workflows stay private simply by not being published. When a package is published, the `exports` field in `METHODS.toml` controls which pipes and concepts are part of the public API, but the fundamental privacy boundary is publication itself.

## See Also

- [The Know-How Graph Viewpoint](https://knowhowgraph.com/) — the extended essay on the Know-How Graph vision and why AI agents need typed methods.
- [Concepts](../language/concepts.md) — how concepts define typed data and refinement.
- [Exports & Visibility](../packages/exports-visibility.md) — which pipes are visible in the graph.
- [Distribution](../packages/distribution.md) — how registries index packages.
- [The Registry](../packages/registry.md) — the HTTP service that exposes the Know-How Graph for remote queries.
- [Registry Search](../packages/registry-search.md) — type-aware search semantics and graph query rules.


# The Language

## Bundles

# Bundles

A **bundle** is a single `.mthds` file. It is the authoring unit of MTHDS — the place where you define typed data and typed transformations.

## A First Look

```toml
domain      = "legal.contracts"
description = "Contract analysis methods for legal documents"
main_pipe   = "extract_clause"

[concept]
ContractClause = "A clause extracted from a legal contract"

[pipe.extract_clause]
type        = "PipeLLM"
description = "Extract the key clause from a contract"
inputs      = { contract_text = "Text" }
output      = "ContractClause"
prompt      = "Extract the key clause from the following contract: @contract_text"
```

This is a complete, valid `.mthds` file. It defines one concept, one pipe, and works on its own — no manifest, no package, no dependencies needed.

## What This Does

The file declares a **domain** (`legal.contracts`), a **concept** (`ContractClause`), and a **pipe** (`extract_clause`) that uses an LLM to transform `Text` into a `ContractClause`. The `main_pipe` header marks `extract_clause` as the bundle's primary entry point.

## File Format

A `.mthds` file is a valid [TOML](https://toml.io/) document encoded in UTF-8. The `.mthds` extension is required. If you know TOML, you already know the syntax — MTHDS adds structure and meaning on top of it.

## Bundle Structure

Every bundle has up to three sections:

1. **Header fields** — top-level key-value pairs that identify the bundle.
2. **Concept definitions** — typed data declarations in `[concept]` tables.
3. **Pipe definitions** — typed transformations in `[pipe.<pipe_code>]` tables.

All three are optional in the TOML sense, but a useful bundle will contain at least one concept or one pipe.

## Header Fields

Header fields appear at the top of the file, before any `[concept]` or `[pipe]` tables.

| Field | Required | Description |
|-------|----------|-------------|
| `domain` | Yes | The domain this bundle belongs to. Determines the namespace for all concepts and pipes defined in this file. |
| `description` | No | A human-readable description of what this bundle provides. |
| `system_prompt` | No | A default system prompt applied to all `PipeLLM` pipes in this bundle that do not define their own. When a PipeLLM pipe omits its own `system_prompt`, it inherits the bundle-level value. A pipe that defines its own `system_prompt` overrides the bundle default. |
| `main_pipe` | No | The pipe code of the bundle's primary entry point. Auto-exported when the bundle is part of a package. |

The `domain` field is the only required header. It assigns a namespace to everything in the file — more on this in [Domains](domains.md).

The `main_pipe` field, if present, must be a valid `snake_case` pipe code and must reference a pipe defined in the same bundle.

### Bundle-Level System Prompt

The `system_prompt` header sets a default system prompt for all PipeLLM pipes in the bundle. Individual pipes can override it by defining their own `system_prompt`.

```toml
domain        = "medical.records"
description   = "Medical record analysis methods"
system_prompt = """
You are a medical records analyst. Follow HIPAA guidelines strictly.
Never include patient identifiers in your output.
"""

[concept]
Diagnosis = "A primary diagnosis extracted from a medical record"

[pipe.extract_diagnosis]
type        = "PipeLLM"
description = "Extract diagnosis from medical records"
inputs      = { record = "Text" }
output      = "Diagnosis"
prompt      = "Extract the primary diagnosis from: @record"
# Inherits the bundle-level system_prompt

[pipe.summarize_for_patient]
type          = "PipeLLM"
description   = "Summarize records in patient-friendly language"
inputs        = { record = "Text" }
output        = "Text"
system_prompt = "You are a helpful medical assistant explaining records to patients in simple terms."
prompt        = "Summarize the following in plain language: @record"
# Overrides the bundle-level system_prompt
```

## Standalone Bundles

A `.mthds` file works on its own, without a package manifest. When used standalone:

- All pipes are treated as public (no visibility restrictions).
- No dependencies are available beyond native concepts.
- The bundle is not distributable (no package address).

This makes `.mthds` files ideal for learning, prototyping, and simple projects. When you need distribution, add a `METHODS.toml` manifest — see [The Package System](../packages/structure.md).


## Concepts

# Concepts

Concepts are typed data declarations. They define the vocabulary of a domain — the kinds of data that pipes accept as input and produce as output.

## Simple Concepts

The simplest form of concept declaration uses a flat `[concept]` table. Each key is a concept code, and the value is a description string:

```toml
[concept]
ContractClause = "A clause extracted from a legal contract"
UserProfile    = "A user's profile information"
```

These concepts exist as named types. They have no internal structure — they are semantic labels that give meaning to data flowing through pipes.

**Naming rule:** Concept codes must be `PascalCase`, matching the pattern `[A-Z][a-zA-Z0-9]*`. Examples: `ContractClause`, `UserProfile`, `CVAnalysis`.

### Naming Guidelines

Beyond the PascalCase rule, follow these principles for clear, reusable concept names:

**1. Define what it is, not what it's for.**

```toml
[concept]
# Avoid — includes usage context
TextToSummarize = "Text that needs to be summarized"

# Prefer — defines the essence
Article = "A written composition on a specific topic"
```

A concept should describe the nature of the data, not the role it plays in a particular pipe.

**2. Use singular forms.**

```toml
[concept]
# Avoid — plural form
Invoices = "Commercial documents from sellers"

# Prefer — singular form
Invoice = "A commercial document issued by a seller to a buyer"
```

Concepts are always singular. Use [multiplicity](multiplicity.md) to express quantity.

**3. Avoid unnecessary adjectives.**

```toml
[concept]
# Avoid — includes subjective qualifier
LongArticle = "A lengthy written composition"

# Prefer — neutral description
Article = "A written composition on a specific topic"
```

Keep concept names factual and neutral. Qualitative distinctions belong in the pipe logic, not in the concept name.

## Structured Concepts

When a concept needs internal structure — specific fields with types — use a `[concept.<ConceptCode>]` sub-table:

```toml
[concept.LineItem]
description = "A single line item in an invoice"

[concept.LineItem.structure]
product_name = { type = "text", description = "Name of the product", required = true }
quantity     = { type = "integer", description = "Quantity ordered", required = true }
unit_price   = { type = "number", description = "Price per unit", required = true }
```

The `structure` table defines the fields of the concept. Each field has a type and a description.

Both simple and structured forms can coexist in the same bundle:

```toml
[concept]
ContractClause = "A clause extracted from a legal contract"

[concept.LineItem]
description = "A single line item in an invoice"

[concept.LineItem.structure]
product_name = { type = "text", description = "Name of the product", required = true }
quantity     = { type = "integer", description = "Quantity ordered", required = true }
unit_price   = { type = "number", description = "Price per unit", required = true }
```

## Concept Blueprint Fields

When using the structured form `[concept.<ConceptCode>]`:

| Field | Required | Description |
|-------|----------|-------------|
| `description` | Yes | Human-readable description of the concept. |
| `structure` | No | Field definitions. If a string, it is a shorthand description (equivalent to a simple declaration). If a table, each key is a field name mapped to a field blueprint. |
| `refines` | No | A concept reference indicating specialization of another concept. |

`refines` and `structure` cannot both be present on the same concept. A concept either refines another concept or defines its own structure, not both.

## Field Types

Each field in a concept's `structure` is defined by a field blueprint. The `type` field determines the kind of data:

| Type | Description | Example `default_value` |
|------|-------------|------------------------|
| `text` | A string value. | `"hello"` |
| `integer` | A whole number. | `42` |
| `number` | A numeric value (integer or floating-point). | `3.14` |
| `boolean` | A true/false value. | `true` |
| `date` | A date value. | *(datetime)* |
| `list` | An ordered collection. Use `item_type` to specify element type. | `["a", "b"]` |
| `dict` | A key-value mapping. Requires `key_type` and `value_type`. | *(table)* |
| `concept` | A reference to another concept. Requires `concept_ref`. Cannot have a `default_value`. | *(not allowed)* |

When `type` is omitted and `choices` is provided, the field becomes an enumeration — its value must be one of the listed strings.

## Field Blueprint Reference

The complete set of attributes available on each field in a concept's `structure`:

| Attribute | Required | Description |
|-----------|----------|-------------|
| `description` | Yes | Human-readable description. |
| `type` | Conditional | The field type (see table above). Required unless `choices` is provided. |
| `required` | No | Whether the field is required. Default: `false`. |
| `default_value` | No | Default value, must match the declared type. |
| `choices` | No | Fixed set of allowed string values. When set, `type` must be omitted. |
| `key_type` | Conditional | Key type for `dict` fields. Required when `type = "dict"`. |
| `value_type` | Conditional | Value type for `dict` fields. Required when `type = "dict"`. |
| `item_type` | No | Item type for `list` fields. When `"concept"`, requires `item_concept_ref`. |
| `concept_ref` | Conditional | Concept reference for `concept`-typed fields. Required when `type = "concept"`. |
| `item_concept_ref` | Conditional | Concept reference for list items when `item_type = "concept"`. |

## A Complete Example

This concept demonstrates every field type:

```toml
[concept.CandidateProfile]
description = "A candidate's profile for job matching"

[concept.CandidateProfile.structure]
full_name        = { type = "text", description = "Full name", required = true }
years_experience = { type = "integer", description = "Years of professional experience" }
gpa              = { type = "number", description = "Grade point average" }
is_active        = { type = "boolean", description = "Whether actively looking", default_value = true }
graduation_date  = { type = "date", description = "Date of graduation" }
skills           = { type = "list", item_type = "text", description = "List of skills" }
metadata         = { type = "dict", key_type = "text", value_type = "text", description = "Additional metadata" }
seniority_level  = { description = "Seniority level", choices = ["junior", "mid", "senior", "lead"] }
address          = { type = "concept", concept_ref = "Address", description = "Home address" }
references       = { type = "list", item_type = "concept", item_concept_ref = "ContactInfo", description = "Professional references" }
```

## Concept Refinement

Refinement establishes a specialization relationship between concepts. A refined concept inherits the semantic meaning of its parent and can be used anywhere the parent is expected.

```toml
[concept.NonCompeteClause]
description = "A non-compete clause in an employment contract"
refines     = "ContractClause"
```

`NonCompeteClause` is a specialization of `ContractClause`. Any pipe that accepts `ContractClause` also accepts `NonCompeteClause`.

!!! note "Type Compatibility — Substitutability"
    Refined concepts are **substitutable** for their parent. If a pipe declares `inputs = { clause = "ContractClause" }`, you can pass a `NonCompeteClause` (or any other concept that refines `ContractClause`) as the `clause` input. This follows the standard subtyping rule: a specialized type is always valid where the general type is expected.

The `refines` field accepts three forms of concept reference:

- **Bare code:** `"ContractClause"` — resolved within the current bundle's domain.
- **Domain-qualified:** `"legal.ContractClause"` — resolved within the current package.
- **Cross-package:** `"acme_legal->legal.contracts.NonDisclosureAgreement"` — resolved from a dependency.

Cross-package refinement is how you build on another package's vocabulary without merging namespaces. See [Namespace Resolution](namespace-resolution.md) for the full resolution rules.

## When to Refine vs. When to Create a New Concept

Choosing between refinement and a new concept depends on the semantic relationship and whether you need custom structure.

**Refine** when the new concept is genuinely a specialized version of an existing one and the parent's structure is sufficient:

```toml
[concept.Invoice]
description = "A commercial document issued by a seller to a buyer"
refines = "Document"

[concept.VIPCustomer]
description = "A high-value customer with special privileges"
refines = "Customer"
```

Refinement is the right choice when you want substitutability — an `Invoice` can be used wherever a `Document` is expected.

**Create a new concept** when the data needs its own structure, or when it does not naturally fit as a subtype of any existing concept:

```toml
[concept.InvoiceData]
description = "Extracted data from an invoice"

[concept.InvoiceData.structure]
invoice_number = { type = "text", description = "Invoice identifier", required = true }
total_amount   = { type = "number", description = "Total amount due" }
line_items     = { type = "list", item_type = "concept", item_concept_ref = "LineItem", description = "Invoice line items" }
```

!!! tip "Don't over-refine"
    Avoid creating refinements for distinctions that belong in processing logic rather than in the type system. For example, `SmallInvoice` and `LargeInvoice` are better handled as a single `Invoice` concept with the pipe deciding how to process different amounts.

## Native Concepts

MTHDS provides a set of built-in concepts that are always available in every bundle without declaration. They belong to the reserved `native` domain.

| Code | Description |
|------|-------------|
| `Dynamic` | A dynamically-typed value. |
| `Text` | A text string. |
| `Image` | An image (binary). |
| `Document` | A document (e.g., PDF, web page). |
| `Html` | HTML content. |
| `TextAndImages` | Combined text and image content. |
| `Number` | A numeric value. |
| `Page` | A single page extracted from a document. |
| `JSON` | A JSON value. |
| `SearchResult` | A web search result with answer and sources. |
| `Anything` | Accepts any type. |

Native concepts can be referenced by bare code (`Text`, `Image`) or by qualified reference (`native.Text`, `native.Image`). Bare native codes always take priority during name resolution.

A bundle cannot declare a concept with the same code as a native concept. For example, defining `[concept] Text = "My custom text"` is an error.

### Native Concept Fields

The most commonly used native concepts have the following fields. These are the fields you can reference in prompts via dot notation (e.g., `$page_content.page_view`).

**Text** — a single `text` field containing the string value.

**Image** — `url` (location of the image), `source_prompt` (the prompt used to generate it, if applicable), `caption` (descriptive text), `base_64` (base64-encoded image data, alternative to URL).

**Document** — `url` (location of the document file or web page), `mime_type` (e.g., `"application/pdf"`), `title` (optional display name), `snippet` (optional text excerpt).

**Number** — a single `number` field (integer or floating-point).

**TextAndImages** — `text` (the text content), `images` (a list of images associated with the text).

**Page** — `text_and_images` (the extracted text and embedded images from the page), `page_view` (a screenshot or rendering of the entire page as an image).

**SearchResult** — `answer` (the synthesized answer text), `sources` (a list of source citations, each a Document with `title`, `url`, and `snippet`).

**JSON** — a single `json_obj` field containing the JSON object.

## See Also

- [Specification: Concept Definitions](../spec/mthds-format.md#concept-definitions) — normative reference for all concept fields and validation rules.
- [Pipes — Operators](pipes-operators.md) — how concepts are used as pipe inputs and outputs.
- [Native Concepts table](../spec/mthds-format.md#native-concepts) — full list with qualified references.


## Pipes — Operators

# Pipes — Operators

Pipes are typed transformations — the actions in MTHDS. Each pipe has a typed signature: it declares what concepts it accepts as input and what concept it produces as output.

MTHDS defines two categories of pipes:

- **Operators** — pipes that perform a single transformation (this page).
- **Controllers** — pipes that orchestrate other pipes (next page).

## Common Fields

All pipe types share these base fields:

| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | The pipe type (e.g., `"PipeLLM"`, `"PipeSequence"`). |
| `description` | Yes | Human-readable description of what this pipe does. |
| `inputs` | No | Input declarations. Keys are input names (`snake_case`), values are concept references. |
| `output` | Yes | The output concept reference. |

**Pipe codes** are the keys in `[pipe.<pipe_code>]` tables. They must be `snake_case`, matching `[a-z][a-z0-9_]*`.

**Concept references in inputs and output** support an optional multiplicity suffix:

| Syntax | Meaning |
|--------|---------|
| `ConceptName` | A single instance. |
| `ConceptName[]` | A variable-length list. |
| `ConceptName[N]` | A fixed-length list of exactly N items (N ≥ 1). |

See [Multiplicity](multiplicity.md) for a detailed guide on when and how to use each form.

## PipeLLM

Generates output by invoking a large language model with a prompt.

```toml
[pipe.analyze_cv]
type = "PipeLLM"
description = "Analyze a CV to extract key professional information"
output = "CVAnalysis"
model = "$writing-factual"
system_prompt = """
You are an expert HR analyst specializing in CV evaluation.
"""
prompt = """
Analyze the following CV and extract the candidate's key professional information.

@cv_pages
"""

[pipe.analyze_cv.inputs]
cv_pages = "Page"
```

**What this does:** Takes a `Page` input, sends it to an LLM with the given prompt and system prompt, and produces a `CVAnalysis` output.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `prompt` | No | The LLM prompt template. Supports Jinja2 syntax and `@variable` / `$variable` shorthand. |
| `system_prompt` | No | System prompt for the LLM. Falls back to the bundle-level `system_prompt` if omitted. |
| `model` | No | Model identifier, model reference (see [Model References](model-references.md)), or an inline settings table (see [Inline Settings](model-references.md#inline-settings)). |
| `model_to_structure` | No | Model used for structuring the LLM output into the declared concept. Accepts the same forms as `model`. |
| `structuring_method` | No | How the output is structured: `"direct"` or `"preliminary_text"`. |

**Prompt template syntax:**

All three syntaxes below compile to the same Jinja2 template under the hood. The shorthands exist to improve readability:

- `{{ variable_name }}` — standard Jinja2 variable substitution.
- `@variable_name` — shorthand designed for **block-level insertion** of an input's full content. Use `@` when the variable stands on its own line or represents a large block of text.
- `$variable_name` — shorthand designed for **inline substitution** within a sentence. Use `$` when the variable is embedded in surrounding text.
- `@?variable_name` — shorthand for **conditional insertion**. Renders the variable only if it is truthy (non-empty, non-null). Use `@?` for optional inputs that may or may not be provided.

For example:

```toml
prompt = """
Summarize the following article about $topic:

@article_text

@?additional_context

Keep the summary under 3 sentences.
"""
```

Here, `$topic` is inline (part of the sentence), `@article_text` is block-level (inserted as a standalone block), and `@?additional_context` is conditionally inserted — it only appears if the variable has a value. All three conventions aid readability — they are not enforced by the runtime.

Dotted paths are supported: `{{ doc_request.document_type }}`, `@doc_request.priority`.

For the full reference on shorthand syntax, template categories, and available filters, see [PipeCompose — Template Mode](#template-mode) below.

Every variable referenced in the prompt must correspond to a declared input, and every declared input must be referenced in the prompt or system prompt. Unused inputs are rejected.

### Image Inputs

PipeLLM supports vision language models that process both text and images. Declare image inputs in the `inputs` field — they are passed to the model alongside the text prompt.

```toml
[pipe.describe_image]
type        = "PipeLLM"
description = "Describe an image"
inputs      = { image = "Image" }
output      = "VisualDescription"
prompt      = "Describe the provided image in great detail: $image"
```

Image variables must be tagged with `@` or `$` in the prompt, just like text variables.

**Sub-attribute access with dot notation:** When an input is a structured concept that contains an image field, use dotted paths to reach the image:

```toml
[pipe.analyze_page_view]
type        = "PipeLLM"
description = "Analyze the visual layout of a page"
inputs      = { "page_content.page_view" = "Image" }
output      = "LayoutAnalysis"
prompt      = """
Analyze the visual layout and design elements of this page: $page_content.page_view
Focus on typography, spacing, and overall composition.
"""
```

**Multiple images:** List each image as a separate input:

```toml
[pipe.compare_images]
type        = "PipeLLM"
description = "Compare two images"
inputs      = { first_image = "Image", second_image = "Image" }
output      = "ImageComparison"
prompt      = "Compare these two images and describe their similarities and differences: $first_image and $second_image"
```

### Document Inputs

PipeLLM supports documents (PDFs, etc.) as inputs. Documents are passed to the model alongside the text prompt.

```toml
[pipe.summarize_document]
type        = "PipeLLM"
description = "Summarize a document"
inputs      = { document = "Document" }
output      = "DocumentSummary"
prompt      = "Summarize the key points from this document: @document"
```

Document variables must be tagged with `@` or `$`, just like text and image variables.

**Multiple documents:**

```toml
[pipe.compare_documents]
type        = "PipeLLM"
description = "Compare two documents"
inputs      = { first_doc = "Document", second_doc = "Document" }
output      = "DocumentComparison"
prompt      = "Compare these two documents and describe their similarities and differences: $first_doc and $second_doc"
```

Text, image, and document inputs can be freely combined in the same pipe.

### Structuring Method

The `structuring_method` field is a directive that controls how PipeLLM produces structured output (when the output concept has a `structure` table):

- `"direct"` — the model generates JSON conforming to the output schema in a single call. This is the fastest option and works well when the output structure is straightforward (few fields, simple types) and the model reliably produces well-formed JSON.
- `"preliminary_text"` — the runtime first produces free-form text from the prompt, then structures that text into the target concept as a second step. Use this mode when the output structure is complex (many fields, nested concepts, or nuanced extraction) or when the model struggles to produce correct JSON in a single pass.

When `structuring_method` is omitted, the runtime chooses a default. In general, start with the default and switch to `"preliminary_text"` if you observe structuring errors or degraded output quality on complex schemas.

The standard does not prescribe how a runtime achieves `"preliminary_text"`. The reference runtime expands the pipe at load time into a [`PipeSequence`](pipes-controllers.md#pipesequence) of `PipeLLM` (producing `Text`) followed by [`PipeStructure`](#pipestructure) (producing the declared output). Authors who want explicit control over each step — for example, to pick a different model for the structuring call or to reuse the structuring step across several upstream text sources — can author the two pipes by hand instead of using the `structuring_method` shorthand.

**More PipeLLM examples:**

Image input with structured output:

```toml
[concept.TableRow]
description = "A single row of data from a table"

[concept.TableRow.structure]
cells = { type = "list", item_type = "text", description = "Cell values in order" }

[concept.TableData]
description = "Structured data extracted from a table image"

[concept.TableData.structure]
headers = { type = "list", item_type = "text", description = "Column headers" }
rows    = { type = "list", item_type = "concept", item_concept_ref = "TableRow", description = "Table rows" }

[pipe.extract_table_from_image]
type        = "PipeLLM"
description = "Extract table data from an image"
inputs      = { image = "Image" }
output      = "TableData"
prompt      = "Extract the table data from this image and return the headers and rows: $image"
```

Combining text and document inputs:

```toml
[pipe.analyze_with_context]
type        = "PipeLLM"
description = "Analyze a document with additional context"
inputs      = { context = "Text", reference_doc = "Document" }
output      = "ContextualAnalysis"
prompt      = """
Given this context: $context

Analyze the document and explain how it relates to the context: $reference_doc
"""
```

## PipeStructure

Turns text into a structured concept — typically by invoking an LLM that fills the declared output schema, though the standard does not prescribe a specific mechanism.

```toml
[concept.RestaurantReview]
description = "A structured restaurant review extracted from prose"

[concept.RestaurantReview.structure]
restaurant_name = { type = "text", description = "Name of the restaurant" }
overall_rating  = { type = "integer", description = "Overall rating from 1 to 5" }
highlights      = { type = "list", item_type = "text", description = "Standout positives" }
complaints      = { type = "list", item_type = "text", description = "Issues mentioned" }

[pipe.structure_review]
type        = "PipeStructure"
description = "Turn a free-form review into a RestaurantReview"
inputs      = { review_text = "Text" }
output      = "RestaurantReview"
```

**What this does:** Takes a single `Text` input and produces a typed object (or list of objects) matching the output concept. The reference runtime implements this with an LLM call that fills the schema.

Reach for `PipeStructure` whenever the text comes from somewhere other than a fresh `PipeLLM` call — text that came from a `PipeExtract` over a PDF, from a `PipeSearch` result, from a user message, or from an upstream `PipeLLM` step that intentionally produced free-form prose.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `inputs` | Yes | Exactly one input. Its concept must be `Text` or a concept that refines `Text`. |
| `output` | Yes | The target structured concept, with optional multiplicity (`Foo`, `Foo[]`, `Foo[N]`). Cannot be `Text` or a concept that refines `Text`. |
| `model` | No | Model identifier, model reference (see [Model References](model-references.md)), or an inline LLM settings table (see [Inline LLM Settings](../spec/mthds-format.md#inline-llm-settings)). |

`PipeStructure` does not accept a user-controlled prompt template — the runtime owns whatever prompting (or other mechanism) it uses to produce the structured output. Use a `PipeLLM` instead when prose generation needs prompt control.

!!! note "No images, no documents"
    `PipeStructure` is intentionally narrow: it takes one `Text` input. To structure an image or a PDF page, run an upstream extraction step (e.g. `PipeExtract` or a vision `PipeLLM`) and feed its text output into `PipeStructure`.

### Output Multiplicity

Use bracket notation in `output` to control how many items the runtime produces:

- `output = "Review"` — exactly one item.
- `output = "Review[]"` — variable-length list (the model decides).
- `output = "Review[3]"` — exactly three items.

See [Multiplicity](multiplicity.md) for the full picture.

### Examples

**Structure a list of objects from a transcript:**

```toml
[pipe.structure_review_batch]
type        = "PipeStructure"
description = "Extract one or more reviews from a transcript"
inputs      = { transcript = "Text" }
output      = "RestaurantReview[]"
```

**After a document extraction step:**

```toml
[pipe.invoice_to_record]
type        = "PipeSequence"
description = "Read an invoice PDF and turn it into an InvoiceRecord"
inputs      = { invoice_pdf = "Document" }
output      = "InvoiceRecord"
steps = [
  { pipe = "extract_invoice_text", result = "invoice_text" },
  { pipe = "structure_invoice",    result = "invoice_record" },
]

[pipe.extract_invoice_text]
type        = "PipeLLM"
description = "Read the invoice PDF and produce a faithful textual transcript"
inputs      = { invoice_pdf = "Document" }
output      = "Text"
prompt      = "Read this invoice and produce a faithful textual transcript of every line item, total, and metadata: @invoice_pdf"

[pipe.structure_invoice]
type        = "PipeStructure"
description = "Turn the invoice transcript into an InvoiceRecord"
inputs      = { invoice_text = "Text" }
output      = "InvoiceRecord"
```

**Pick a different model for the structuring step:**

```toml
[pipe.structure_review_premium]
type        = "PipeStructure"
description = "Use a stronger model for tricky structurings"
inputs      = { review_text = "Text" }
output      = "RestaurantReview"
model       = "@default-premium"
```

## PipeFunc

Calls a registered Python function.

```toml
[pipe.capitalize_text]
type          = "PipeFunc"
description   = "Capitalize the input text"
inputs        = { text = "Text" }
output        = "Text"
function_name = "my_package.text_utils.capitalize"
```

**What this does:** Passes the `Text` input to the Python function `my_package.text_utils.capitalize` and returns the result as `Text`.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `function_name` | Yes | The fully-qualified name of the Python function to call. |

PipeFunc bridges MTHDS with custom code. The function must be registered in the runtime.

## PipeImgGen

Generates images using an image generation model.

```toml
[pipe.generate_portrait]
type        = "PipeImgGen"
description = "Generate a portrait image from a description"
inputs      = { description = "Text" }
output      = "Image"
prompt      = "A professional portrait: $description"
model       = "$gen-image-testing"
```

**What this does:** Renders the `prompt` template — interpolating the `Text` input `description` — sends the result to an image generation model, and produces an `Image` output.

PipeImgGen does not consume a dedicated "prompt" concept. The prompt is a string template declared directly on the pipe, and the pipe's declared `inputs` are injected into that template at runtime:

- **`Text` inputs** are interpolated into the prompt text via `$variable` shorthand or Jinja2.
- **`Image` inputs** (a single image or a list) are referenced in the prompt and injected as **reference images**: each referenced image is replaced by an `[Image N]` token in the rendered text and passed to the generator alongside it. This is the same vision pattern used for image inputs to `PipeLLM`, and it enables image-to-image, reference-image, and image-editing generation, bounded by the model's image limit.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `prompt` | Yes | The image generation prompt template. Supports Jinja2 and `$variable` shorthand; declared inputs are injected into it. |
| `negative_prompt` | No | An optional prompt template describing what to avoid in the generated image. |
| `model` | No | Model identifier, model reference (see [Model References](model-references.md)), or an inline settings table (see [Inline Settings](model-references.md#inline-settings)). |
| `aspect_ratio` | No | Desired aspect ratio. Values: `square`, `landscape_4_3`, `landscape_3_2`, `landscape_16_9`, `landscape_21_9`, `portrait_3_4`, `portrait_2_3`, `portrait_9_16`, `portrait_9_21`. |
| `is_raw` | No | Whether to use raw mode (less post-processing). |
| `seed` | No | Random seed for reproducibility. Integer value, or `"auto"` to let the model randomize it. |
| `background` | No | Background setting. Values: `transparent`, `opaque`, `auto`. |
| `output_format` | No | Image output format. Values: `png`, `jpeg`, `webp`. |

**Reference images (image-to-image):** declare `Image` inputs and reference them in the prompt to condition generation on existing images. A single reference image:

```toml
[pipe.restyle_photo]
type        = "PipeImgGen"
description = "Restyle a photo following a textual instruction"
inputs      = { source = "Image", instruction = "Text" }
output      = "Image"
prompt      = "Restyle this image: $source. $instruction"
```

A list of reference images, referenced as a group:

```toml
[pipe.blend_references]
type        = "PipeImgGen"
description = "Blend multiple reference images into one composition"
inputs      = { refs = "Image[]" }
output      = "Image"
prompt      = "Combine these references into a single coherent scene: $refs"
```

Each referenced image becomes an `[Image N]` token in the rendered prompt and is passed to the generator as a reference image, up to the model's image limit.

## PipeExtract

Extracts structured content from documents (e.g., PDF, web pages).

```toml
[pipe.extract_cv]
type        = "PipeExtract"
description = "Extract text content from a CV PDF document"
inputs      = { cv_pdf = "Document" }
output      = "Page[]"
model       = "@default-text-from-pdf"
```

**Web page extraction:**

```toml
[pipe.extract_web_article]
type        = "PipeExtract"
description = "Extract content from a web page"
inputs      = { article_url = "Document" }
output      = "Page[]"
model       = "@default-extract-web-page"
```

**What this does:** Takes a `Document` input (a file path, storage URL, or web page URL) and extracts its content as a variable-length list of `Page` objects.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `model` | No | Model identifier, model reference (see [Model References](model-references.md)), or an inline settings table (see [Inline Settings](model-references.md#inline-settings)). |
| `max_page_images` | No | Maximum number of page images to process. |
| `page_image_captions` | No | Whether to generate captions for page images. |
| `page_views` | No | Whether to generate page views. |
| `page_views_dpi` | No | DPI for page view rendering. |
| `render_js` | No | For web-page extraction: render JavaScript before fetching the page content (when the backend supports it). Default: `false`. |
| `include_raw_html` | No | For web-page extraction: include the page's raw HTML in each extracted `Page`'s `raw_html` field. Default: `false`. |

**Constraints:** PipeExtract requires exactly one input (typically `Document` or a concept refining it) and the output must be `"Page[]"`. The input document URL can be a web page URL for web content extraction. `render_js` and `include_raw_html` apply only to web-page extraction; backends extracting from local documents may ignore them.

## PipeSearch

Searches the web using a search provider and returns structured results with an answer and source citations.

```toml
[pipe.search_topic]
type        = "PipeSearch"
description = "Search the web for information about a topic"
inputs      = { topic = "Text" }
output      = "SearchResult"
model       = "$standard"
prompt      = "What's the latest news on $topic?"
```

**What this does:** Takes a `Text` input, sends a search query to a web search provider, and produces a `SearchResult` output containing a synthesized answer and a list of sources.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `prompt` | Yes | The search query template. Supports Jinja2 syntax and `$variable` shorthand. |
| `model` | No | Model identifier, model reference (see [Model References](model-references.md)), or an inline settings table (see [Inline Settings](model-references.md#inline-settings)). |
| `from_date` | No | Start date filter in ISO 8601 format (YYYY-MM-DD). Only return results from this date onwards. |
| `to_date` | No | End date filter in ISO 8601 format (YYYY-MM-DD). Only return results up to this date. |
| `include_domains` | No | Restrict search to these domains only (e.g., `["reuters.com", "bbc.com"]`). |
| `exclude_domains` | No | Exclude results from these domains. |

**Constraints:** The output must be `SearchResult` or a concept that refines `SearchResult`.

## PipeCompose

Composes output by assembling data from working memory. PipeCompose has two modes: **template mode** and **construct mode**. Exactly one must be used.

### Template Mode

Uses a Jinja2 template to produce text output:

```toml
[pipe.format_report]
type        = "PipeCompose"
description = "Format analysis results into a report"
inputs      = { analysis = "CVAnalysis", candidate_name = "Text" }
output      = "Text"
template    = """
# Report for $candidate_name

@analysis.summary

Skills: $analysis.skills
"""
```

The `template` field can be a plain string (as above) or a table with additional options:

```toml
[pipe.format_report.template]
template = "# Report for $candidate_name"
category = "markdown"

[pipe.format_report.template.templating_style]
tag_style   = "xml"
text_format = "markdown"
```

#### Template Shorthand Syntax

MTHDS templates support three shorthand patterns that are preprocessed into Jinja2 before rendering. Raw Jinja2 (`{{ }}`, `{% %}`) is always available alongside the shorthands.

| Shorthand | Jinja2 Expansion | Purpose |
|-----------|-----------------|---------|
| `$variable` | `{{ variable|format() }}` | **Inline substitution** — embed a value within a sentence. |
| `@variable` | `{{ variable|tag("variable") }}` | **Block insertion** — insert content as a standalone, tagged block. |
| `@?variable` | `{% if variable %}{{ variable|tag("variable") }}{% endif %}` | **Conditional insertion** — render only if the variable is truthy. |

**Notes:**

- **Dotted paths** are supported with all three patterns: `$user.name`, `@doc.summary`, `@?extra.notes`.
- **Trailing dots** are treated as punctuation, not part of the path: `$amount.` expands to `{{ amount|format() }}.`
- **Dollar amounts** like `$100` or `$1,000` are **not** matched — the character after `$` must be a letter or underscore.
- **Raw Jinja2** is always available: `{{ variable_name }}`, `{% for item in items %}`, etc.

**Example combining all three patterns:**

```toml
[pipe.compose_prompt]
type        = "PipeCompose"
description = "Build an LLM prompt from structured inputs"
inputs      = { topic = "Text", context = "Text", guidelines = "Text" }
output      = "Text"
template    = """
Write an article about $topic.

@context

@?guidelines
"""
```

Here, `$topic` is inlined into the sentence, `@context` is inserted as a tagged block, and `@?guidelines` appears only if the variable has a value.

#### Template Categories

The `category` field determines which Jinja2 filters are registered and how the template environment is configured.

| Category | Use When | Autoescape | Trim Blocks | Available Filters |
|----------|----------|:----------:|:-----------:|-------------------|
| `basic` | General-purpose text composition | No | No | `format`, `tag` |
| `expression` | Simple expression evaluation | No | No | *(none)* |
| `html` | Generating HTML content | Yes | Yes | `format`, `tag`, `escape_script_tag` |
| `markdown` | Generating Markdown content | No | Yes | `format`, `tag`, `escape_script_tag` |
| `mermaid` | Generating Mermaid diagrams | No | No | *(none)* |
| `llm_prompt` | Composing prompts for LLMs | No | No | `format`, `tag`, `with_images` |
| `img_gen_prompt` | Composing prompts for image generation | No | No | `format`, `tag`, `with_images` |

**Guidance:** Use `basic` for most templates. Use `html` when generating web content (autoescape prevents XSS). Use `llm_prompt` when composing prompts that may include image references.

#### Available Filters

Filters transform variable content during rendering. The `$` and `@` shorthands apply `format()` and `tag()` automatically — you only need to call filters explicitly when using raw Jinja2 syntax.

| Filter | Syntax | Description | Categories |
|--------|--------|-------------|------------|
| `format` | `{{ var|format(text_format?) }}` | Formats a value as text. Optional `text_format` parameter: `plain`, `markdown`, `html`, `json`. Uses the context default if omitted. Applied automatically by `$`. | basic, html, markdown, llm_prompt, img_gen_prompt |
| `tag` | `{{ var|tag(tag_name?) }}` | Wraps content in tags based on the template's `tag_style`. The tag name defaults to the variable name. Applied automatically by `@` and `@?`. | basic, html, markdown, llm_prompt, img_gen_prompt |
| `escape_script_tag` | `{{ var|escape_script_tag() }}` | Escapes `</script>` tags to prevent injection. | html, markdown |
| `with_images` | `{{ var|with_images() }}` | Extracts nested images from structured content and returns text with `[Image N]` placeholders. | llm_prompt, img_gen_prompt |

#### Template Context

All declared inputs are available as variables in the template. The optional `extra_context` field (table form only) injects additional static variables:

```toml
[pipe.format_report.template]
template = "Version: $version — Report for $candidate_name"
category = "basic"

[pipe.format_report.template.extra_context]
version = "2.0"
```

Every variable referenced in the template must correspond to a declared input or an `extra_context` key.

**`category` values:** `basic`, `expression`, `html`, `markdown`, `mermaid`, `llm_prompt`, `img_gen_prompt`.

The optional `templating_style` table controls output formatting with `tag_style` (`no_tag`, `ticks`, `xml`, `square_brackets`) and `text_format` (`plain`, `markdown`, `html`, `json`). See the [specification](../spec/mthds-format.md#templating-style) for details.

### Construct Mode

Composes structured output field-by-field from working memory:

```toml
[pipe.compose_interview_sheet]
type        = "PipeCompose"
description = "Compose the final interview sheet"
inputs      = { match_analysis = "MatchAnalysis", interview_questions = "InterviewQuestion[]" }
output      = "InterviewSheet"

[pipe.compose_interview_sheet.construct]
overall_match_score  = { from = "match_analysis.overall_match_score" }
matching_skills      = { from = "match_analysis.matching_skills" }
missing_skills       = { from = "match_analysis.missing_skills" }
questions            = { from = "interview_questions" }
```

Each field in the `construct` table defines how a field of the output concept is composed:

| Value form | Method | Description |
|------------|--------|-------------|
| Literal (`string`, `integer`, `float`, `boolean`, `array`) | Fixed | The field value is the literal. |
| `{ from = "path" }` | Variable reference | The field value comes from a variable in working memory. |
| `{ from = "path", list_to_dict_keyed_by = "attr" }` | Variable reference with transform | Converts a list to a dict keyed by the named attribute. |
| `{ template = "..." }` | Template | The field value is rendered from a Jinja2 template string. |
| Nested table (no `from` or `template` key) | Nested construct | The field is recursively composed. |

**Constraint:** PipeCompose output must be a single concept — multiplicity (`[]` or `[N]`) is not allowed.

## See Also

- [Specification: Pipe Definitions](../spec/mthds-format.md#pipe-definitions) — normative reference for all pipe types and validation rules.
- [Pipes — Controllers](pipes-controllers.md) — orchestrating multiple pipes.


## Pipes — Controllers

# Pipes — Controllers

Controllers are pipes that orchestrate other pipes. They do not perform transformations themselves — they arrange when and how operator pipes (and other controllers) execute.

## PipeSequence

Executes a series of pipes in order. Each step's output is added to [working memory](working-memory.md), where subsequent steps can consume it.

```toml
[pipe.process_document]
type        = "PipeSequence"
description = "Full document processing pipeline"
inputs      = { document = "Document" }
output      = "AnalysisResult"
steps = [
    { pipe = "extract_pages", result = "pages" },
    { pipe = "analyze_content", result = "analysis" },
    { pipe = "generate_summary", result = "summary" },
]
```

**What this does:** Runs `extract_pages` first, stores its output as `pages` in working memory. Then runs `analyze_content` (which can use `pages`), stores the result as `analysis`. Finally runs `generate_summary`, producing the final `AnalysisResult`.

**Step fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `pipe` | Yes | Pipe reference (bare, domain-qualified, or package-qualified). |
| `result` | No | Name under which the step's output is stored in working memory. |
| `nb_output` | No | Expected number of output items. Mutually exclusive with `multiple_output`. |
| `multiple_output` | No | Whether to expect multiple output items. Mutually exclusive with `nb_output`. |
| `batch_over` | No | Working memory variable to iterate over (inline batch). Requires `batch_as`. |
| `batch_as` | No | Name for each item during inline batch iteration. Requires `batch_over`. |

A sequence must contain at least one step.

Inline batching (`batch_over` / `batch_as`) allows iterating over a list within a sequence step, without needing a dedicated `PipeBatch`. Both must be provided together, and they must not have the same value.

## PipeParallel

Executes multiple pipes concurrently. Each branch operates independently.

```toml
[pipe.extract_documents]
type        = "PipeParallel"
description = "Extract text from both CV and job offer concurrently"
inputs      = { cv_pdf = "Document", job_offer_pdf = "Document" }
output      = "Page[]"
add_each_output = true
branches = [
    { pipe = "extract_cv", result = "cv_pages" },
    { pipe = "extract_job_offer", result = "job_offer_pages" },
]
```

**What this does:** Runs `extract_cv` and `extract_job_offer` at the same time. With `add_each_output = true`, each branch's output is individually stored in working memory under its `result` name.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `branches` | Yes | List of sub-pipe invocations to execute concurrently. |
| `add_each_output` | No | If `true`, each branch's output is stored individually. Default: `false`. |
| `combined_output` | No | Concept reference for a combined output that merges all branch results. |

At least one of `add_each_output` or `combined_output` must be set — otherwise the pipe produces no usable output.

## PipeCondition

Routes execution to different pipes based on an evaluated condition.

```toml
[pipe.route_by_document_type]
type                = "PipeCondition"
description         = "Route processing based on document type"
inputs              = { doc_request = "DocumentRequest" }
output              = "Text"
expression_template = "{{ doc_request.document_type }}"
default_outcome     = "continue"

[pipe.route_by_document_type.outcomes]
technical = "process_technical"
business  = "process_business"
legal     = "process_legal"
```

**What this does:** Evaluates `doc_request.document_type` and routes to the matching pipe. If the document type is `"technical"`, it runs `process_technical`. If no outcome matches, `"continue"` means execution proceeds without running a sub-pipe.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `expression_template` | Conditional | A Jinja2 template that evaluates to a string matching an outcome key. Exactly one of `expression_template` or `expression` is required. |
| `expression` | Conditional | A static expression string. Exactly one of `expression_template` or `expression` is required. |
| `outcomes` | Yes | Maps outcome strings to pipe references. Must have at least one entry. |
| `default_outcome` | Yes | The pipe reference (or special outcome) to use when no outcome key matches. |
| `add_alias_from_expression_to` | No | If set, stores the evaluated expression value in working memory under this name. |

**Special outcomes:** Two string values have special meaning and are not treated as pipe references:

- `"fail"` — abort execution with an error.
- `"continue"` — skip this branch and continue without executing a sub-pipe.

## PipeBatch

Maps a single pipe over each item in a list input, producing a list output.

```toml
[pipe.batch_generate_jokes]
type             = "PipeBatch"
description      = "Generate a joke for each topic"
inputs           = { topics = "Topic[]" }
output           = "Joke[]"
branch_pipe_code = "generate_joke"
input_list_name  = "topics"
input_item_name  = "topic"
```

**What this does:** Takes a list of `Topic` items and runs `generate_joke` on each one, producing a list of `Joke` outputs.

**Key fields:**

| Field | Required | Description |
|-------|----------|-------------|
| `branch_pipe_code` | Yes | The pipe reference to invoke for each item. |
| `input_list_name` | Yes | The name of the input that contains the list to iterate over. Must exist as a key in `inputs`. |
| `input_item_name` | Yes | The name under which each individual item is passed to the branch pipe. |

**Constraints:**

- `input_item_name` must not equal `input_list_name`.
- `input_item_name` must not equal any key in `inputs`.

A naming tip: use the plural for the list and its singular form for the item (e.g., list `"topics"` → item `"topic"`).

## Pipe Reference Syntax in Controllers

Every location in a controller that references another pipe supports three forms:

| Form | Syntax | Example |
|------|--------|---------|
| Bare | `pipe_code` | `"extract_clause"` |
| Domain-qualified | `domain.pipe_code` | `"legal.contracts.extract_clause"` |
| Package-qualified | `alias->domain.pipe_code` | `"docproc->extraction.extract_text"` |

These references appear in:

- `steps[].pipe` (PipeSequence)
- `branches[].pipe` (PipeParallel)
- `outcomes` values (PipeCondition)
- `default_outcome` (PipeCondition)
- `branch_pipe_code` (PipeBatch)

Pipe *definitions* (the `[pipe.<pipe_code>]` table keys) are always bare `snake_case` names. Namespacing applies only to pipe *references*.

## See Also

- [Specification: Controller Definitions](../spec/mthds-format.md#controller-pipesequence) — normative reference for all controller types and validation rules.
- [Pipes — Operators](pipes-operators.md) — the individual transformations that controllers orchestrate.


## Working Memory

# Working Memory

Working memory is the mechanism that enables data flow between pipes. It acts as a temporary store that exists for the duration of a single pipeline run.

## How Data Flows Between Pipes

When pipes are composed inside a controller (such as `PipeSequence`), the output of each pipe needs to reach subsequent pipes. Working memory handles this.

```toml
[pipe.description_to_tagline]
type = "PipeSequence"
description = "From product description to tagline and keywords"
inputs = { description = "ProductDescription" }
output = "Keyword[]"
steps = [
    { pipe = "generate_tagline", result = "tagline" },
    { pipe = "extract_keywords", result = "keywords" },
]
```

How does `extract_keywords` access the output of `generate_tagline`? Through working memory:

1. When `generate_tagline` completes, its output is stored in working memory under the name specified by `result` — here, `"tagline"`.
2. `extract_keywords` declares `tagline` in its `inputs`. The runtime matches the input name to the working memory entry and passes the data in.

This name-matching mechanism chains pipes together into a flow of typed data.

## Lifecycle

Working memory follows a simple lifecycle within a pipeline run:

1. **Creation** — Working memory is initialized when the pipeline run starts.
2. **Population** — The caller's inputs are placed into working memory before the first pipe executes.
3. **Updates** — After each pipe completes, its output is stored under the name given by the `result` field.
4. **Access** — Any subsequent pipe can consume data from working memory by declaring a matching name in its `inputs`.
5. **Disposal** — Working memory is cleared when the pipeline run completes.

## Best Practices

- **Use meaningful names.** Choose descriptive `result` values so the pipeline reads like a narrative: `result = "candidate_skills"` is clearer than `result = "data"`.
- **Declare clear contracts.** Each pipe's `inputs` field explicitly states what it needs from working memory. This makes dependencies visible at a glance.
- **Fail fast.** If a required input is missing from working memory, a compliant runtime rejects the run before the pipe executes, producing a clear error rather than a silent failure.

## See Also

- [Pipes — Controllers](pipes-controllers.md) — how PipeSequence, PipeParallel, and PipeBatch use working memory.
- [Putting It All Together](putting-it-all-together.md) — a complete bundle walkthrough showing working memory in action.


## Multiplicity

# Multiplicity

Multiplicity defines how many items a pipe accepts as input or produces as output. It is expressed with bracket notation on concept references and is fundamental to building methods that handle both single items and collections.

## Philosophy: Concepts Are Always Singular

Concepts are always defined in the singular form. Define `Keyword`, not `Keywords`. Define `Invoice`, not `Invoices`.

This is not just a naming convention — it is a design principle. A concept describes a semantic entity: what something *is*. The number of items is a circumstantial detail of the method, not part of the concept's identity:

- A pipe that extracts keywords might find 3 or 30 — each is still a `Keyword`.
- A pipe that generates product ideas might produce 5 or 10 — each remains a `ProductIdea`.

By keeping concepts singular and expressing quantity through multiplicity, MTHDS maintains a clean separation between semantics (what) and cardinality (how many).

## Output Multiplicity

Output multiplicity controls how many items a pipe produces. It is specified using bracket notation in the `output` field.

### Single Output (default)

When no brackets are used, the pipe produces exactly one item:

```toml
[pipe.summarize]
type        = "PipeLLM"
description = "Create a summary of the document"
inputs      = { document = "Text" }
output      = "Summary"
prompt      = "Summarize this document concisely: @document"
```

### Variable Output (`[]`)

Empty brackets let the model decide how many items to produce:

```toml
[pipe.extract_line_items]
type        = "PipeLLM"
description = "Extract all line items from an invoice"
inputs      = { invoice_text = "Text" }
output      = "LineItem[]"
prompt      = """
Extract all line items from this invoice:

@invoice_text

For each line item, extract the description, quantity, unit price, and total amount.
"""
```

The pipe extracts however many line items appear in the invoice — 2 for a simple receipt, 50 for a detailed purchase order.

**Common use cases:**

- Extract entities from text (unknown count in advance)
- List all items that match criteria
- Identify all occurrences of a pattern

### Fixed Output (`[N]`)

A number in brackets produces an exact count:

```toml
[pipe.generate_headline_options]
type        = "PipeLLM"
description = "Generate headline alternatives"
inputs      = { article_text = "Text" }
output      = "Headline[5]"
prompt      = """
Read this article:

@article_text

Generate 5 different headline options for this article. Make each one unique and compelling.
"""
```

**Common use cases:**

- Generate N alternatives for A/B testing
- Create a fixed set of options for user selection
- Produce a specific number of variations for comparison

### The `nb_output` Field

In a `PipeSequence` step, the `nb_output` field provides an alternative to bracket notation. It overrides the output multiplicity declared on the called pipe for that particular step invocation:

```toml
[pipe.generate_email_variants]
type        = "PipeSequence"
description = "Generate subject lines for an email"
inputs      = { email_body = "EmailContent" }
output      = "SubjectLine[]"
steps = [
    { pipe = "generate_subject_lines", nb_output = 3, result = "subject_lines" },
]
```

Here, `generate_subject_lines` may declare `output = "SubjectLine"` (singular), but `nb_output = 3` on the step tells the runtime to produce 3 items for this invocation. The `$_nb_output` variable is automatically available in the called pipe's prompt and reflects this value.

## Input Multiplicity

Input multiplicity specifies whether a pipe expects a single item or a list. It uses the same bracket notation, applied to concept references in the `inputs` field.

### Single Input (default)

No brackets — the pipe expects exactly one item:

```toml
inputs = { report = "Report" }
```

### Variable Input (`[]`)

Empty brackets — the pipe expects a list of indeterminate length:

```toml
[pipe.summarize_all_documents]
type        = "PipeLLM"
description = "Create a unified summary of multiple documents"
inputs      = { documents = "Document[]" }
output      = "Summary"
prompt      = """
Analyze all of these documents:

@documents

Create a single unified summary that captures the key points across all documents.
"""
```

### Fixed Input (`[N]`)

A number in brackets — the pipe expects exactly that many items:

```toml
[pipe.compare_two_images]
type        = "PipeLLM"
description = "Compare exactly two images side by side"
inputs      = { images = "Image[2]" }
output      = "Comparison"
prompt      = """
Compare these two images in detail:

@images

Describe their similarities, differences, and relative strengths.
"""
```

## Practical Use Cases

### Batch Processing with Variable Input

Process an unknown number of invoices, extracting structured data from each:

```toml
[pipe.extract_single_invoice]
type        = "PipeLLM"
description = "Extract data from one invoice"
inputs      = { invoice_image = "InvoiceImage" }
output      = "InvoiceData"
prompt      = "Extract all fields from this invoice: @invoice_image"

[pipe.process_invoice_batch]
type        = "PipeSequence"
description = "Process multiple invoices"
inputs      = { invoice_images = "InvoiceImage[]" }
output      = "InvoiceData[]"
steps = [
    { pipe = "extract_single_invoice", batch_over = "invoice_images", batch_as = "invoice_image", result = "all_invoice_data" }
]
```

### Fixed Alternatives for Comparison

Generate exactly 3 subject line variations for A/B testing:

```toml
[pipe.generate_subject_lines]
type        = "PipeLLM"
description = "Generate 3 subject line options"
inputs      = { email_body = "EmailContent" }
output      = "SubjectLine[3]"
prompt      = """
Email content:
@email_body

Generate 3 compelling subject lines for this email.
Each should use a different persuasion technique.
"""
```

### Entity Extraction with Unknown Count

Extract all company names mentioned in a document:

```toml
[pipe.extract_companies]
type        = "PipeLLM"
description = "Extract all company names from an article"
inputs      = { article = "Article" }
output      = "CompanyName[]"
prompt      = """
Read this article:

@article

Extract all company and organization names mentioned in the article.
Only include entities that are explicitly named.
"""
```

## Best Practices

**When to use variable output (`[]`):**

- The number of outputs depends on the content being analyzed
- You are extracting or identifying items (entities, keywords, issues)
- The count is not known until after processing

**When to use fixed output (`[N]`):**

- You need a specific number for downstream processes
- You are generating alternatives for comparison or selection
- External requirements dictate a fixed count

**When to use variable input (`[]`):**

- The pipe should handle batches of unknown size
- You are aggregating or summarizing multiple items
- You want maximum flexibility in how the pipe is called

**When to use fixed input (`[N]`):**

- The operation inherently requires a specific count (e.g., comparison of 2 items)
- The prompt logic depends on an exact number of inputs

## See Also

- [Pipes — Operators](pipes-operators.md) — multiplicity syntax in the Common Fields table.
- [Pipes — Controllers](pipes-controllers.md) — how `PipeBatch` and inline batching interact with list inputs.
- [Specification: Pipe Definitions](../spec/mthds-format.md#pipe-definitions) — normative reference for multiplicity syntax.


## Model References

# Model References

Model references tell pipes which AI model to use. Every `PipeLLM`, `PipeStructure`, `PipeImgGen`, `PipeExtract`, and `PipeSearch` accepts an optional `model` field — a string that identifies the model and, depending on its prefix, how that model is configured.

## At a Glance

MTHDS defines the following forms of model reference, most distinguished by a single-character prefix:

| Prefix | Kind | Example | Purpose |
|--------|------|---------|---------|
| `@` | Alias | `@best-claude` | Simple name-to-model-handle mapping. |
| `$` | Preset | `$writing-factual` | Model handle bundled with parameters (temperature, quality, etc.). |
| `~` | Waterfall | `~fallback-chain` | Ordered fallback list for resilience. |
| *(none)* | Handle | `claude-4.5-sonnet` | Direct model identifier. |

All forms apply uniformly to all pipe types that accept a `model` field — there is no prefix reserved for a specific operator.

## Aliases (`@`)

An alias maps a short, memorable name to a specific model handle.

```toml
[pipe.extract_cv]
type        = "PipeExtract"
description = "Extract text content from a CV PDF document"
inputs      = { cv_pdf = "Document" }
output      = "Page[]"
model       = "@default-text-from-pdf"
```

An alias is a pure name-to-handle mapping. Use aliases when you want a readable name but have no need to override model parameters.

## Presets (`$`)

A preset bundles a model handle with extra parameters, letting method authors express *how* a model should behave without hardcoding provider-specific tuning. A preset can reference an alias as its underlying handle.

```toml
[pipe.analyze_cv]
type        = "PipeLLM"
description = "Analyze a CV to extract key professional information"
output      = "CVAnalysis"
model       = "$writing-factual"
prompt      = "Analyze the following CV: @cv_pages"

[pipe.analyze_cv.inputs]
cv_pages = "Page"
```

The `$writing-factual` preset might resolve to a specific model handle with a low temperature and deterministic sampling — but the bundle author does not need to know those details. Presets decouple *intent* ("factual writing") from *implementation* ("claude-4.6-opus at temperature 0.1"). A preset can also carry parameters like `reasoning_effort` — for instance, a `$deep-analysis` preset might set `reasoning_effort = "high"` to enable extended reasoning.

## Waterfalls (`~`)

A waterfall defines an ordered fallback list of model handles. The runtime tries each model in sequence until one succeeds, providing resilience against model unavailability.

```toml
[pipe.generate_summary]
type        = "PipeLLM"
description = "Generate a summary with fallback models"
output      = "Text"
model       = "~summary-fallback"
prompt      = "Summarize the following: @document"

[pipe.generate_summary.inputs]
document = "Text"
```

The `~summary-fallback` waterfall might try a primary model first, then fall back to a secondary model if the primary is unavailable. This is useful for production methods that must remain operational even when a specific model provider has an outage.

## Handles (bare string)

A bare string — no prefix — is a direct model handle. The runtime resolves it to a concrete model without any indirection.

```toml
[pipe.generate_portrait]
type        = "PipeImgGen"
description = "Generate a portrait image from a description"
inputs      = { description = "Text" }
output      = "Image"
prompt      = "A professional portrait: $description"
model       = "nano-banana-pro"
```

Handles are the simplest form. They are convenient for quick experiments but couple the bundle to a specific model identifier.

## Which Pipes Use Model References

The following operator pipe types accept the `model` field:

| Pipe Type | Typical Use |
|-----------|-------------|
| `PipeLLM` | Large language model invocation. |
| `PipeStructure` | Structuring text into a typed concept. |
| `PipeImgGen` | Image generation. |
| `PipeExtract` | Document extraction (e.g., PDF to pages). |
| `PipeSearch` | Web search with structured results. |

All four reference forms (`$`, `@`, `~`, bare) work identically across these pipe types.

## Choosing a Reference Type

- **Use an alias (`@`)** when you want a readable, stable name for a model handle but do not need to override parameters.
- **Use a preset (`$`)** when the model needs specific parameters (temperature, quality, token limits). Presets express intent without hardcoding provider details.
- **Use a waterfall (`~`)** when resilience matters — production methods that must survive model outages benefit from ordered fallbacks.
- **Use a bare handle** for quick prototyping or when the exact model identifier is known and no indirection is needed.

!!! note "Runtime compliance"
    The MTHDS standard requires only that the `model` field be a string. The prefix convention (`$`, `@`, `~`) is a standard pattern that runtimes are expected to support, but a compliant runtime may implement model references differently — for example, treating all model strings as direct identifiers.

## Inline Settings

The `model` field can also be a TOML table instead of a string, providing full model configuration directly in the pipe definition. This is useful when a pipe needs specific model parameters that do not warrant creating a named preset.

Each pipe type that accepts `model` has a corresponding inline settings structure:

- **PipeLLM** uses `LLMSetting` — includes `model`, `temperature`, `max_tokens`, `image_detail`, `prompting_target`, `reasoning_effort`, `reasoning_budget`.
- **PipeStructure** uses `LLMSetting` — the same shape as `PipeLLM`, because the typical implementation issues an LLM call for the structuring step.
- **PipeImgGen** uses `ImgGenSetting` — includes `model`, `quality`, `nb_steps`, `guidance_scale`, `is_moderated`, `safety_tolerance`.
- **PipeExtract** uses `ExtractSetting` — includes `model`, `max_nb_images`, `image_min_size`.
- **PipeSearch** uses `SearchSetting` — includes `model`, `include_images`, `include_inline_citations`, `max_results`.

All require a `model` field (the model handle) and accept an optional `description`.

**Example — PipeLLM with inline settings:**

```toml
[pipe.analyze_cv]
type = "PipeLLM"
description = "Analyze a CV"
output = "CVAnalysis"
prompt = "Analyze: @cv_pages"
model = { model = "claude-4.5-sonnet", temperature = 0.1, max_tokens = 4096 }

[pipe.analyze_cv.inputs]
cv_pages = "Page"
```

**Example — PipeImgGen with inline settings:**

```toml
[pipe.generate_portrait]
type        = "PipeImgGen"
description = "Generate a portrait"
inputs      = { description = "Text" }
output      = "Image"
prompt      = "A professional portrait: $description"
model       = { model = "flux-pro", quality = "high" }
```

**Example — PipeExtract with inline settings:**

```toml
[pipe.extract_cv]
type        = "PipeExtract"
description = "Extract text content from a CV PDF"
inputs      = { cv_pdf = "Document" }
output      = "Page[]"
model       = { model = "gpt-4.1", max_nb_images = 10 }
```

**Example — PipeSearch with inline settings:**

```toml
[pipe.deep_search]
type        = "PipeSearch"
description = "Deep research on a topic"
inputs      = { topic = "Text" }
output      = "SearchResult"
prompt      = "What are the main details about $topic?"
model       = { model = "linkup-deep", include_images = false }
```

Inline settings and string references are mutually exclusive — the `model` field is either a string or a table, never both.

For the full field reference of each settings structure, see the [.mthds File Format specification](../spec/mthds-format.md#inline-llm-settings).

## See Also

- [Pipes — Operators](pipes-operators.md) — the pipe types that use model references.
- [Specification: Pipe Definitions](../spec/mthds-format.md#pipe-definitions) — normative reference for all pipe fields.
- [Building a Runtime: Model References](../implementers/runtime.md#model-references) — how runtimes resolve model reference strings.


## Putting It All Together

# Putting It All Together

Before moving on to domains and namespace resolution, here is a complete bundle that uses both operators and controllers. It shows how concepts, pipes, and [working memory](working-memory.md) flow together.

```toml
domain      = "joke_generation"
description = "Generating one-liner jokes from topics"
main_pipe   = "generate_jokes_from_topics"

[concept.Topic]
description = "A subject or theme that can be used as the basis for a joke."
refines     = "Text"

[concept.Joke]
description = "A humorous one-liner intended to make people laugh."
refines     = "Text"

[pipe.generate_jokes_from_topics]
type        = "PipeSequence"
description = "Generate 3 joke topics and create a joke for each"
output      = "Joke[]"
steps = [
    { pipe = "generate_topics", result = "topics" },
    { pipe = "batch_generate_jokes", result = "jokes" },
]

[pipe.generate_topics]
type   = "PipeLLM"
description = "Generate 3 distinct topics suitable for jokes"
output = "Topic[3]"
prompt = "Generate 3 distinct and varied topics for crafting one-liner jokes."

[pipe.batch_generate_jokes]
type             = "PipeBatch"
description      = "Generate a joke for each topic"
inputs           = { topics = "Topic[]" }
output           = "Joke[]"
branch_pipe_code = "generate_joke"
input_list_name  = "topics"
input_item_name  = "topic"

[pipe.generate_joke]
type        = "PipeLLM"
description = "Write a clever one-liner joke about the given topic"
inputs      = { topic = "Topic" }
output      = "Joke"
prompt      = "Write a clever one-liner joke about $topic. Be concise and witty."
```

## How It Works

1. `generate_jokes_from_topics` is a `PipeSequence` — the entry point.
2. Step 1 calls `generate_topics`, a `PipeLLM` that produces exactly 3 `Topic` items (`Topic[3]`). The result is stored in working memory as `topics`.
3. Step 2 calls `batch_generate_jokes`, a `PipeBatch` that iterates over `topics`. For each `Topic`, it invokes `generate_joke`.
4. `generate_joke` is a `PipeLLM` that takes one `topic` and produces one `Joke`.
5. The batch collects all jokes into `Joke[]`, which becomes the final output.

The concepts (`Topic` and `Joke`) both refine the native `Text` concept. The pipes — a sequence, a batch, and LLM operators — work together through working memory.


## Domains

# Domains

Domains are namespaces for concepts and pipes within a bundle. Every bundle declares exactly one domain in its header, and all concepts and pipes in that bundle belong to that domain.

## What Domains Are For

Domains serve two purposes:

1. **Organization** — group related concepts and pipes under a meaningful name. A domain like `legal.contracts` tells you what the bundle is about.
2. **Namespacing** — prevent naming collisions. Two bundles in different domains can define concepts or pipes with the same name without conflict.

## Declaring a Domain

The `domain` field in the bundle header sets the namespace:

```toml
domain = "legal.contracts"
```

Everything in this file — every concept and every pipe — belongs to `legal.contracts`.

## Hierarchical Domains

Domains can be hierarchical, using `.` as the separator:

```toml
legal
legal.contracts
legal.contracts.shareholder
```

This allows natural organization of complex knowledge areas. A large package covering legal methods might structure its domains as a tree:

- `legal` — general legal concepts and utilities
- `legal.contracts` — contract-specific methods
- `legal.contracts.shareholder` — shareholder agreement specifics

**The hierarchy is purely organizational.** There is no implicit scope or inheritance between parent and child domains. `legal.contracts` does not automatically have access to concepts defined in `legal`. If a bundle in `legal.contracts` needs a concept from `legal`, it uses an explicit domain-qualified reference — the same as any other cross-domain reference.

## Domain Naming Rules

- A domain code is one or more `snake_case` segments separated by `.`.
- Each segment must match `[a-z][a-z0-9_]*`.
- Recommended depth: 1–3 levels.
- Recommended segment length: 1–4 words.

## Reserved Domains

Three domain names are reserved and cannot be used as the first segment of any user-defined domain:

| Domain | Purpose |
|--------|---------|
| `native` | Built-in concept types (`Text`, `Image`, `Document`, etc.). |
| `mthds` | Reserved for the MTHDS standard. |
| `pipelex` | Reserved for the reference implementation. |

For example, `native.custom` and `pipelex.utils` are invalid domain names.

## Same Domain Across Bundles

Within a single package, multiple bundles can share the same domain. When they do, their concepts and pipes merge into a single namespace:

```
my-package/
├── METHODS.toml
├── general_legal.mthds       # domain = "legal"
└── legal_utils.mthds         # domain = "legal"
```

Both files contribute concepts and pipes to the `legal` domain. If both files define a concept `ContractClause`, that is a conflict — an error at load time.

## Domains Across Packages

Two packages can both declare `domain = "recruitment"`. Their concepts and pipes are completely independent — there is no merging of namespaces across packages. The package boundary is the true isolation boundary.

This means `recruitment.CandidateProfile` from Package A and `recruitment.CandidateProfile` from Package B are different things. To use something from another package, you must qualify the reference with the package alias (see [Namespace Resolution](namespace-resolution.md)).

The domain name remains valuable for **discovery**: searching for "all packages in the recruitment domain" is a meaningful query. But discovery does not merge namespaces.

## See Also

- [Specification: Domain Naming Rules](../spec/mthds-format.md#domain-naming-rules) — normative reference.
- [Namespace Resolution](namespace-resolution.md) — how references are resolved across bundles and packages.


## Namespace Resolution

# Namespace Resolution

When a pipe references a concept or another pipe, MTHDS resolves that reference through a well-defined set of rules. Understanding these rules is essential for working with multi-bundle packages and cross-package dependencies.

## Three Forms of Reference

Every reference to a concept or pipe uses one of three forms:

| Form | Syntax | Example |
|------|--------|---------|
| **Bare** | `name` | `ContractClause`, `extract_clause` |
| **Domain-qualified** | `domain_path.name` | `legal.contracts.NonCompeteClause`, `scoring.compute_score` |
| **Package-qualified** | `alias->domain_path.name` | `acme->legal.ContractClause`, `docproc->extraction.extract_text` |

## How References Are Parsed

**Cross-package references** (`->` syntax): The string is split on the first `->`. The left part is the package alias, the right part is parsed as a domain-qualified or bare reference.

**Domain-qualified references** (`.` syntax): The string is split on the **last `.`**. The left part is the domain path, the right part is the local code (concept code or pipe code).

**Disambiguation** between concepts and pipes in a domain-qualified reference relies on casing:

- `snake_case` final segment → pipe code (e.g., `scoring.compute_score`)
- `PascalCase` final segment → concept code (e.g., `scoring.WeightedScore`)

This is unambiguous because concept codes and pipe codes follow mutually exclusive casing conventions.

## Resolution Order for Bare References

### Bare Concept References

When resolving a bare concept code like `ContractClause`:

1. **Native concepts** — check if it matches a native concept code (`Text`, `Image`, etc.). Native concepts always take priority.
2. **Current bundle** — check concepts declared in the same `.mthds` file.
3. **Same domain, other bundles** — if the bundle is part of a package, check concepts in other bundles that declare the same domain.
4. **Error** — if not found in any of the above.

Bare concept references do not fall through to other domains or other packages.

### Bare Pipe References

When resolving a bare pipe code like `extract_clause`:

1. **Current bundle** — check pipes declared in the same `.mthds` file.
2. **Same domain, other bundles** — if the bundle is part of a package, check pipes in other bundles that declare the same domain.
3. **Error** — if not found.

Bare pipe references do not fall through to other domains or other packages.

## Resolution of Domain-Qualified References

When resolving `domain_path.name` (e.g., `legal.contracts.extract_clause`):

1. Look in the named domain within the **current package**.
2. If not found: **error**.

Domain-qualified references are explicit about which domain to look in. They do not fall through to dependencies.

## Resolution of Package-Qualified References

When resolving `alias->domain_path.name` (e.g., `docproc->extraction.extract_text`):

1. Identify the dependency by the alias. The alias must match a key in the `[dependencies]` section of the consuming package's `METHODS.toml`.
2. Look in the named domain of the **resolved dependency package**.
3. If not found: **error**.

**Visibility rules for cross-package pipe references:**

- The referenced pipe must be exported by the dependency package (listed in its `[exports]` section or declared as `main_pipe` in a bundle header).
- If the pipe is not exported, the reference fails with a visibility error.

**Concepts are always public.** No visibility check is needed for cross-package concept references.

## Visibility Within a Package

When a package has a `METHODS.toml` manifest:

- **Same-domain references** — always allowed. A pipe in `legal.contracts` can reference any other pipe in `legal.contracts`.
- **Cross-domain references** (within the same package) — the target pipe must be exported. A pipe in `scoring` referencing `legal.contracts.extract_clause` requires that `extract_clause` is listed in `[exports.legal.contracts]` or is the `main_pipe` of a bundle in that domain.
- **Bare references** — always allowed (they resolve within the same domain).

When no manifest is present (standalone bundle), all pipes are treated as public.

## A Concrete Example

Package A depends on Package B with alias `scoring_lib`.

Package B's manifest (`METHODS.toml`):

```toml
[package]
address = "github.com/mthds/scoring-lib"
version = "0.5.0"
description = "Scoring utilities"

[exports.scoring]
pipes = ["compute_weighted_score"]
```

Package B's bundle (`scoring.mthds`):

```toml
domain    = "scoring"
main_pipe = "compute_weighted_score"

[concept.ScoreResult]
description = "A weighted score result"

[pipe.compute_weighted_score]
type        = "PipeLLM"
description = "Compute a weighted score"
inputs      = { item = "Text" }
output      = "ScoreResult"
prompt      = "Compute a weighted score for: $item"

[pipe.internal_helper]
type        = "PipeLLM"
description = "Internal helper (not exported)"
inputs      = { data = "Text" }
output      = "Text"
prompt      = "Process: $data"
```

Package A's bundle (`analysis.mthds`):

```toml
domain = "analysis"

[pipe.analyze_item]
type        = "PipeSequence"
description = "Analyze using scoring dependency"
inputs      = { item = "Text" }
output      = "Text"
steps = [
    { pipe = "scoring_lib->scoring.compute_weighted_score", result = "score" },
    { pipe = "summarize", result = "summary" },
]
```

**Resolution of `scoring_lib->scoring.compute_weighted_score`:**

1. `->` detected — split into alias `scoring_lib` and remainder `scoring.compute_weighted_score`.
2. Look up `scoring_lib` in Package A's `[dependencies]` — found.
3. Parse remainder: split on last `.` → domain `scoring`, pipe code `compute_weighted_score`.
4. Look in domain `scoring` of Package B — pipe found.
5. Visibility check: `compute_weighted_score` is in `[exports.scoring]` — accessible.
6. Resolution succeeds.

**If Package A tried `scoring_lib->scoring.internal_helper`:**

Steps 1–4 would succeed (the pipe exists), but the visibility check would fail — `internal_helper` is not in `[exports.scoring]` and is not `main_pipe`. This is a visibility error.

**Cross-package concept references** work the same way but skip the visibility check, since concepts are always public:

```toml
[concept.DetailedScore]
description = "An extended score with additional analysis"
refines     = "scoring_lib->scoring.ScoreResult"
```

## Resolution Flowchart

Given a reference string `R`:

```
1. Does R contain "->"?
   YES → Split into (alias, remainder).
         Look up alias in [dependencies].
         Parse remainder as domain-qualified or bare ref.
         Resolve in the dependency's namespace.
         For pipes: check export visibility.
   NO  → Continue to step 2.

2. Does R contain "."?
   YES → Split on last "." into (domain_path, local_code).
         Resolve in domain_path within current package.
   NO  → R is a bare name. Continue to step 3.

3. Is R a concept code (PascalCase)?
   YES → Check native concepts → current bundle → same domain.
   NO  → R is a pipe code (snake_case).
         Check current bundle → same domain.

4. Not found? → Error.
```

## See Also

- [Specification: Namespace Resolution Rules](../spec/namespace-resolution.md) — the normative, formal definition of all resolution rules.
- [Domains](domains.md) — how domains organize concepts and pipes.
- [The Package System: Exports & Visibility](../packages/exports-visibility.md) — how packages control what they expose.


# The Package System

## Package Structure

# Package Structure

A **package** is the distribution unit of MTHDS. It is a directory that contains a manifest (`METHODS.toml`) and one or more bundles (`.mthds` files).

## A Minimal Package

```
my_tool/
├── METHODS.toml
└── main.mthds
```

This is the smallest distributable package: one manifest, one bundle. The manifest gives the package an identity — an address, a version, a description — turning a standalone bundle into something that other packages can depend on.

## A Full Package

```
legal_tools/
├── METHODS.toml
├── methods.lock
├── general_legal.mthds
├── contract_analysis.mthds
├── shareholder_agreements.mthds
├── scoring.mthds
├── README.md
└── LICENSE
```

This package has multiple bundles, each declaring its own domain (`legal`, `legal.contracts`, `legal.contracts.shareholder`, `scoring`). The `methods.lock` file records exact dependency versions for reproducible builds.

## Directory Layout Rules

- `METHODS.toml` must be at the directory root.
- `methods.lock` must be alongside `METHODS.toml` at the root.
- `.mthds` files can be at the root or in subdirectories. A compliant runtime discovers all `.mthds` files recursively.
- A single directory should contain one package.

## Standalone Bundles (No Package)

A `.mthds` file works without a package manifest. When used standalone:

- All pipes are treated as public (no visibility restrictions).
- No dependencies are available beyond [native concepts](../language/concepts.md#native-concepts).
- The bundle is not distributable (no package address).

This preserves the "single file = working method" experience for learning, prototyping, and simple projects. When you need distribution, add a `METHODS.toml` — the rest of this section shows how.

## Progressive Enhancement

The package system follows a progressive enhancement principle:

1. **Single file** — a `.mthds` bundle works on its own. No configuration, no manifest.
2. **Package** — add a `METHODS.toml` to get exports, visibility, and a globally unique identity.
3. **Dependencies** — add `[dependencies]` to compose with other packages.
4. **Ecosystem** — publish, search, and discover through the Know-How Graph.

Each layer adds capability without breaking the previous one.

## Manifest Discovery

When loading a `.mthds` bundle, a compliant runtime discovers the manifest by walking up the directory tree:

1. Check the bundle's directory for `METHODS.toml`.
2. If not found, move to the parent directory.
3. Stop when `METHODS.toml` is found, a `.git` directory is encountered, or the filesystem root is reached.
4. If no manifest is found, the bundle is treated as a standalone bundle.

## See Also

- [Specification: Package Directory Structure](../spec/manifest-format.md#package-directory-structure) — normative reference for layout rules.
- [The Manifest](manifest.md) — what goes inside `METHODS.toml`.


## The Manifest

# The Manifest

`METHODS.toml` is the package manifest — the identity card and dependency declaration for a package. It is a TOML file at the root of the package directory.

## A First Look

```toml
[package]
name          = "nda_analyzer"
address       = "github.com/acme/legal-tools"
display_name  = "Nda Analyzer"
version       = "0.3.0"
description   = "Legal document analysis and contract review methods."
authors       = ["ACME Legal Tech <legal@acme.com>"]
license       = "MIT"
mthds_version = ">=1.0.0"
main_pipe     = "analyze_nda"

[exports.legal]
pipes = ["classify_document"]

[exports.legal.contracts]
pipes = ["extract_clause", "analyze_nda", "compare_contracts"]

[exports.scoring]
pipes = ["compute_weighted_score"]
```

This manifest declares a method called `nda_analyzer` hosted at `github.com/acme/legal-tools`, version `0.3.0`. It exports specific pipes from three domains and designates `analyze_nda` as its main entry point.

## The `[package]` Section

The `[package]` section defines the package's identity:

| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | The name of the method. Must be `snake_case` (matching `[a-z][a-z0-9_]*`), 2-25 characters. The package directory name must match the `name` field exactly. |
| `address` | Yes | Globally unique identifier. Must follow the hostname/path pattern (e.g., `github.com/org/repo`). |
| `display_name` | No | Human-friendly label for CLI output and registry listings. Cosmetic only — never used as an identifier. Max 128 characters. |
| `version` | Yes | [Semantic version](https://semver.org/) (`MAJOR.MINOR.PATCH`, with optional pre-release and build metadata). |
| `description` | Yes | Human-readable summary of the package's purpose. Must not be empty. |
| `authors` | No | List of author identifiers (e.g., `"Name <email>"`). Default: empty list. |
| `license` | No | [SPDX license identifier](https://spdx.org/licenses/) (e.g., `"MIT"`, `"Apache-2.0"`). |
| `mthds_version` | No | MTHDS standard version constraint. The current standard version is `1.0.0`. |
| `main_pipe` | No | The package's entry-point pipe. Must reference a pipe declared in the `[exports]` section. See [Main Pipe](#main-pipe) below. |

## Package Addresses

The address is the globally unique identifier for a package. It doubles as the fetch location for distribution (see [Distribution](distribution.md)).

Addresses follow a hostname/path pattern:

```
github.com/acme/legal-tools
github.com/mthds/document-processing
gitlab.com/company/internal-methods
```

The address must start with a hostname (containing at least one dot), followed by a `/`, followed by one or more path segments.

Invalid addresses:

```
legal-tools               # No hostname
acme/legal-tools          # No dot in hostname
```

## Version Format

The `version` field must conform to [Semantic Versioning 2.0.0](https://semver.org/):

```
MAJOR.MINOR.PATCH[-pre-release][+build-metadata]
```

Examples: `1.0.0`, `0.3.0`, `2.1.3-beta.1`, `1.0.0-rc.1+build.42`

## Main Pipe

The `main_pipe` field designates the package's primary entry point — the pipe that runs when a user invokes the package by slug or address:

```bash
mthds run method nda_analyzer
mthds run method github.com/acme/legal-tools
```

Both commands above execute the pipe referenced by `main_pipe`.

The value is a `snake_case` pipe code that **must** match a pipe declared in the `[exports]` section:

```toml
main_pipe = "analyze_nda"
```

**Validation rules:**

- The value MUST be a valid `snake_case` pipe code (matching `[a-z][a-z0-9_]*`).
- The referenced pipe **must** be declared in the `[exports]` section. A manifest that sets `main_pipe` to a pipe not present in exports is invalid.
- The field is optional. Packages without a `main_pipe` can still be used as libraries — consumers import specific pipes by their qualified names.

## The `[dependencies]` Section

> **Not yet implemented.** Dependencies between packages are planned but not yet supported.

Dependencies are covered in detail on the [Dependencies](dependencies.md) page.

## The `[exports]` Section

Exports are covered in detail on the [Exports & Visibility](exports-visibility.md) page.

## See Also

- [Specification: METHODS.toml Manifest Format](../spec/manifest-format.md) — normative reference for all fields and validation rules.
- [Dependencies](dependencies.md) — how to declare and manage dependencies.
- [Exports & Visibility](exports-visibility.md) — how to control which pipes are public.


## Exports & Visibility

# Exports & Visibility

When a bundle is part of a package, not every pipe needs to be visible to consumers. The `[exports]` section of `METHODS.toml` controls which pipes are part of the public API.

## Default Visibility Rules

Three rules govern visibility:

- **Concepts are always public.** Concepts are vocabulary — they are always accessible from outside the package.
- **Pipes are private by default.** A pipe not listed in `[exports]` is an implementation detail, invisible to consumers.
- **`main_pipe` must be exported.** If a package declares a `main_pipe`, that pipe MUST appear in the `[exports]` section.

## Declaring Exports

The `[exports]` section uses nested TOML tables that mirror the domain hierarchy. The domain path maps directly to the TOML table path:

```toml
[exports.legal]
pipes = ["classify_document"]

[exports.legal.contracts]
pipes = ["extract_clause", "analyze_nda", "compare_contracts"]

[exports.scoring]
pipes = ["compute_weighted_score"]
```

Each table contains a `pipes` list — the pipe codes that are public from that domain. A domain can have both a `pipes` list and sub-domain tables (e.g., `[exports.legal]` with `pipes` and `[exports.legal.contracts]`).

## How Visibility Works in Practice

Consider a package with two domains and this manifest:

```toml
[exports.scoring]
pipes = ["compute_weighted_score"]
```

**Bundles in the `scoring` domain** can reference any pipe within `scoring` freely — same-domain references are always allowed.

**Bundles in other domains** (say, `analysis`) can reference `scoring.compute_weighted_score` because it is exported. They cannot reference `scoring.internal_helper` because it is not in the exports list.

**External packages** that depend on this package follow the same rule: only exported pipes are accessible via [cross-package references](cross-package-references.md).

## Intra-Package Visibility Summary

| Reference type | Allowed? |
|---------------|----------|
| Bare references (same bundle or same domain) | Always |
| Cross-domain references to exported pipes | Yes |
| Cross-domain references to non-exported pipes | No — visibility error |

## Standalone Bundles

When no manifest is present (standalone bundle), all pipes are treated as public. Visibility restrictions only apply when a `METHODS.toml` exists.

## Reserved Domains in Exports

Domain paths in `[exports]` must not start with a reserved domain segment (`native`, `mthds`, `pipelex`). A manifest with `[exports.native]` or `[exports.pipelex.utils]` is invalid.

## See Also

- [Specification: The `[exports]` Section](../spec/manifest-format.md#the-exports-section) — normative reference.
- [Namespace Resolution](../language/namespace-resolution.md) — how visibility interacts with reference resolution.


## Dependencies

# Dependencies

> **Not yet implemented.** Dependencies between packages are planned but not yet supported. The documentation below describes the intended behavior for a future release.

Dependencies allow a package to build on other packages. Each dependency is declared in the `[dependencies]` section of `METHODS.toml` with an alias, an address, and a version constraint.

## Declaring Dependencies

```toml
[dependencies]
docproc     = { address = "github.com/mthds/document-processing", version = "^1.0.0" }
scoring_lib = { address = "github.com/mthds/scoring-lib", version = "^0.5.0" }
```

Each key (`docproc`, `scoring_lib`) is the **alias** — a short `snake_case` name used in [cross-package references](cross-package-references.md) (`alias->domain.name`).

## Dependency Fields

| Field | Required | Description |
|-------|----------|-------------|
| `address` | Yes | The dependency's package address (hostname/path pattern). |
| `version` | Yes | Version constraint (see below). |
| `path` | No | Local filesystem path, for development-time workflows. |

## Aliases

The alias is the TOML key for each dependency entry. It must be `snake_case` (matching `[a-z][a-z0-9_]*`), and all aliases within a single manifest must be unique.

Aliases appear in cross-package references:

```toml
steps = [
    { pipe = "docproc->extraction.extract_text", result = "pages" },
    { pipe = "scoring_lib->scoring.compute_weighted_score", result = "score" },
]
```

Choose aliases that are short, meaningful, and easy to read in references.

## Version Constraints

Version constraints specify which versions of a dependency are acceptable:

| Form | Syntax | Example | Meaning |
|------|--------|---------|---------|
| Exact | `MAJOR.MINOR.PATCH` | `1.0.0` | Exactly this version. |
| Caret | `^MAJOR.MINOR.PATCH` | `^1.0.0` | Compatible release (same major version). |
| Tilde | `~MAJOR.MINOR.PATCH` | `~1.0.0` | Approximately compatible (same major.minor). |
| Greater-or-equal | `>=MAJOR.MINOR.PATCH` | `>=1.0.0` | This version or newer. |
| Less-than | `<MAJOR.MINOR.PATCH` | `<2.0.0` | Older than this version. |
| Compound | constraint `, ` constraint | `>=1.0.0, <2.0.0` | Both constraints must be satisfied. |
| Wildcard | `*`, `MAJOR.*` | `1.*` | Any version matching the prefix. |

Additional operators `>`, `<=`, `==`, and `!=` are also supported. Partial versions are allowed: `1.0` is equivalent to `1.0.*`.

## Local Path Dependencies

For development-time workflows where packages are co-located on disk, add a `path` field:

```toml
[dependencies]
scoring = { address = "github.com/mthds/scoring-lib", version = "^0.5.0", path = "../scoring-lib" }
```

When `path` is set, the dependency is resolved from the local filesystem instead of being fetched via VCS. The path is resolved relative to the directory containing `METHODS.toml`.

This is similar to Cargo's `path` dependencies or Go's `replace` directives.

**Important behaviors of local path dependencies:**

- They are NOT resolved transitively — only the root package's local paths are honored.
- They are excluded from the [lock file](lock-file.md).
- When publishing, the `path` field is informational — consumers fetch via the `address`.

## See Also

- [Specification: The `[dependencies]` Section](../spec/manifest-format.md#the-dependencies-section) — normative reference for all fields.
- [Specification: Version Constraint Syntax](../spec/manifest-format.md#version-constraint-syntax) — full syntax reference.
- [Version Resolution](version-resolution.md) — how dependency versions are selected.
- [Cross-Package References](cross-package-references.md) — how aliases are used in `.mthds` files.


## Cross-Package References

# Cross-Package References

When your bundle needs a pipe or concept from another package, you use a **cross-package reference** — the `->` syntax that reaches into a dependency.

## The `->` Syntax

```toml
steps = [
    { pipe = "scoring_lib->scoring.compute_weighted_score", result = "score" },
]
```

This reference reads as: "from the package aliased as `scoring_lib`, get the pipe `compute_weighted_score` in the `scoring` domain."

The `->` separator was chosen for readability. It reads as natural language — "from scoring_lib, get..." — and is visually distinct from the `.` used for domain paths.

## Anatomy of a Cross-Package Reference

```
scoring_lib -> scoring.compute_weighted_score
  alias     ↑     domain   pipe code
         separator
```

1. **Alias** — the `snake_case` key from `[dependencies]` in `METHODS.toml`.
2. **`->`** — the cross-package separator.
3. **Domain-qualified name** — parsed by splitting on the last `.`: domain path `scoring`, pipe code `compute_weighted_score`.

## Referencing Pipes

Cross-package pipe references appear in all the same locations as domain-qualified pipe references:

- `steps[].pipe` in PipeSequence
- `branches[].pipe` in PipeParallel
- `outcomes` values in PipeCondition
- `default_outcome` in PipeCondition
- `branch_pipe_code` in PipeBatch

```toml
[pipe.full_analysis]
type        = "PipeSequence"
description = "Run external scoring and local summary"
inputs      = { item = "Text" }
output      = "Text"
steps = [
    { pipe = "scoring_lib->scoring.compute_weighted_score", result = "score" },
    { pipe = "summarize_score", result = "summary" },
]
```

**Visibility constraint:** The referenced pipe must be exported by the dependency package — listed in its `[exports]` section.

## Referencing Concepts

Cross-package concept references work the same way, appearing in `inputs`, `output`, `refines`, `concept_ref`, `item_concept_ref`, and `combined_output`:

```toml
[concept.DetailedScore]
description = "An extended score with additional analysis"
refines     = "scoring_lib->scoring.ScoreResult"
```

**Concepts are always public.** No visibility check is needed for cross-package concept references.

## A Complete Example

**Setup:** Package A depends on Package B with alias `scoring_lib`.

Package B's manifest:

```toml
[package]
address     = "github.com/mthds/scoring-lib"
version     = "0.5.0"
description = "Scoring utilities"

[exports.scoring]
pipes = ["compute_weighted_score"]
```

Package B's bundle (`scoring.mthds`):

```toml
domain    = "scoring"
main_pipe = "compute_weighted_score"

[concept.ScoreResult]
description = "A weighted score result"

[pipe.compute_weighted_score]
type        = "PipeLLM"
description = "Compute a weighted score"
inputs      = { item = "Text" }
output      = "ScoreResult"
prompt      = "Compute a weighted score for: $item"

[pipe.internal_helper]
type        = "PipeLLM"
description = "Internal helper (not exported)"
inputs      = { data = "Text" }
output      = "Text"
prompt      = "Process: $data"
```

Package A's bundle (`analysis.mthds`):

```toml
domain = "analysis"

[pipe.analyze_item]
type        = "PipeSequence"
description = "Analyze using scoring dependency"
inputs      = { item = "Text" }
output      = "Text"
steps = [
    { pipe = "scoring_lib->scoring.compute_weighted_score", result = "score" },
    { pipe = "summarize", result = "summary" },
]
```

**What works:**

- `scoring_lib->scoring.compute_weighted_score` resolves because `compute_weighted_score` is exported.
- `scoring_lib->scoring.ScoreResult` (concept reference) resolves because concepts are always public.

**What fails:**

- `scoring_lib->scoring.internal_helper` — visibility error: `internal_helper` is not in `[exports.scoring]`.

## See Also

- [Specification: Namespace Resolution Rules](../spec/namespace-resolution.md) — formal resolution algorithm.
- [Namespace Resolution](../language/namespace-resolution.md) — the three tiers of reference resolution.
- [Exports & Visibility](exports-visibility.md) — how exports control what is accessible.


## The Lock File

# The Lock File

The `methods.lock` file records the exact resolved versions and integrity hashes for all remote dependencies. It enables reproducible builds — every developer and CI system gets the same dependency versions.

## What It Looks Like

```toml
["github.com/mthds/document-processing"]
version = "1.2.3"
hash    = "sha256:a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2"
source  = "https://github.com/mthds/document-processing"

["github.com/mthds/scoring-lib"]
version = "0.5.1"
hash    = "sha256:e5f6a7b8c9d0e5f6a7b8c9d0e5f6a7b8c9d0e5f6a7b8c9d0e5f6a7b8c9d0e5f6"
source  = "https://github.com/mthds/scoring-lib"
```

Each entry records a package address, the exact resolved version, a SHA-256 integrity hash, and the HTTPS source URL.

## File Location

The lock file must be named `methods.lock` and placed at the package root, alongside `METHODS.toml`. It should be committed to version control.

## Locked Package Fields

| Field | Description |
|-------|-------------|
| `version` | The exact resolved version (valid semver). |
| `hash` | SHA-256 integrity hash of the package contents (`sha256:` followed by 64 hex characters). |
| `source` | The HTTPS URL from which the package was fetched. |

## Which Packages Are Locked

- **Remote dependencies** (those without a `path` field) are locked, including all transitive remote dependencies.
- **Local path dependencies** are NOT locked. They are resolved from the filesystem at load time and are expected to change during development.

## How the Hash Is Computed

The integrity hash is a deterministic SHA-256 hash of the package directory:

1. Collect all regular files recursively under the package directory.
2. Exclude any path containing `.git` in its components.
3. Sort files by their POSIX-normalized relative path (for cross-platform determinism).
4. For each file in sorted order, feed into the hasher:
    - The relative path string, encoded as UTF-8.
    - The raw file bytes.
5. Format as `sha256:` followed by the 64-character lowercase hex digest.

## When the Lock File Updates

The lock file is regenerated when:

- `mthds pkg lock` is run — resolves all dependencies and writes the lock file.
- `mthds pkg update` is run — re-resolves to latest compatible versions and rewrites the lock file.
- `mthds pkg add` is run — adds a new dependency and may trigger re-resolution.

## Verification

When installing from a lock file (`mthds pkg install`), the runtime:

1. Locates the cached package directory for each entry.
2. Recomputes the SHA-256 hash using the algorithm above.
3. Compares the computed hash with the lock file's `hash` field.
4. Rejects the installation if any hash does not match.

## Deterministic Output

Lock file entries are sorted by package address (lexicographic ascending) to produce clean version control diffs.

## See Also

- [Specification: methods.lock Format](../spec/lock-format.md) — normative reference.
- [Distribution](distribution.md) — how packages are fetched and cached.
- [Version Resolution](version-resolution.md) — how versions are selected.


## Distribution

# Distribution

MTHDS packages are distributed using a federated model: decentralized storage with centralized discovery.

## Storage: Git Repositories

Packages live in Git repositories. The repository IS the package — no upload step, no proprietary hosting. Authors retain full control.

A repository can contain one package (at the root) or multiple packages (in subdirectories with distinct addresses).

## Addressing and Fetching

Package addresses map directly to Git clone URLs:

1. Prepend `https://`.
2. Append `.git` (if not already present).

```
github.com/acme/legal-tools → https://github.com/acme/legal-tools.git
```

The resolution chain when fetching a dependency:

1. **Local path** — if the dependency has a `path` field in `METHODS.toml`, resolve from the local filesystem.
2. **Local cache** — check `~/.mthds/packages/{address}/{version}/` for a cached copy.
3. **VCS fetch** — clone the repository at the resolved version tag using `git clone --depth 1 --branch {tag}`.

## Version Tags

Version tags in remote repositories may use a `v` prefix (e.g., `v1.0.0`). The prefix is stripped during version parsing. Both `v1.0.0` and `1.0.0` are recognized.

Tags are listed using `git ls-remote --tags`, and only those that parse as valid semantic versions are considered.

## Package Cache

Fetched packages are cached locally to avoid repeated clones:

```
~/.mthds/packages/{address}/{version}/
```

For example:

```
~/.mthds/packages/github.com/acme/legal-tools/1.0.0/
```

The `.git` directory is removed from cached copies to save space. Cache writes use a staging directory with atomic rename for safety.

## Discovery: Registry Indexes

One or more registry services index packages without owning them. A registry provides:

- **Search** — by domain, by concept, by pipe signature, by description.
- **Type-compatible search** — "find all pipes that accept `LoanApplication` and produce something refining `RiskAssessment`" (enabled by the MTHDS type system).
- **Metadata** — versions, descriptions, licenses, dependency graphs.
- **Concept/pipe browsing** — navigate the refinement hierarchy, explore pipe signatures.

Registries build their index by crawling known package addresses, parsing `METHODS.toml` for metadata, and parsing `.mthds` files for concept definitions and pipe signatures. No data is duplicated — everything is derived from the source files.

## Multi-Tier Deployment

MTHDS supports multiple deployment tiers, from local to community-wide:

| Tier | Scope | Typical use |
|------|-------|-------------|
| **Local** | Single `.mthds` file, no manifest | Learning, prototyping, one-off methods |
| **Project** | Package in a project repo | Team methods, versioned with the codebase |
| **Organization** | Internal registry/proxy | Company-wide approved methods, governance |
| **Community** | Public Git repos + public registries | Open-source Know-How Graph |

## See Also

- [The Registry](registry.md) — the HTTP service that indexes packages and powers discovery.
- [Registry Distribution Protocol](registry-distribution.md) — proxy chains, signed manifests, and multi-tier deployment.
- [Specification: Fetching Remote Dependencies](../spec/namespace-resolution.md#fetching-remote-dependencies) — normative reference for the fetch algorithm.
- [Specification: Cache Layout](../spec/namespace-resolution.md#cache-layout) — normative reference for cache paths.
- [The Lock File](lock-file.md) — how fetched versions are pinned.
- [The Know-How Graph](../know-how-graph/index.md) — typed discovery across packages.


## Version Resolution

# Version Resolution

When multiple packages depend on different versions of the same dependency, MTHDS needs a strategy to pick a single version. MTHDS uses **Minimum Version Selection** (MVS), the same approach used by Go modules.

## How MVS Works

Given a set of version constraints for a package, MVS:

1. Collects all version constraints from all dependents (direct and transitive).
2. Lists all available versions from VCS tags.
3. Sorts versions in ascending order.
4. Selects the **minimum** version that satisfies **all** constraints simultaneously.

If no version satisfies all constraints, the resolution fails with an error.

## An Example

Package A requires `>=1.0.0` of Library X. Package B requires `>=1.2.0` of Library X. Available versions of Library X: `1.0.0`, `1.1.0`, `1.2.0`, `1.3.0`, `2.0.0`.

MVS selects `1.2.0` — the minimum version that satisfies both `>=1.0.0` and `>=1.2.0`.

A maximum-version resolver would select `2.0.0`. MVS deliberately avoids this: you get the version you asked for, not the latest one.

## Why MVS?

- **Deterministic** — the same set of constraints always produces the same result, regardless of when you run the resolver.
- **Reproducible** — no dependency on a "latest" query or timestamp. The result depends only on the constraints and the available tags.
- **Simple** — no backtracking solver needed. Sort and pick the first match.
- **Conservative** — you get the minimum version that works, reducing the risk of pulling in untested changes.

## Transitive Dependencies

Dependencies are resolved transitively with these rules:

- **Remote dependencies** are resolved recursively. If Package A depends on Package B, and Package B depends on Package C, then Package C is also resolved.
- **Local path dependencies** are resolved at the root level only. They are NOT resolved transitively — only the root package's local paths are honored.
- **Cycle detection** — if a dependency is encountered while it is already being resolved, the resolver reports a cycle error.
- **Diamond dependencies** — when the same package address is required by multiple dependents with different version constraints, MVS selects the minimum version satisfying all constraints simultaneously.

## Diamond Dependencies

Diamond dependencies occur when two or more packages depend on the same third package:

```
Your Package
├── Package A (requires Library X ^1.0.0)
└── Package B (requires Library X ^1.2.0)
```

MVS handles this naturally: it collects both constraints (`^1.0.0` and `^1.2.0`), lists available versions, and picks the minimum version satisfying both. If constraints are incompatible (e.g., `^1.0.0` and `^2.0.0` have no overlapping range), the resolver reports an error.

## See Also

- [Specification: Version Resolution Strategy](../spec/namespace-resolution.md#version-resolution-strategy) — normative reference.
- [Specification: Transitive Dependency Resolution](../spec/namespace-resolution.md#transitive-dependency-resolution) — normative reference for transitive resolution rules.
- [Dependencies](dependencies.md) — how to declare version constraints.
- [The Lock File](lock-file.md) — how resolved versions are recorded.


## Registry Overview

# The Registry

!!! note "Implementation status"
    An early beta of the MTHDS registry is available at [mthds.sh](https://mthds.sh). The full specification below describes the target API surface.

A **registry** is an HTTP service that indexes MTHDS packages and exposes them for discovery. Registries do not host package source code — they index metadata from Git-hosted packages and serve it through a structured API.

## Role in the Ecosystem

The [distribution model](distribution.md) separates storage from discovery:

- **Storage** remains decentralized — packages live in Git repositories controlled by their authors.
- **Discovery** is centralized per registry — a registry crawls known package addresses, builds an index, constructs the [Know-How Graph](../know-how-graph/index.md), and serves queries over HTTP.

Multiple registries can coexist. A client can query several registries in order (analogous to Go's `GOPROXY` chain), falling back from one to the next.

## API Versioning

All endpoints are prefixed with `/v1/`. The version number increments only for breaking changes. Non-breaking additions (new optional fields, new endpoints) do not require a version bump.

```
https://registry.example.com/v1/packages
```

## Endpoints

### List Packages

```
GET /v1/packages?offset=0&limit=20
```

Returns a paginated list of indexed packages.

**Response:**

```json
{
  "items": [
    {
      "address": "github.com/acme/legal-tools",
      "version": "1.2.0",
      "description": "Contract analysis and clause extraction methods",
      "authors": ["Acme Legal Team"],
      "license": "Apache-2.0",
      "domains": [
        { "domain_code": "legal.contracts", "description": "Contract processing domain" }
      ],
      "concept_count": 5,
      "pipe_count": 8,
      "dependency_count": 2
    }
  ],
  "total": 47,
  "offset": 0,
  "limit": 20
}
```

### Get Package Detail

```
GET /v1/packages/{address}
```

The `{address}` path parameter is the full package address, URL-encoded where necessary (e.g., `github.com%2Facme%2Flegal-tools`).

Returns the full `PackageIndexEntry` for a single package.

**Response:**

```json
{
  "address": "github.com/acme/legal-tools",
  "version": "1.2.0",
  "description": "Contract analysis and clause extraction methods",
  "authors": ["Acme Legal Team"],
  "license": "Apache-2.0",
  "domains": [
    { "domain_code": "legal.contracts", "description": "Contract processing domain" }
  ],
  "concepts": [
    {
      "concept_code": "ContractClause",
      "domain_code": "legal.contracts",
      "concept_ref": "legal.contracts.ContractClause",
      "description": "A single clause extracted from a contract",
      "refines": "native.Text",
      "structure_fields": ["clause_type", "text", "section_number"]
    }
  ],
  "pipes": [
    {
      "pipe_code": "extract_clause",
      "pipe_type": "PipeLLM",
      "domain_code": "legal.contracts",
      "description": "Extract a specific clause from a contract document",
      "input_specs": { "source": "ContractDocument" },
      "output_spec": "ContractClause",
      "is_exported": true
    }
  ],
  "dependencies": ["github.com/mthds/document-processing"],
  "dependency_aliases": { "doc_processing": "github.com/mthds/document-processing" }
}
```

### Text Search

```
GET /v1/search?q=contract&type=concept&domain=legal.contracts&offset=0&limit=20
```

**Query parameters:**

| Parameter | Required | Description |
|-----------|----------|-------------|
| `q` | Yes | Search term. Case-insensitive substring match against concept codes, pipe codes, descriptions, and domain codes. |
| `type` | No | Filter by entity type: `concept`, `pipe`, or omit for both. |
| `domain` | No | Filter results to a specific domain code. |
| `offset` | No | Pagination offset. Default: `0`. |
| `limit` | No | Page size. Default: `20`. Maximum: `100`. |

**Response:**

```json
{
  "items": [
    {
      "kind": "concept",
      "package_address": "github.com/acme/legal-tools",
      "concept_code": "ContractClause",
      "domain_code": "legal.contracts",
      "description": "A single clause extracted from a contract",
      "refines": "native.Text"
    },
    {
      "kind": "pipe",
      "package_address": "github.com/acme/legal-tools",
      "pipe_code": "extract_clause",
      "pipe_type": "PipeLLM",
      "domain_code": "legal.contracts",
      "description": "Extract a specific clause from a contract document",
      "input_specs": { "source": "ContractDocument" },
      "output_spec": "ContractClause",
      "is_exported": true
    }
  ],
  "total": 2,
  "offset": 0,
  "limit": 20
}
```

### Type-Compatible Search

```
GET /v1/search/typed?accepts=Document&produces=ContractClause
```

Finds pipes by their typed signatures, using the [Know-How Graph](../know-how-graph/index.md) to resolve concept compatibility through refinement chains.

**Query parameters:**

| Parameter | Required | Description |
|-----------|----------|-------------|
| `accepts` | No | Concept code or reference. Finds pipes that accept this concept as input (including concepts that this one refines). |
| `produces` | No | Concept code or reference. Finds pipes that produce this concept as output (including concepts that refine this one). |
| `offset` | No | Pagination offset. Default: `0`. |
| `limit` | No | Page size. Default: `20`. Maximum: `100`. |

At least one of `accepts` or `produces` MUST be provided.

**Response:**

```json
{
  "items": [
    {
      "package_address": "github.com/acme/legal-tools",
      "pipe_code": "extract_clause",
      "pipe_type": "PipeLLM",
      "domain_code": "legal.contracts",
      "description": "Extract a specific clause from a contract document",
      "input_specs": { "source": "ContractDocument" },
      "output_spec": "ContractClause",
      "is_exported": true
    }
  ],
  "total": 1,
  "offset": 0,
  "limit": 20
}
```

### Graph Chain Query

```
GET /v1/graph/chains?from={concept_id}&to={concept_id}&max_depth=3
```

Finds multi-step pipe chains that transform one concept into another, using BFS traversal of the Know-How Graph.

**Query parameters:**

| Parameter | Required | Description |
|-----------|----------|-------------|
| `from` | Yes | Source concept ID in `package_address::concept_ref` format (e.g., `__native__::native.Document`). |
| `to` | Yes | Target concept ID in `package_address::concept_ref` format. |
| `max_depth` | No | Maximum number of pipes in a chain. Default: `3`. |

**Response:**

```json
{
  "chains": [
    {
      "steps": [
        {
          "pipe_key": "github.com/acme/legal-tools::extract_pages",
          "pipe_code": "extract_pages",
          "package_address": "github.com/acme/legal-tools",
          "input_specs": { "source": "Document" },
          "output_spec": "PageContent"
        },
        {
          "pipe_key": "github.com/acme/legal-tools::extract_clause",
          "pipe_code": "extract_clause",
          "package_address": "github.com/acme/legal-tools",
          "input_specs": { "source": "PageContent" },
          "output_spec": "ContractClause"
        }
      ]
    }
  ],
  "from": "__native__::native.Document",
  "to": "github.com/acme/legal-tools::legal.contracts.ContractClause"
}
```

Chains are sorted shortest-first. If no chain exists within `max_depth`, the `chains` array is empty.

### Compatibility Check

```
GET /v1/graph/compatibility?source={pipe_key}&target={pipe_key}
```

Checks whether the output of one pipe is type-compatible with an input of another.

**Query parameters:**

| Parameter | Required | Description |
|-----------|----------|-------------|
| `source` | Yes | Source pipe key in `package_address::pipe_code` format. |
| `target` | Yes | Target pipe key in `package_address::pipe_code` format. |

**Response:**

```json
{
  "compatible": true,
  "compatible_params": ["source"],
  "source_output": "ContractClause",
  "target_inputs": { "source": "Text", "context": "AnalysisContext" }
}
```

The `compatible_params` array lists the target pipe's input parameter names that can accept the source pipe's output. An empty array means the pipes are incompatible.

## Pagination

All list endpoints use offset-based pagination:

| Field | Description |
|-------|-------------|
| `offset` | Number of items to skip. Default: `0`. |
| `limit` | Maximum items per page. Default: `20`. Maximum: `100`. |
| `total` | Total number of matching items (returned in every response). |

A registry MUST return a `total` field in paginated responses. Clients SHOULD use `total` to determine whether more pages exist.

## Authentication

A registry MAY require authentication. When authentication is required:

- The registry MUST accept a Bearer token in the `Authorization` header.
- The registry MUST return `401 Unauthorized` for requests that require authentication but lack a valid token.
- The registry MUST return `403 Forbidden` for requests with a valid token that lacks the required scope.

```
Authorization: Bearer <token>
```

Token provisioning is outside the scope of this specification. Registries may use API keys, OAuth tokens, or any scheme that produces a Bearer token.

## Rate Limiting

A registry SHOULD enforce rate limits to ensure fair use. When rate-limited:

- The registry MUST return `429 Too Many Requests`.
- The registry SHOULD include a `Retry-After` header with the number of seconds to wait.

## Error Format

All error responses use a consistent JSON format:

```json
{
  "error": {
    "code": "not_found",
    "message": "Package 'github.com/acme/unknown' is not indexed by this registry."
  }
}
```

**Standard error codes:**

| HTTP Status | Error Code | Description |
|-------------|------------|-------------|
| `400` | `bad_request` | Malformed query parameters or missing required fields. |
| `401` | `unauthorized` | Missing or invalid authentication token. |
| `403` | `forbidden` | Valid token but insufficient permissions. |
| `404` | `not_found` | Package or resource not found in the index. |
| `422` | `invalid_concept` | Concept ID format is invalid or concept not found in the graph. |
| `429` | `rate_limited` | Too many requests. |
| `500` | `internal_error` | Unexpected server error. |

## Content Type

All responses use `Content-Type: application/json; charset=utf-8`. A registry MUST return JSON for all API endpoints. A registry MUST set the `Content-Type` header on every response.

## See Also

- [Registry Indexing](registry-indexing.md) — how registries crawl and index packages.
- [Registry Search](registry-search.md) — type-aware search semantics and graph query rules.
- [Registry Distribution Protocol](registry-distribution.md) — proxy chains, signed manifests, and multi-tier deployment.
- [Distribution](distribution.md) — the federated model that registries build upon.
- [The Know-How Graph](../know-how-graph/index.md) — the typed network that powers registry search.


## Registry Indexing

# Registry Indexing

A registry builds its index by crawling Git-hosted packages, parsing their manifests and bundles, and constructing a [Know-How Graph](../know-how-graph/index.md) from the extracted metadata. This page specifies the indexing pipeline.

## Indexing Pipeline

The indexing pipeline transforms a package address into a `PackageIndexEntry`:

```
address → git clone → parse METHODS.toml → scan .mthds files → PackageIndexEntry
```

### Step 1: Clone the Repository

The registry resolves the package address to a Git clone URL:

1. Prepend `https://`.
2. Append `.git` (if not already present).

```
github.com/acme/legal-tools → https://github.com/acme/legal-tools.git
```

The registry MUST use `git ls-remote --tags` to enumerate available version tags before cloning. Only tags that parse as valid [semantic versions](../packages/version-resolution.md) are considered. Both `v`-prefixed (e.g., `v1.0.0`) and bare (e.g., `1.0.0`) tags are recognized.

The registry clones at the latest stable version tag using `git clone --depth 1 --branch {tag}`.

### Step 2: Parse the Manifest

The registry reads `METHODS.toml` from the package root. This provides:

| Field | Type | Description |
|-------|------|-------------|
| `address` | `string` | Package address (e.g., `github.com/acme/legal-tools`). |
| `version` | `string` | Semantic version (e.g., `1.2.0`). |
| `description` | `string` | Human-readable package description. |
| `authors` | `list[string]` | Package authors. |
| `license` | `string \| null` | SPDX license identifier. |
| `dependencies` | `table[alias, PackageDependency]` | Declared dependencies with address, version constraint, and alias. |
| `exports` | `table[domain_path, DomainExports]` | Which pipes are publicly visible, grouped by domain path. |

If `METHODS.toml` is missing or fails validation, the registry MUST skip the package and log a warning. A malformed manifest MUST NOT cause the registry to stop indexing other packages.

### Step 3: Scan Bundles

The registry collects all `.mthds` files recursively from the package root. For each bundle file, it parses the MTHDS content and extracts:

**Domains:**

Each bundle declares a domain. The registry builds a `DomainEntry` for each unique domain encountered:

```json
{
  "domain_code": "legal.contracts",
  "description": "Contract processing domain"
}
```

**Concepts:**

Each concept definition produces a `ConceptEntry`:

```json
{
  "concept_code": "ContractClause",
  "domain_code": "legal.contracts",
  "concept_ref": "legal.contracts.ContractClause",
  "description": "A single clause extracted from a contract",
  "refines": "native.Text",
  "structure_fields": ["clause_type", "text", "section_number"]
}
```

The `concept_ref` is always `{domain_code}.{concept_code}`. The `refines` field is the raw string from the bundle — it is resolved to a cross-package identity during [graph construction](#know-how-graph-construction).

**Pipes:**

Each pipe definition produces a `PipeSignature`:

```json
{
  "pipe_code": "extract_clause",
  "pipe_type": "PipeLLM",
  "domain_code": "legal.contracts",
  "description": "Extract a specific clause from a contract document",
  "input_specs": { "source": "ContractDocument" },
  "output_spec": "ContractClause",
  "is_exported": true
}
```

The `is_exported` flag is determined by the manifest's `[exports]` section:

- If the manifest declares exports for the pipe's domain and the pipe code appears in the exports list, `is_exported` is `true`.
- If no exports are declared at all (no manifest), all pipes are considered exported.

### Step 4: Assemble the Index Entry

The registry assembles a `PackageIndexEntry` from the parsed manifest and scanned bundles:

```json
{
  "address": "github.com/acme/legal-tools",
  "version": "1.2.0",
  "description": "Contract analysis and clause extraction methods",
  "authors": ["Acme Legal Team"],
  "license": "Apache-2.0",
  "domains": [
    { "domain_code": "legal.contracts", "description": "Contract processing domain" }
  ],
  "concepts": [ ... ],
  "pipes": [ ... ],
  "dependencies": ["github.com/mthds/document-processing"],
  "dependency_aliases": { "doc_processing": "github.com/mthds/document-processing" }
}
```

The `dependencies` list contains raw addresses. The `dependency_aliases` map aliases to addresses, enabling [cross-package concept resolution](registry-search.md#cross-package-concept-resolution) during graph construction.

Domains are sorted alphabetically by `domain_code`. Parse errors in individual bundles are logged as warnings — a single broken bundle MUST NOT prevent the rest of the package from being indexed.

## The Package Index

The `PackageIndex` is the collection of all `PackageIndexEntry` records, keyed by address:

```json
{
  "entries": {
    "github.com/acme/legal-tools": { ... },
    "github.com/mthds/document-processing": { ... }
  }
}
```

Operations on the index:

| Operation | Description |
|-----------|-------------|
| `add_entry` | Add or replace a package entry by address. |
| `get_entry` | Retrieve an entry by address. Returns null if not indexed. |
| `remove_entry` | Remove an entry by address. Returns whether the entry existed. |
| `all_concepts` | Return all concepts across all packages as `(address, ConceptEntry)` pairs. |
| `all_pipes` | Return all pipes across all packages as `(address, PipeSignature)` pairs. |

## Know-How Graph Construction

After building the package index, the registry enables the Know-How Graph — a directed graph of concepts and pipes that enables [type-aware search](registry-search.md). The construction follows these steps:

### Step 1: Build Concept Nodes

For every concept in every indexed package, the registry creates a `ConceptNode` with a globally unique `ConceptId`:

```json
{
  "concept_id": {
    "package_address": "github.com/acme/legal-tools",
    "concept_ref": "legal.contracts.ContractClause"
  },
  "description": "A single clause extracted from a contract",
  "refines": null,
  "structure_fields": ["clause_type", "text", "section_number"]
}
```

The node key is `{package_address}::{concept_ref}` (e.g., `github.com/acme/legal-tools::legal.contracts.ContractClause`).

The registry MUST also create concept nodes for all native concepts (`Dynamic`, `Text`, `Image`, `Document`, `Html`, `TextAndImages`, `Number`, `Page`, `JSON`, `SearchResult`, `Anything`). Native concepts use the package address `__native__` and concept references prefixed with `native.` (e.g., `native.Text`).

### Step 2: Resolve Refinement Targets

For each concept with a `refines` string, the registry resolves it to a `ConceptId`:

- **Local reference** (e.g., `ContractClause` or `legal.contracts.ContractClause`): resolved within the same package by matching against concept refs or bare concept codes.
- **Cross-package reference** (e.g., `dep_alias->domain.ConceptCode`): the alias is looked up in the package's `dependency_aliases` map to find the target package address, then the concept is resolved in the target package.

If the `refines` target cannot be resolved (unknown alias, missing concept), the registry MUST log a warning and leave the `refines` field as `null`. Unresolvable refinement targets MUST NOT prevent the concept from appearing in the graph.

### Step 3: Build Pipe Nodes

For every pipe in the index, the registry creates a `PipeNode` with resolved concept identities for all inputs and the output:

```json
{
  "package_address": "github.com/acme/legal-tools",
  "pipe_code": "extract_clause",
  "pipe_type": "PipeLLM",
  "domain_code": "legal.contracts",
  "description": "Extract a specific clause from a contract document",
  "is_exported": true,
  "input_concept_ids": {
    "source": {
      "package_address": "github.com/acme/legal-tools",
      "concept_ref": "legal.contracts.ContractDocument"
    }
  },
  "output_concept_id": {
    "package_address": "github.com/acme/legal-tools",
    "concept_ref": "legal.contracts.ContractClause"
  }
}
```

Concept resolution for pipe inputs and outputs follows the same rules as refinement resolution: native concepts, local references, domain-qualified references, and cross-package references are all supported.

If a pipe's output concept or any input concept cannot be resolved, the registry MUST exclude the entire pipe from the graph. Pipes with unresolvable concepts MUST NOT create dangling references.

### Step 4: Build Refinement Edges

For each concept node whose `refines` field is non-null, the registry creates a `REFINEMENT` edge:

```json
{
  "kind": "refinement",
  "source_concept_id": {
    "package_address": "github.com/acme/legal-tools",
    "concept_ref": "legal.contracts.NonCompeteClause"
  },
  "target_concept_id": {
    "package_address": "github.com/acme/legal-tools",
    "concept_ref": "legal.contracts.ContractClause"
  }
}
```

The source is the more specific concept; the target is the more general concept it refines.

### Step 5: Build Data Flow Edges

A data flow edge connects two pipes when the output of one can satisfy an input of the other. Compatibility is determined by the refinement hierarchy:

> A pipe's output concept is compatible with another pipe's input concept if the output concept is exactly the input concept, OR the output concept is a refinement (descendant) of the input concept.

The registry walks up the refinement chain from each pipe's output concept, collecting all ancestor node keys (cycle-safe). For each pipe's input, it looks up compatible producers from this reverse index.

```json
{
  "kind": "data_flow",
  "source_pipe_key": "github.com/acme/legal-tools::extract_pages",
  "target_pipe_key": "github.com/acme/legal-tools::extract_clause",
  "input_param": "source"
}
```

Self-loops (a pipe feeding into itself) are excluded.

## Index Refresh

A registry MUST support at least one mechanism for keeping the index current:

- **Manual trigger** — an API call or administrative action that re-indexes a specific package address.
- **Polling** — periodic re-crawl of known package addresses, comparing the latest version tag against the indexed version.
- **Webhook** — a Git hosting webhook (e.g., GitHub push event) that triggers re-indexing when a new tag is pushed.

A registry SHOULD expose the index freshness for each package (e.g., `indexed_at` timestamp) so that clients can assess staleness.

## Error Handling

Indexing errors are non-fatal at the individual package and bundle level:

| Error | Behavior |
|-------|----------|
| `METHODS.toml` missing or invalid | Skip the package. Log a warning. |
| Individual `.mthds` file fails to parse | Skip the bundle. Index remaining bundles. Log a warning. |
| Concept `refines` target unresolvable | Set `refines` to null. Log a warning. |
| Pipe input/output concept unresolvable | Exclude the pipe from the graph. Log a warning. |
| Git clone fails | Skip the package. Log a warning. |

A registry MUST NOT stop its indexing run because of errors in individual packages.

## See Also

- [The Registry](registry.md) — API endpoints for querying the index.
- [Registry Search](registry-search.md) — how the index and graph power type-aware queries.
- [The Know-How Graph](../know-how-graph/index.md) — conceptual overview of the typed network.
- [The Manifest](manifest.md) — the `METHODS.toml` fields that the registry parses.


## Registry Search

# Registry Search

The registry exposes two search modes: **text search** (substring matching on names and descriptions) and **type-compatible search** (signature-based queries that understand the concept refinement hierarchy). This page specifies the semantics of type-compatible search and graph queries.

## Concept Compatibility

Type-compatible search is built on a single rule:

> An output concept is **compatible** with an input concept if the output concept is exactly the input concept, OR the output concept is a refinement (descendant) of the input concept.

Compatibility is resolved by walking up the refinement chain from the output concept. If any ancestor in the chain matches the input concept, the concepts are compatible.

### Example

Given this refinement chain:

```
NonCompeteClause → ContractClause → Text
```

- `NonCompeteClause` is compatible with `Text` (descendant).
- `NonCompeteClause` is compatible with `ContractClause` (direct child).
- `NonCompeteClause` is compatible with `NonCompeteClause` (identity).
- `Text` is NOT compatible with `NonCompeteClause` (ancestor, not descendant).

The walk is cycle-safe: if a refinement chain contains a cycle (which violates the specification but can occur in malformed data), the walk terminates when a previously visited node is encountered.

## Query Types

### "What can I do with X?"

Given a concept, find all pipes that accept it as input.

A pipe accepts the concept if **any** of its input parameters expects the exact concept or an ancestor concept (i.e., the given concept is compatible with the input expectation via the refinement chain).

**API:**

```
GET /v1/search/typed?accepts=Document
```

**Semantics:**

For each pipe in the graph, for each input parameter:

1. Resolve the `accepts` parameter to a `ConceptId`.
2. Check if the given concept is compatible with the input concept (walk up from the given concept).
3. If any input parameter matches, include the pipe in results.

Each pipe appears at most once in the results, even if multiple input parameters match.

### "What produces Y?"

Given a concept, find all pipes that produce it.

A pipe produces the concept if its output is the exact concept or a refinement (descendant) of the requested concept.

**API:**

```
GET /v1/search/typed?produces=ContractClause
```

**Semantics:**

For each pipe in the graph:

1. Resolve the `produces` parameter to a `ConceptId`.
2. Check if the pipe's output concept is compatible with the requested concept (walk up from the pipe's output).
3. If compatible, include the pipe in results.

### Combined: Accepts and Produces

When both `accepts` and `produces` are specified, the registry returns pipes that satisfy both conditions simultaneously.

**API:**

```
GET /v1/search/typed?accepts=Document&produces=ContractClause
```

### "I have X, I need Y" — Chain Discovery

When no single pipe transforms concept X into concept Y, the registry searches for multi-step pipe chains using breadth-first search (BFS).

**API:**

```
GET /v1/graph/chains?from=__native__::native.Document&to=github.com/acme/legal-tools::legal.contracts.ContractClause&max_depth=3
```

**Algorithm:**

1. Find all starter pipes — those that accept the `from` concept (using "What can I do with X?" logic).
2. Initialize a BFS queue with each starter pipe as a single-step chain.
3. For each chain in the queue:
    - If the last pipe's output is compatible with the `to` concept, record the chain as a result.
    - Otherwise, if the chain has not reached `max_depth`, find all pipes that accept the last pipe's output and extend the chain.
4. Visited pipe keys are tracked per chain to prevent cycles.
5. Results are sorted shortest-first.

The `max_depth` parameter limits the maximum number of pipes in a single chain. The default is `3`.

**Example result:**

A query from `native.Document` to `legal.contracts.ContractClause` might discover:

```
Chain 1 (2 steps):
  extract_pages → extract_clause

Chain 2 (3 steps):
  extract_pages → analyze_content → extract_clause
```

### Compatibility Check

Given two pipe keys, determine whether the output of the first pipe can satisfy any input of the second.

**API:**

```
GET /v1/graph/compatibility?source=pkg::extract_pages&target=pkg::extract_clause
```

**Semantics:**

1. Look up both pipe nodes in the graph.
2. For each input parameter of the target pipe, check if the source pipe's output concept is compatible with the input concept.
3. Return the list of compatible parameter names.

An empty list means the pipes are incompatible.

## Cross-Package Concept Resolution

Type-compatible search works across package boundaries. When a concept in package A refines a concept in package B, the refinement chain spans both packages:

```
Package A:  EmploymentNDA → (refines) → Package B: NonDisclosureAgreement → (refines) → Text
```

Cross-package references are resolved during [graph construction](registry-indexing.md#step-2-resolve-refinement-targets) using the `dependency_aliases` map from `METHODS.toml`:

1. The `refines` string `acme_legal->legal.contracts.NonDisclosureAgreement` is split into alias (`acme_legal`) and remainder (`legal.contracts.NonDisclosureAgreement`).
2. The alias is resolved to a package address via the declaring package's `dependency_aliases`.
3. The concept is looked up in the target package by `concept_ref` or by bare concept code.

This resolution is transitive — a chain can span any number of packages as long as each link has a declared dependency with a valid alias.

## Refinement Chain Resolution

The registry exposes refinement chain information to help clients understand concept hierarchies. Given a concept, the registry walks up through `refines` links and returns the full chain:

```
[EmploymentNDA, NonDisclosureAgreement, ContractClause, Text]
```

The chain starts at the given concept and ends at the root (a concept with no `refines` link, or a native concept). The walk is cycle-safe.

## Concept Identification

Concepts in the graph are identified by a `ConceptId` with two components:

| Field | Description | Example |
|-------|-------------|---------|
| `package_address` | The package that defines the concept. `__native__` for native concepts. | `github.com/acme/legal-tools` |
| `concept_ref` | Domain-qualified concept reference. | `legal.contracts.ContractClause` |

The full node key is `{package_address}::{concept_ref}`.

When search queries use bare concept codes (e.g., `Document` rather than `__native__::native.Document`), the registry SHOULD resolve the code by:

1. Checking native concepts first.
2. Falling back to a unique match across all indexed packages.
3. Returning an error if the code is ambiguous (multiple packages define the same bare code).

## See Also

- [The Registry](registry.md) — API endpoint reference and request/response schemas.
- [Registry Indexing](registry-indexing.md) — how the graph is constructed from package data.
- [The Know-How Graph](../know-how-graph/index.md) — conceptual overview of typed discovery.
- [Concepts](../language/concepts.md) — how concepts define typed data and refinement.


## Registry Distribution Protocol

# Registry Distribution Protocol

This page specifies how registries participate in the package distribution chain — acting as proxies, verifying package integrity, surfacing community signals, and supporting multi-tier deployment from local development to community-wide distribution.

## Proxy Chain

A client can be configured with an ordered list of registry URLs, analogous to Go's `GOPROXY` protocol. When resolving a package, the client queries registries in order:

```
MTHDS_REGISTRY=https://registry.example.com,https://community.mthds.ai,direct
```

| Entry | Behavior |
|-------|----------|
| A registry URL | Query the registry's `/v1/packages/{address}` endpoint. If the package is found, use it. If not (404), try the next entry. |
| `direct` | Bypass registries and fetch directly from the Git repository at the package address. |

The special value `direct` as the final entry ensures that packages not indexed by any registry can still be fetched from source. If `direct` is not present and no registry returns the package, resolution fails.

### Proxy Mode

A registry MAY operate in proxy mode, where it does not maintain its own index but forwards requests to an upstream registry and caches the results. A proxy registry:

- MUST forward the original request path and query parameters.
- MUST cache successful responses for a configurable duration.
- MUST forward `404 Not Found` without caching (the upstream may index the package later).
- SHOULD pass through the upstream's `Retry-After` headers on `429` responses.

### Mirror Mode

A registry MAY operate in mirror mode, where it maintains a full copy of another registry's index. A mirror registry:

- MUST periodically synchronize with the upstream registry.
- MUST serve requests from its local copy without forwarding.
- SHOULD expose a `last_synced_at` timestamp so clients can assess freshness.

## Signed Manifests

To ensure package integrity, a registry MAY require or serve **signed manifests**. A signed manifest binds the `METHODS.toml` content to a cryptographic signature.

### Signature Format

A signature is a detached JSON object:

```json
{
  "manifest_sha256": "a1b2c3d4e5f6...",
  "address": "github.com/acme/legal-tools",
  "version": "1.2.0",
  "signed_at": "2026-01-15T10:30:00Z",
  "signer": "acme-ci-bot",
  "algorithm": "ed25519",
  "public_key_id": "key-2026-01",
  "signature": "base64-encoded-signature..."
}
```

| Field | Description |
|-------|-------------|
| `manifest_sha256` | SHA-256 hash of the raw `METHODS.toml` file content. |
| `address` | Package address, matching the manifest. |
| `version` | Package version, matching the manifest. |
| `signed_at` | ISO 8601 timestamp of when the signature was created. |
| `signer` | Identifier of the signing entity (human or automation). |
| `algorithm` | Signature algorithm. MUST be `ed25519`. |
| `public_key_id` | Identifier for the public key used, for key rotation. |
| `signature` | Base64-encoded Ed25519 signature over the canonical representation of the preceding fields. |

### Verification

When verifying a signed manifest:

1. Compute the SHA-256 hash of the `METHODS.toml` file content.
2. Compare it against `manifest_sha256`.
3. Reconstruct the canonical signing payload (all fields except `signature`, serialized as sorted-key JSON with no whitespace).
4. Verify the Ed25519 signature against the payload using the public key identified by `public_key_id`.

A client SHOULD verify signatures when available. A client MUST NOT treat an unsigned package as verified.

### Trust Store

Public keys are stored in a trust store. A compliant runtime SHOULD support trust stores at two levels:

| Level | Location | Purpose |
|-------|----------|---------|
| **System** | `~/.mthds/trust/` | Keys trusted for all projects. |
| **Project** | `.mthds/trust/` in the project root | Keys trusted for this project only. |

Each key file is named `{public_key_id}.pub` and contains the raw Ed25519 public key in base64 encoding.

## Social Signals

A registry MAY track and expose social signals to help users evaluate packages:

| Signal | Description |
|--------|-------------|
| `install_count` | Number of times the package has been fetched through this registry. |
| `star_count` | Number of users who have starred the package. |
| `endorsed_by` | List of known organizations or users who endorse the package. |
| `last_updated` | Timestamp of the latest indexed version. |
| `indexed_at` | Timestamp of when the registry last crawled the package. |

Social signals are informational. They MUST NOT affect search ranking in type-compatible queries (which are purely type-driven). A registry MAY use social signals to influence text search ranking.

### Social Signals Endpoint

```
GET /v1/packages/{address}/signals
```

**Response:**

```json
{
  "address": "github.com/acme/legal-tools",
  "install_count": 1247,
  "star_count": 42,
  "endorsed_by": ["mthds-foundation"],
  "last_updated": "2026-01-15T10:30:00Z",
  "indexed_at": "2026-02-01T08:00:00Z"
}
```

## Multi-Tier Deployment

Registries support the same deployment tiers described in [Distribution](distribution.md):

### Local Tier

No registry involved. The CLI operates on the current project and its local cache (`~/.mthds/packages/`). Search and graph queries run against a locally-built index.

### Project Tier

A project team runs an internal registry indexing their shared packages. The registry URL is configured in the project's `.mthds/config.toml`:

```toml
[registry]
urls = ["https://registry.internal.acme.com"]
```

### Organization Tier

An organization runs a registry that acts as a proxy to the community registry, adding organization-specific packages and governance policies:

```
MTHDS_REGISTRY=https://registry.internal.acme.com,https://community.mthds.ai,direct
```

The internal registry can:

- Index private packages not available on the community registry.
- Enforce approval policies before indexing external packages.
- Cache community packages for air-gapped environments.

### Community Tier

A public registry indexes open-source packages from public Git repositories. Anyone can notify the registry of a new package address. The registry crawls and indexes it.

## Configuration

Registry URLs are resolved in this order of precedence:

1. **Environment variable**: `MTHDS_REGISTRY` (comma-separated list).
2. **Project config**: `.mthds/config.toml` `[registry].urls` array.
3. **User config**: `~/.mthds/config.toml` `[registry].urls` array.
4. **Default**: `direct` (no registry, fetch from Git directly).

## See Also

- [The Registry](registry.md) — API endpoints and schemas.
- [Registry Indexing](registry-indexing.md) — how registries build the index.
- [Distribution](distribution.md) — the federated storage model that registries build upon.
- [Version Resolution](version-resolution.md) — how version constraints are resolved.


# Reference

## .mthds File Format

# .mthds File Format

The `.mthds` file is a TOML document that defines typed data (concepts) and typed transformations (pipes) within a single domain. This page is the normative reference for every field, validation rule, and structural constraint of the format.

## File Encoding and Syntax

A `.mthds` file MUST be a valid TOML document encoded in UTF-8. The file extension MUST be `.mthds`. Parsers MUST reject files that are not valid TOML before any MTHDS-specific validation occurs.

## Top-Level Structure

A `.mthds` file is called a **bundle**. It consists of:

1. **Header fields** — top-level key-value pairs that identify the bundle.
2. **Concept definitions** — a `[concept]` table and/or `[concept.<ConceptCode>]` sub-tables.
3. **Pipe definitions** — `[pipe.<pipe_code>]` sub-tables.

All three sections are optional in the TOML sense (an empty `.mthds` file is valid TOML), but a useful bundle will contain at least one concept or one pipe.

## Header Fields

Header fields appear at the top level of the TOML document, before any `[concept]` or `[pipe]` tables.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `domain` | string | Yes | The domain this bundle belongs to. Determines the namespace for all concepts and pipes defined in this file. |
| `description` | string | No | A human-readable description of what this bundle provides. |
| `system_prompt` | string | No | A default system prompt applied to all `PipeLLM` pipes in this bundle that do not define their own `system_prompt`. |
| `main_pipe` | string | No | The pipe code of the bundle's primary entry point. If set, this pipe is auto-exported when the bundle is part of a package. |

**Validation rules:**

- `domain` MUST be a valid domain code (see [Domain Naming Rules](#domain-naming-rules)).
- `main_pipe`, if present, MUST be a valid pipe code (`snake_case`) and MUST reference a pipe defined in this bundle.

**Example:**

```toml
domain      = "legal.contracts"
description = "Contract analysis methods for legal documents"
main_pipe   = "extract_clause"
```

## Domain Naming Rules

Domain codes define the namespace for all concepts and pipes in a bundle.

**Syntax:**

- A domain code is one or more `snake_case` segments separated by `.` (dot).
- Each segment MUST match the pattern `[a-z][a-z0-9_]*`.
- Domains MAY be hierarchical: `legal`, `legal.contracts`, `legal.contracts.shareholder`.

**Reserved domains:**

The following domain names are reserved and MUST NOT be used as the first segment of any user-defined domain:

- `native` — built-in concept types
- `mthds` — reserved for the MTHDS standard
- `pipelex` — reserved for the reference implementation

A compliant implementation MUST reject bundles that declare a domain starting with a reserved segment (e.g., `native.custom` is invalid).

**Recommendations:**

- Depth SHOULD be 1–3 levels.
- Each segment SHOULD be 1–4 words.

## Concept Definitions

Concepts are typed data declarations. They define the vocabulary of a domain — the kinds of data that pipes accept and produce.

### Simple Concept Declarations

The simplest form of concept declaration uses a flat `[concept]` table where each key is a concept code and the value is a description string:

```toml
[concept]
ContractClause = "A clause extracted from a legal contract"
UserProfile    = "A user's profile information"
```

This form declares concepts with no structure and no refinement. They exist as named types.

### Structured Concept Declarations

A concept with fields uses a `[concept.<ConceptCode>]` sub-table:

```toml
[concept.LineItem]
description = "A single line item in an invoice"

[concept.LineItem.structure]
product_name = { type = "text", description = "Name of the product", required = true }
quantity     = { type = "integer", description = "Quantity ordered", required = true }
unit_price   = { type = "number", description = "Price per unit", required = true }
```

Both forms MAY coexist in the same bundle. A bundle MAY mix simple declarations in `[concept]` with structured declarations as `[concept.<Code>]` sub-tables.

### Concept Blueprint Fields

When using the structured form `[concept.<ConceptCode>]`, the following fields are available:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `description` | string | Yes | Human-readable description of the concept. |
| `structure` | table or string | No | Field definitions for the concept. If a string, it is a shorthand description (equivalent to a simple declaration). If a table, each key is a field name mapped to a field blueprint. |
| `refines` | string | No | A concept reference indicating that this concept is a specialization of another concept. |

**Validation rules:**

- `refines` and `structure` MUST NOT both be present on the same concept. A concept either refines another concept or defines its own structure, not both.
- `refines`, if present, MUST be a valid concept reference: either a bare concept code (`PascalCase`) or a domain-qualified reference (`domain.ConceptCode`). Cross-package references (`alias->domain.ConceptCode`) are also valid.
- Concept codes MUST be `PascalCase`, matching the pattern `[A-Z][a-zA-Z0-9]*`.
- Concept codes MUST NOT collide with native concept codes (see [Native Concepts](#native-concepts)).

### Concept Refinement

Refinement establishes a specialization relationship between concepts. A concept that refines another inherits its semantic meaning and can be used anywhere the parent concept is expected.

```toml
[concept.NonCompeteClause]
description = "A non-compete clause in an employment contract"
refines     = "ContractClause"
```

The `refines` field accepts:

- A bare concept code: `"ContractClause"` — resolved within the current bundle's domain.
- A domain-qualified reference: `"legal.ContractClause"` — resolved within the current package.
- A cross-package reference: `"acme_legal->legal.contracts.NonDisclosureAgreement"` — resolved from a dependency.

### Concept Structure Fields

When `structure` is a table, each key is a field name and each value is a field blueprint. Field names MUST NOT start with an underscore (`_`), as these are reserved for internal use. Field names MUST NOT collide with reserved field names (Pydantic model attributes and internal metadata fields).

#### Field Blueprint

Each field in a concept structure is defined by a field blueprint:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `description` | string | Yes | Human-readable description of the field. |
| `type` | string | Conditional | The field type. Required unless `choices` is provided. |
| `required` | boolean | No | Whether the field is required. Default: `false`. |
| `default_value` | any | No | Default value for the field. Must match the declared type. |
| `choices` | array of strings | No | Fixed set of allowed string values. When `choices` is set, `type` MUST be omitted (the type is implicitly an enum of the given choices). |
| `key_type` | string | Conditional | Key type for `dict` fields. Required when `type = "dict"`. |
| `value_type` | string | Conditional | Value type for `dict` fields. Required when `type = "dict"`. |
| `item_type` | string | No | Item type for `list` fields. When set to `"concept"`, `item_concept_ref` is required. |
| `concept_ref` | string | Conditional | Concept reference for `concept`-typed fields. Required when `type = "concept"`. |
| `item_concept_ref` | string | Conditional | Concept reference for list items when `item_type = "concept"`. |

#### Field Types

The `type` field accepts the following values:

| Type | Description | `default_value` type |
|------|-------------|---------------------|
| `text` | A string value. | `string` |
| `integer` | A whole number. | `integer` |
| `number` | A numeric value (integer or floating-point). | `integer` or `float` |
| `boolean` | A true/false value. | `boolean` |
| `date` | A date value. | `datetime` |
| `list` | An ordered collection. Use `item_type` to specify element type. | `array` |
| `dict` | A key-value mapping. Requires `key_type` and `value_type`. | `table` |
| `concept` | A reference to another concept. Requires `concept_ref`. Cannot have `default_value`. | *(not allowed)* |

When `type` is omitted and `choices` is provided, the field is an enumeration field. The value MUST be one of the strings in the `choices` array.

**Validation rules for field types:**

- `type = "dict"`: `key_type` and `value_type` MUST both be non-empty.
- `type = "concept"`: `concept_ref` MUST be set. `default_value` MUST NOT be set.
- `type = "list"` with `item_type = "concept"`: `item_concept_ref` MUST be set.
- `item_concept_ref` MUST NOT be set unless `item_type = "concept"`.
- `concept_ref` MUST NOT be set unless `type = "concept"`.
- If `choices` is provided and `type` is omitted, `default_value` (if present) MUST be one of the values in `choices`.
- If both `type` and `default_value` are set, the runtime type of `default_value` MUST match the declared `type`.

**Example — concept with all field types:**

```toml
[concept.CandidateProfile]
description = "A candidate's profile for job matching"

[concept.CandidateProfile.structure]
full_name        = { type = "text", description = "Full name", required = true }
years_experience = { type = "integer", description = "Years of professional experience" }
gpa              = { type = "number", description = "Grade point average" }
is_active        = { type = "boolean", description = "Whether actively looking", default_value = true }
graduation_date  = { type = "date", description = "Date of graduation" }
skills           = { type = "list", item_type = "text", description = "List of skills" }
metadata         = { type = "dict", key_type = "text", value_type = "text", description = "Additional metadata" }
seniority_level  = { description = "Seniority level", choices = ["junior", "mid", "senior", "lead"] }
address          = { type = "concept", concept_ref = "Address", description = "Home address" }
references       = { type = "list", item_type = "concept", item_concept_ref = "ContactInfo", description = "Professional references" }
```

## Native Concepts

Native concepts are built-in types that are always available in every bundle without declaration. They belong to the reserved `native` domain.

| Code | Qualified Reference | Description |
|------|-------------------|-------------|
| `Dynamic` | `native.Dynamic` | A dynamically-typed value. |
| `Text` | `native.Text` | A text string. |
| `Image` | `native.Image` | An image (binary). |
| `Document` | `native.Document` | A document (e.g., PDF, web page). |
| `Html` | `native.Html` | HTML content. |
| `TextAndImages` | `native.TextAndImages` | Combined text and image content. |
| `Number` | `native.Number` | A numeric value. |
| `Page` | `native.Page` | A single page extracted from a document. |
| `JSON` | `native.JSON` | A JSON value. |
| `SearchResult` | `native.SearchResult` | A web search result with answer and sources. |
| `Anything` | `native.Anything` | Accepts any type. |

Native concepts MAY be referenced by bare code (`Text`, `Image`) or by qualified reference (`native.Text`, `native.Image`). Bare native concept codes always take priority during resolution.

A bundle MUST NOT declare a concept with the same code as a native concept. A compliant implementation MUST reject such declarations.

## Pipe Definitions

Pipes are typed transformations. Each pipe has a typed signature: it declares what concepts it accepts as input and what concept it produces as output.

### Common Pipe Fields

All pipe types share these base fields:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | string | Yes | The pipe type. Determines which category and additional fields are available. |
| `description` | string | Yes | Human-readable description of what this pipe does. |
| `inputs` | table | No | Input declarations. Keys are input names (`snake_case`), values are concept references with optional multiplicity. |
| `output` | string | Yes | The output concept reference with optional multiplicity. |

**Pipe codes:**

- Pipe codes are the keys in `[pipe.<pipe_code>]` tables.
- Pipe codes MUST be `snake_case`, matching the pattern `[a-z][a-z0-9_]*`.

**Input names:**

- Input names MUST be `snake_case`.
- Dotted input names are allowed for nested field access (e.g., `my_input.field_name`), where each segment MUST be `snake_case`.

**Concept references in inputs and output:**

Concept references in `inputs` and `output` support an optional multiplicity suffix:

| Syntax | Meaning |
|--------|---------|
| `ConceptName` | A single instance. |
| `ConceptName[]` | A variable-length list (runtime determines count). |
| `ConceptName[N]` | A fixed-length list of exactly N items (N ≥ 1). |

Concept references MAY be bare codes (`Text`), domain-qualified (`legal.ContractClause`), or cross-package qualified (`alias->domain.ConceptCode`).

**Example:**

```toml
[pipe.analyze_contract]
type        = "PipeLLM"
description = "Analyze a legal contract and extract key clauses"
output      = "ContractClause[5]"

[pipe.analyze_contract.inputs]
contract_text = "Text"
```

### Pipe Types

MTHDS defines pipe types in two categories:

**Operators** — pipes that perform a single transformation:

| Type | Value | Description |
|------|-------|-------------|
| PipeLLM | `"PipeLLM"` | Generates output using a large language model. |
| PipeStructure | `"PipeStructure"` | Turns text into a structured concept using a large language model. |
| PipeFunc | `"PipeFunc"` | Calls a registered Python function. |
| PipeImgGen | `"PipeImgGen"` | Generates images using an image generation model. |
| PipeExtract | `"PipeExtract"` | Extracts structured content from documents. |
| PipeSearch | `"PipeSearch"` | Searches the web and returns structured results. |
| PipeCompose | `"PipeCompose"` | Composes output from templates or constructs. |

**Controllers** — pipes that orchestrate other pipes:

| Type | Value | Description |
|------|-------|-------------|
| PipeSequence | `"PipeSequence"` | Executes a series of pipes in order. |
| PipeParallel | `"PipeParallel"` | Executes pipes concurrently. |
| PipeCondition | `"PipeCondition"` | Routes execution based on a condition. |
| PipeBatch | `"PipeBatch"` | Maps a pipe over each item in a list. |

## Operator: PipeLLM

Generates output by invoking a large language model with a prompt.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeLLM"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | — |
| `prompt` | string | No | The LLM prompt template. Supports Jinja2 syntax and the `@variable` / `$variable` shorthand. |
| `system_prompt` | string | No | System prompt for the LLM. If omitted, the bundle-level `system_prompt` is used (if any). |
| `model` | string or table | No | Model identifier, model reference (see [Model References](../language/model-references.md)), or an inline [LLM settings](#inline-llm-settings) table. |
| `model_to_structure` | string or table | No | Model used for structuring the LLM output into the declared concept. Accepts the same forms as `model`. |
| `structuring_method` | string | No | Directive controlling how the output is structured. Values: `"direct"` (single LLM call producing JSON) or `"preliminary_text"` (the runtime first produces text, then structures it as a second step). The standard does not prescribe HOW the runtime implements `"preliminary_text"`. |

**Prompt template syntax:**

- `{{ variable_name }}` — standard Jinja2 variable substitution.
- `@variable_name` — shorthand, preprocessed to Jinja2 syntax.
- `$variable_name` — shorthand, preprocessed to Jinja2 syntax.
- Dotted paths are supported: `{{ doc_request.document_type }}`, `@doc_request.priority`.

**Validation rules:**

- Every variable referenced in `prompt` and `system_prompt` MUST correspond to a declared input (by root name). Internal variables starting with `_` and the special names `preliminary_text` and `place_holder` are excluded from this check.
- Every declared input MUST be referenced by at least one variable in `prompt` or `system_prompt`. Unused inputs are rejected.

**Example:**

```toml
[pipe.analyze_cv]
type = "PipeLLM"
description = "Analyze a CV to extract key professional information"
output = "CVAnalysis"
model = "$writing-factual"
system_prompt = """
You are an expert HR analyst specializing in CV evaluation.
"""
prompt = """
Analyze the following CV and extract the candidate's key professional information.

@cv_pages
"""

[pipe.analyze_cv.inputs]
cv_pages = "Page"
```

### Inline LLM Settings

When the `model` field is a table instead of a string, it defines inline model settings using the `LLMSetting` structure:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | The model handle (e.g., `"claude-4.5-sonnet"`). |
| `temperature` | number | Yes | Sampling temperature. Range: 0–1. |
| `max_tokens` | integer, `"auto"`, or null | No | Maximum tokens for the response. `"auto"` lets the model choose. |
| `image_detail` | string | No | Image detail level for vision inputs. Values: `high`, `low`, `auto`. |
| `prompting_target` | string | No | Target provider for prompt formatting. Values: `openai`, `anthropic`, `mistral`, `gemini`, `fal`. |
| `reasoning_effort` | string | No | Level of reasoning effort. Values: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`. `xhigh` sits between `high` and `max` and maps to provider-specific xhigh values where supported. |
| `reasoning_budget` | integer | No | Token budget for reasoning. Must be > 0. |
| `description` | string | No | Human-readable description of this model configuration. |

**Validation rules:**

- `reasoning_effort` and `reasoning_budget` MUST NOT both be set on the same inline LLM settings table.

**Example — inline LLM settings:**

```toml
[pipe.analyze_cv]
type = "PipeLLM"
description = "Analyze a CV"
output = "CVAnalysis"
prompt = "Analyze: @cv_pages"
model = { model = "claude-4.5-sonnet", temperature = 0.1, max_tokens = 4096 }

[pipe.analyze_cv.inputs]
cv_pages = "Page"
```

## Operator: PipeStructure

Turns text into a structured concept matching the declared output schema. The standard does not prescribe how a runtime achieves this; a typical implementation uses an LLM call.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeStructure"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | Yes | MUST contain exactly one entry. |
| `output` | string | Yes | The target structured concept reference, with optional multiplicity. MUST NOT be `Text` or a concept that refines `Text`. |
| `model` | string or table | No | Model identifier, model reference (see [Model References](../language/model-references.md)), or an inline [LLM settings](#inline-llm-settings) table. |

**Validation rules:**

- `inputs` MUST contain exactly one entry. The single input concept MUST be `Text` or a concept that refines `Text`.
- `output` MUST NOT be `Text` and MUST NOT be a concept that refines `Text`.
- `output` MAY use multiplicity (`Foo`, `Foo[]`, `Foo[N]`).
- `PipeStructure` MUST NOT accept image or document inputs. Use an upstream extraction step to produce text first.

**Example:**

```toml
[pipe.structure_review]
type        = "PipeStructure"
description = "Turn a free-form review into a RestaurantReview"
inputs      = { review_text = "Text" }
output      = "RestaurantReview"
```

**Example — with an explicit structuring model:**

```toml
[pipe.structure_review_premium]
type        = "PipeStructure"
description = "Use a stronger model for tricky structurings"
inputs      = { review_text = "Text" }
output      = "RestaurantReview"
model       = "@default-premium"
```

## Operator: PipeFunc

Calls a registered Python function.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeFunc"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | — |
| `function_name` | string | Yes | The fully-qualified name of the Python function to call. |

**Example:**

```toml
[pipe.capitalize_text]
type          = "PipeFunc"
description   = "Capitalize the input text"
inputs        = { text = "Text" }
output        = "Text"
function_name = "my_package.text_utils.capitalize"
```

## Operator: PipeImgGen

Generates images using an image generation model. The pipe carries a required `prompt` string template (and an optional `negative_prompt` template); it does not take a dedicated prompt concept as input. Declared `inputs` are injected into the `prompt` at runtime: `Text` inputs are interpolated into the prompt text, while `Image` inputs (a single image or a list) are referenced in the prompt and injected as reference images — each becomes an `[Image N]` token in the rendered text and is passed to the generator alongside it, enabling image-to-image, reference-image, and image-editing generation.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeImgGen"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | — |
| `prompt` | string | Yes | The image generation prompt template. Supports Jinja2 and `$variable` shorthand; declared inputs are injected into it. |
| `negative_prompt` | string | No | An optional prompt template describing what to avoid in the generated image. |
| `model` | string or table | No | Model identifier, model reference (see [Model References](../language/model-references.md)), or an inline [image generation settings](#inline-image-generation-settings) table. |
| `aspect_ratio` | string | No | Desired aspect ratio. Values: `square`, `landscape_4_3`, `landscape_3_2`, `landscape_16_9`, `landscape_21_9`, `portrait_3_4`, `portrait_2_3`, `portrait_9_16`, `portrait_9_21`. |
| `is_raw` | boolean | No | Whether to use raw mode (less post-processing). |
| `seed` | integer or `"auto"` | No | Random seed for reproducibility. `"auto"` lets the model choose. |
| `background` | string | No | Background setting. Values: `transparent`, `opaque`, `auto`. |
| `output_format` | string | No | Image output format. Values: `png`, `jpeg`, `webp`. |

**Validation rules:**

- Every variable referenced in `prompt` or `negative_prompt` MUST correspond to a declared input.
- `output` MUST resolve to an `Image`-compatible concept.
- Any input referenced as a reference image in the `prompt` or `negative_prompt` MUST resolve to an `Image`-compatible concept (single or list).

**Example:**

```toml
[pipe.generate_portrait]
type        = "PipeImgGen"
description = "Generate a portrait image from a description"
inputs      = { description = "Text" }
output      = "Image"
prompt      = "A professional portrait: $description"
model       = "$gen-image-testing"
```

**Example with a reference image (image-to-image):**

```toml
[pipe.restyle_photo]
type        = "PipeImgGen"
description = "Restyle a source photo following a textual instruction"
inputs      = { source = "Image", instruction = "Text" }
output      = "Image"
prompt      = "Restyle this image: $source. $instruction"
```

### Inline Image Generation Settings

When the `model` field is a table instead of a string, it defines inline model settings using the `ImgGenSetting` structure:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | The model handle. |
| `quality` | string | No | Image quality. Values: `low`, `medium`, `high`. |
| `nb_steps` | integer | No | Number of generation steps. Must be > 0. |
| `guidance_scale` | number | No | Guidance scale for generation. Must be > 0. |
| `is_moderated` | boolean | No | Whether to apply content moderation. Default: `false`. |
| `safety_tolerance` | integer | No | Safety tolerance level. Range: 1–6. |
| `description` | string | No | Human-readable description of this model configuration. |

**Validation rules:**

- `quality` and `nb_steps` MUST NOT both be set on the same inline image generation settings table.

**Example — inline image generation settings:**

```toml
[pipe.generate_portrait]
type        = "PipeImgGen"
description = "Generate a portrait image"
inputs      = { description = "Text" }
output      = "Image"
prompt       = "A professional portrait: $description"
aspect_ratio = "portrait_3_4"
model        = { model = "flux-pro", quality = "high" }
```

## Operator: PipeExtract

Extracts structured content from documents (e.g., PDF, web pages).

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeExtract"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | Yes | MUST contain exactly one input. |
| `output` | string | Yes | MUST be `"Page[]"`. |
| `model` | string or table | No | Model identifier, model reference (see [Model References](../language/model-references.md)), or an inline [extract settings](#inline-extract-settings) table. |
| `max_page_images` | integer | No | Maximum number of page images to process. |
| `page_image_captions` | boolean | No | Whether to generate captions for page images. |
| `page_views` | boolean | No | Whether to generate page views. |
| `page_views_dpi` | integer | No | DPI for page view rendering. |
| `render_js` | boolean | No | Web-page extraction only: render JavaScript before fetching the page content. Honored by backends that support headless rendering. Default: `false`. |
| `include_raw_html` | boolean | No | Web-page extraction only: include the fetched HTML in each extracted `Page`'s `raw_html` field. Default: `false`. |

**Validation rules:**

- `inputs` MUST contain exactly one entry. The input concept SHOULD be `Document` or a concept that refines `Document` or `Image`.
- `output` MUST be `"Page[]"` (a variable-length list of `Page`).
- When the document URL is a web page, PipeExtract fetches and extracts the page content. `render_js` and `include_raw_html` apply only to this case; backends that target local documents (PDFs, images) MAY ignore them.

**Example:**

```toml
[pipe.extract_cv]
type        = "PipeExtract"
description = "Extract text content from a CV PDF document"
inputs      = { cv_pdf = "Document" }
output      = "Page[]"
model       = "@default-text-from-pdf"
```

### Inline Extract Settings

When the `model` field is a table instead of a string, it defines inline model settings using the `ExtractSetting` structure:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | The model handle. |
| `max_nb_images` | integer | No | Maximum number of images to extract. Must be >= 0. |
| `image_min_size` | integer | No | Minimum image size in pixels. Must be >= 0. |
| `description` | string | No | Human-readable description of this model configuration. |

**Example — inline extract settings:**

```toml
[pipe.extract_cv]
type        = "PipeExtract"
description = "Extract text content from a CV PDF document"
inputs      = { cv_pdf = "Document" }
output      = "Page[]"
model       = { model = "gpt-4.1", max_nb_images = 10, image_min_size = 100 }
```

## Operator: PipeSearch

Searches the web using a search provider and returns structured results.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeSearch"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | MUST be `SearchResult` or a concept that refines `SearchResult`. |
| `prompt` | string | Yes | The search query template. Supports Jinja2 syntax and the `@variable` / `$variable` shorthand. |
| `model` | string or table | No | Model identifier, model reference (see [Model References](../language/model-references.md)), or an inline [search settings](#inline-search-settings) table. |
| `from_date` | string | No | Start date filter in ISO 8601 format (YYYY-MM-DD). Only return results from this date onwards. |
| `to_date` | string | No | End date filter in ISO 8601 format (YYYY-MM-DD). Only return results up to this date. |
| `include_domains` | array of strings | No | Restrict search to these domains only (e.g., `["reuters.com", "bbc.com"]`). |
| `exclude_domains` | array of strings | No | Exclude results from these domains. |

**Validation rules:**

- Every variable referenced in `prompt` MUST correspond to a declared input.
- `output` MUST be `SearchResult` or a concept that refines `SearchResult`.

**Example:**

```toml
[pipe.search_topic]
type        = "PipeSearch"
description = "Search the web for information about a topic"
inputs      = { topic = "Text" }
output      = "SearchResult"
model       = "$standard"
prompt      = "What is $topic?"
```

**Example — with date and domain filters:**

```toml
[pipe.search_recent_from_sources]
type            = "PipeSearch"
description     = "Search specific sources for recent news"
inputs          = { topic = "Text" }
output          = "SearchResult"
model           = "$standard"
prompt          = "What are the latest developments about $topic?"
from_date       = "2026-01-01"
include_domains = ["reuters.com", "apnews.com", "bbc.com"]
```

### Inline Search Settings

When the `model` field is a table instead of a string, it defines inline model settings using the `SearchSetting` structure:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | The model handle. |
| `include_images` | boolean | No | Whether to include images in results. Default: `false`. |
| `include_inline_citations` | boolean | No | Whether to include inline citations. Default: `true`. |
| `max_results` | integer or null | No | Maximum number of results. Must be ≥ 1. |
| `description` | string | No | Human-readable description of this model configuration. |

**Example — inline search settings:**

```toml
[pipe.deep_search]
type        = "PipeSearch"
description = "Deep research on a topic"
inputs      = { topic = "Text" }
output      = "SearchResult"
prompt      = "What are the main details about $topic?"
model       = { model = "linkup-deep", include_images = false }
```

## Operator: PipeCompose

Composes output by assembling data from working memory using either a template or a construct. Exactly one of `template` or `construct` MUST be provided.

### Template Mode

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeCompose"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | MUST be a single concept (no multiplicity). |
| `template` | string or table | Yes (if no `construct`) | A Jinja2 template string, or a template blueprint table with `template`, `category`, `templating_style`, and `extra_context` fields. |

When `template` is a string, it is a Jinja2 template rendered with the input variables. When `template` is a table, it MUST contain a `template` field (string) and a `category` field, and MAY contain `templating_style` and `extra_context`.

**Template shorthand syntax:**

MTHDS defines three shorthand patterns that a compliant preprocessor MUST expand before Jinja2 rendering:

| Pattern | Expansion | Description |
|---------|-----------|-------------|
| `$name` | `{{ name|format() }}` | Inline substitution with formatting. |
| `@name` | `{{ name|tag("name") }}` | Block insertion with tagging. |
| `@?name` | `{% if name %}{{ name|tag("name") }}{% endif %}` | Conditional block insertion (renders only if truthy). |

**Rules:**

- A shorthand pattern MUST NOT match when the character immediately following `$`, `@`, or `@?` is a digit (`0`–`9`). This prevents dollar amounts (e.g., `$100`) and version-like strings (e.g., `@2.0`) from being treated as variables.
- Dotted paths are supported: `$user.name`, `@doc.summary`, `@?extra.notes`. Each segment of the dotted path MUST be a valid identifier.
- When a matched name ends with a `.` (dot), the preprocessor MUST strip the trailing dot from the variable name and place it after the expanded expression (treating it as sentence punctuation).
- Raw Jinja2 syntax (`{{ }}`, `{% %}`) MUST always be accepted alongside the shorthands.

These shorthands apply to the `template` field of PipeCompose, the `prompt` and `system_prompt` fields of PipeLLM, the `prompt` and `negative_prompt` fields of PipeImgGen, and the `prompt` field of PipeSearch. See [Pipes — Operators: Template Mode](../language/pipes-operators.md#template-mode) for the full reference on categories and filters.

**Template blueprint fields (table form):**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `template` | string | Yes | The Jinja2 template string. |
| `category` | string | Yes | Template category. Values: `basic`, `expression`, `html`, `markdown`, `mermaid`, `llm_prompt`, `img_gen_prompt`. |
| `templating_style` | table | No | Rendering style configuration. See [Templating Style](#templating-style) below. |
| `extra_context` | table | No | Additional context variables for template rendering. |

#### Templating Style

The `templating_style` field controls how template output is formatted, particularly useful for templates that produce prompts for different LLM providers.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `tag_style` | string | Yes | How variables are tagged in output. Values: `no_tag`, `ticks`, `xml`, `square_brackets`. |
| `text_format` | string | No | Output text format. Values: `plain`, `markdown`, `html`, `json`. Default: `plain`. |

**Validation rules (template mode):**

- Every variable referenced in the template MUST correspond to a declared input.
- `output` MUST NOT use multiplicity brackets (`[]` or `[N]`).

### Construct Mode

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeCompose"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | MUST be a single concept (no multiplicity). |
| `construct` | table | Yes (if no `template`) | A field-by-field composition blueprint. |

The `construct` table defines how each field of the output concept is composed. Each key is a field name, and the value defines the composition method:

| Value form | Method | Description |
|------------|--------|-------------|
| Literal (`string`, `integer`, `float`, `boolean`, `array`) | Fixed | The field value is the literal. |
| `{ from = "path" }` | Variable reference | The field value comes from a variable in working memory. `path` is a dotted path (e.g., `"match_analysis.score"`). |
| `{ from = "path", list_to_dict_keyed_by = "attr" }` | Variable reference with transform | Converts a list to a dict keyed by the named attribute. |
| `{ template = "..." }` | Template | The field value is rendered from a Jinja2 template string. |
| Nested table (no `from` or `template` key) | Nested construct | The field is recursively composed from a nested construct. |

**Validation rules (construct mode):**

- The root variable of every `from` path and every template variable MUST correspond to a declared input.
- `from` and `template` are mutually exclusive within a single field definition.

**Example — construct mode:**

```toml
[pipe.compose_interview_sheet]
type        = "PipeCompose"
description = "Compose the final interview sheet"
inputs      = { match_analysis = "MatchAnalysis", interview_questions = "InterviewQuestion[]" }
output      = "InterviewSheet"

[pipe.compose_interview_sheet.construct]
overall_match_score  = { from = "match_analysis.overall_match_score" }
matching_skills      = { from = "match_analysis.matching_skills" }
missing_skills       = { from = "match_analysis.missing_skills" }
questions            = { from = "interview_questions" }
```

## Controller: PipeSequence

Executes a series of sub-pipes in order. The output of each step is added to working memory and can be consumed by subsequent steps.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeSequence"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | — |
| `steps` | array of tables | Yes | Ordered list of sub-pipe invocations. MUST contain at least one step. |

Each step is a **sub-pipe blueprint**:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `pipe` | string | Yes | Pipe reference (bare, domain-qualified, or package-qualified). |
| `result` | string | No | Name under which the step's output is stored in working memory. |
| `nb_output` | integer | No | Expected number of output items. Mutually exclusive with `multiple_output`. |
| `multiple_output` | boolean | No | Whether to expect multiple output items. Mutually exclusive with `nb_output`. |
| `batch_over` | string | No | Working memory variable to iterate over (inline batch). Requires `batch_as`. |
| `batch_as` | string | No | Name for each item during inline batch iteration. Requires `batch_over`. |

**Validation rules:**

- `steps` MUST contain at least one entry.
- `nb_output` and `multiple_output` MUST NOT both be set on the same step.
- `batch_over` and `batch_as` MUST either both be present or both be absent.
- `batch_over` and `batch_as` MUST NOT be the same value.

**Example:**

```toml
[pipe.process_document]
type        = "PipeSequence"
description = "Full document processing pipeline"
inputs      = { document = "Document" }
output      = "AnalysisResult"
steps = [
    { pipe = "extract_pages", result = "pages" },
    { pipe = "analyze_content", result = "analysis" },
    { pipe = "generate_summary", result = "summary" },
]
```

## Controller: PipeParallel

Executes multiple sub-pipes concurrently. Each branch operates independently.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeParallel"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | — |
| `branches` | array of tables | Yes | List of sub-pipe invocations to execute concurrently. |
| `add_each_output` | boolean | No | If `true`, each branch's output is individually added to working memory under its `result` name. Default: `false`. |
| `combined_output` | string | No | Concept reference for a combined output that merges all branch results. |

**Validation rules:**

- At least one of `add_each_output` or `combined_output` MUST be set (otherwise the pipe produces no output).
- `combined_output`, if present, MUST be a valid concept reference.
- Each branch follows the same sub-pipe blueprint format as `PipeSequence` steps.

**Example:**

```toml
[pipe.extract_documents]
type        = "PipeParallel"
description = "Extract text from both CV and job offer concurrently"
inputs      = { cv_pdf = "Document", job_offer_pdf = "Document" }
output      = "Page[]"
add_each_output = true
branches = [
    { pipe = "extract_cv", result = "cv_pages" },
    { pipe = "extract_job_offer", result = "job_offer_pages" },
]
```

## Controller: PipeCondition

Routes execution to different pipes based on an evaluated condition.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeCondition"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | No | — |
| `output` | string | Yes | — |
| `expression_template` | string | Conditional | A Jinja2 template that evaluates to a string matching an outcome key. Exactly one of `expression_template` or `expression` MUST be provided. |
| `expression` | string | Conditional | A static expression string. Exactly one of `expression_template` or `expression` MUST be provided. |
| `outcomes` | table | Yes | Maps outcome strings to pipe references. MUST have at least one entry. |
| `default_outcome` | string | Yes | The pipe reference (or special outcome) to use when no outcome key matches. |
| `add_alias_from_expression_to` | string | No | If set, stores the evaluated expression value in working memory under this name. |

**Special outcomes:**

Certain string values in `outcomes` values and `default_outcome` have special meaning and are not treated as pipe references:

| Value | Meaning |
|-------|---------|
| `"fail"` | Abort execution with an error. |
| `"continue"` | Skip this branch and continue without executing a sub-pipe. |

**Example:**

```toml
[pipe.route_by_document_type]
type                = "PipeCondition"
description         = "Route processing based on document type"
inputs              = { doc_request = "DocumentRequest" }
output              = "Text"
expression_template = "{{ doc_request.document_type }}"
default_outcome     = "continue"

[pipe.route_by_document_type.outcomes]
technical = "process_technical"
business  = "process_business"
legal     = "process_legal"
```

## Controller: PipeBatch

Maps a single pipe over each item in a list input, producing a list output.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"PipeBatch"` | Yes | — |
| `description` | string | Yes | — |
| `inputs` | table | Yes | MUST include an entry whose name matches `input_list_name`. |
| `output` | string | Yes | — |
| `branch_pipe_code` | string | Yes | The pipe reference to invoke for each item. |
| `input_list_name` | string | Yes | The name of the input that contains the list to iterate over. |
| `input_item_name` | string | Yes | The name under which each individual item is passed to the branch pipe. |

**Validation rules:**

- `input_list_name` MUST exist as a key in `inputs`.
- `input_item_name` MUST NOT be empty.
- `input_item_name` MUST NOT equal `input_list_name`.
- `input_item_name` MUST NOT equal any key in `inputs`.

**Example:**

```toml
[pipe.batch_generate_jokes]
type             = "PipeBatch"
description      = "Generate a joke for each topic"
inputs           = { topics = "Topic[]" }
output           = "Joke[]"
branch_pipe_code = "generate_joke"
input_list_name  = "topics"
input_item_name  = "topic"
```

## Pipe Reference Syntax

Every location in a `.mthds` file that references another pipe supports three forms:

| Form | Syntax | Example | Resolution |
|------|--------|---------|------------|
| Bare | `pipe_code` | `"extract_clause"` | Resolved within the current bundle and its domain. |
| Domain-qualified | `domain.pipe_code` | `"legal.contracts.extract_clause"` | Resolved within the named domain of the current package. |
| Package-qualified | `alias->domain.pipe_code` | `"docproc->extraction.extract_text"` | Resolved in the named domain of the dependency identified by the alias. |

Pipe references appear in:

- `steps[].pipe` (PipeSequence)
- `branches[].pipe` (PipeParallel)
- `outcomes` values (PipeCondition)
- `default_outcome` (PipeCondition)
- `branch_pipe_code` (PipeBatch)

Pipe *definitions* (the `[pipe.<pipe_code>]` table keys) are always bare `snake_case` names. Namespacing applies only to pipe *references*.

## Concept Reference Syntax

Every location that references a concept supports three forms, symmetric with pipe references:

| Form | Syntax | Example | Resolution |
|------|--------|---------|------------|
| Bare | `ConceptCode` | `"ContractClause"` | Resolved in order: native concepts → current bundle → same domain. |
| Domain-qualified | `domain.ConceptCode` | `"legal.contracts.NonCompeteClause"` | Resolved within the named domain of the current package. |
| Package-qualified | `alias->domain.ConceptCode` | `"acme->legal.ContractClause"` | Resolved in the named domain of the dependency identified by the alias. |

The disambiguation between concepts and pipes in a domain-qualified reference relies on casing:

- `snake_case` final segment → pipe code
- `PascalCase` final segment → concept code

Concept references appear in:

- `inputs` values
- `output`
- `refines`
- `concept_ref` and `item_concept_ref` in structure field blueprints
- `combined_output` (PipeParallel)

## Complete Bundle Example

```toml
domain      = "joke_generation"
description = "Generating one-liner jokes from topics"
main_pipe   = "generate_jokes_from_topics"

[concept.Topic]
description = "A subject or theme that can be used as the basis for a joke."
refines     = "Text"

[concept.Joke]
description = "A humorous one-liner intended to make people laugh."
refines     = "Text"

[pipe.generate_jokes_from_topics]
type        = "PipeSequence"
description = "Generate 3 joke topics and create a joke for each"
output      = "Joke[]"
steps = [
    { pipe = "generate_topics", result = "topics" },
    { pipe = "batch_generate_jokes", result = "jokes" },
]

[pipe.generate_topics]
type   = "PipeLLM"
description = "Generate 3 distinct topics suitable for jokes"
output = "Topic[3]"
prompt = "Generate 3 distinct and varied topics for crafting one-liner jokes."

[pipe.batch_generate_jokes]
type             = "PipeBatch"
description      = "Generate a joke for each topic"
inputs           = { topics = "Topic[]" }
output           = "Joke[]"
branch_pipe_code = "generate_joke"
input_list_name  = "topics"
input_item_name  = "topic"

[pipe.generate_joke]
type        = "PipeLLM"
description = "Write a clever one-liner joke about the given topic"
inputs      = { topic = "Topic" }
output      = "Joke"
prompt      = "Write a clever one-liner joke about $topic. Be concise and witty."
```


## METHODS.toml Format

# METHODS.toml Manifest Format

The `METHODS.toml` file is the package manifest — the identity card and dependency declaration for an MTHDS package. It MUST be named exactly `METHODS.toml` and MUST be located at the root of the package directory.

## File Encoding and Syntax

`METHODS.toml` MUST be a valid TOML document encoded in UTF-8.

## Top-Level Sections

A `METHODS.toml` file contains up to three top-level sections:

| Section | Required | Description |
|---------|----------|-------------|
| `[package]` | Yes | Package identity and metadata. |
| `[dependencies]` | No | Dependencies on other MTHDS packages. **Not yet implemented.** |
| `[exports]` | No | Visibility declarations for pipes. |

## The `[package]` Section

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | Yes | The name of the method. MUST be `snake_case` (matching `[a-z][a-z0-9_]*`), 2-25 characters. See [Name](#name). |
| `address` | string | Yes | Globally unique package identifier. MUST follow the hostname/path pattern. |
| `display_name` | string | No | Human-friendly display label. When provided, MUST NOT be empty or whitespace-only, MUST NOT exceed 128 characters, and MUST NOT contain Unicode control characters (category Cc). Leading and trailing whitespace is stripped. |
| `version` | string | Yes | Package version. MUST be valid [semantic versioning](https://semver.org/) (`MAJOR.MINOR.PATCH`, with optional pre-release and build metadata). |
| `description` | string | Yes | Human-readable summary of the package's purpose. MUST NOT be empty. |
| `authors` | array of strings | No | List of author identifiers (e.g., `"Name <email>"`). Default: empty list. |
| `license` | string | No | SPDX license identifier (e.g., `"MIT"`, `"Apache-2.0"`). |
| `mthds_version` | string | No | MTHDS standard version constraint. If set, MUST be a valid version constraint. |
| `main_pipe` | string | No | The package's entry-point pipe code. MUST be `snake_case` (matching `[a-z][a-z0-9_]*`). MUST reference a pipe declared in the `[exports]` section. See [Main Pipe](#main-pipe). |

### Address Format

The package address is the globally unique identifier for the package. It doubles as the fetch location for VCS-based distribution.

**Pattern:** `^[a-zA-Z0-9._-]+\.[a-zA-Z0-9._-]+/[a-zA-Z0-9._/-]+$`

In plain language: the address MUST start with a hostname (containing at least one dot), followed by a `/`, followed by one or more path segments.

**Examples of valid addresses:**

```
github.com/acme/legal-tools
github.com/mthds/document-processing
gitlab.com/company/internal-methods
```

**Examples of invalid addresses:**

```
legal-tools                     # No hostname
acme/legal-tools                # No dot in hostname
```

### Name

The `name` field is the name of the method. It is the primary identifier for the method — it appears in CLI output, registry listings, and documentation.

**Constraints:**

- MUST be `snake_case`, matching the pattern `[a-z][a-z0-9_]*`.
- MUST be between 2 and 25 characters.
- MUST NOT be empty.

The package directory name MUST match the `name` field exactly. A compliant tool MUST reject a package whose directory name does not match its manifest name.

**Example:**

```toml
# Inside nda_analyzer/METHODS.toml
[package]
name    = "nda_analyzer"
address = "github.com/acme/legal-tools"
```

### Display Name

The optional `display_name` field provides a human-friendly label for the package. It appears in CLI output, registry listings, and error messages. It is cosmetic only — the `address` remains the sole canonical identifier.

**Constraints:**

- MUST NOT be empty or whitespace-only when provided.
- MUST NOT exceed 128 characters (after leading/trailing whitespace is stripped).
- MUST NOT contain Unicode control characters (Unicode general category `Cc`).
- Emojis and other Unicode characters are allowed.
- Leading and trailing whitespace is stripped by a compliant implementation.

**Example:**

```toml
[package]
name         = "nda_analyzer"
address      = "github.com/acme/legal-tools"
display_name = "Nda Analyzer"
```

### Version Format

The `version` field MUST conform to [Semantic Versioning 2.0.0](https://semver.org/):

```
MAJOR.MINOR.PATCH[-pre-release][+build-metadata]
```

**Examples:** `1.0.0`, `0.3.0`, `2.1.3-beta.1`, `1.0.0-rc.1+build.42`

### mthds_version Constraints

The `mthds_version` field, if present, declares which versions of the MTHDS standard this package is compatible with. It uses version constraint syntax (see [Version Constraint Syntax](#version-constraint-syntax)).

The current MTHDS standard version is `1.0.0`.

### Main Pipe

The optional `main_pipe` field designates the package's primary entry point — the pipe that runs when a user invokes the package by slug or address:

```bash
mthds run method nda_analyzer
mthds run method github.com/acme/legal-tools
```

**Constraints:**

- The value MUST be a valid `snake_case` pipe code (matching `[a-z][a-z0-9_]*`).
- The referenced pipe MUST be declared in the `[exports]` section. A manifest that sets `main_pipe` to a pipe not listed in exports is invalid.
- When `main_pipe` is not set, the package has no default entry point. It can still be consumed as a library by importing specific pipes.

**Example:**

```toml
[package]
name      = "nda_analyzer"
address   = "github.com/acme/legal-tools"
version   = "0.3.0"
main_pipe = "analyze_nda"
```

## The `[dependencies]` Section

> **Not yet implemented.** Dependencies between packages are planned but not yet supported. The specification below describes the intended behavior for a future release.

Each entry in `[dependencies]` declares a dependency on another MTHDS package. The key is the **alias** — a `snake_case` identifier used in cross-package references (`->` syntax).

```toml
[dependencies]
docproc     = { address = "github.com/mthds/document-processing", version = "^1.0.0" }
scoring_lib = { address = "github.com/mthds/scoring-lib", version = "^0.5.0" }
```

### Dependency Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `address` | string | Yes | The dependency's package address. MUST follow the hostname/path pattern. |
| `version` | string | Yes | Version constraint for the dependency (see [Version Constraint Syntax](#version-constraint-syntax)). |
| `path` | string | No | Local filesystem path to the dependency, resolved relative to the manifest directory. For development-time workflows. |

### Alias Rules

- The alias (the TOML key) MUST be `snake_case`, matching `[a-z][a-z0-9_]*`.
- All aliases within a single `[dependencies]` section MUST be unique.
- The alias is used in cross-package references: `alias->domain.name`.

### The `path` Field

When `path` is set, the dependency is resolved from the local filesystem instead of being fetched via VCS. This supports development-time workflows where packages are co-located on disk, similar to Cargo's `path` dependencies or Go's `replace` directives.

- The path is resolved relative to the directory containing `METHODS.toml`.
- Local path dependencies are NOT resolved transitively — only the root package's local paths are honored.
- Local path dependencies are excluded from the lock file.

**Example:**

```toml
[dependencies]
scoring = { address = "github.com/mthds/scoring-lib", version = "^0.5.0", path = "../scoring-lib" }
```

### Version Constraint Syntax

Version constraints specify which versions of a dependency are acceptable.

| Form | Syntax | Example | Meaning |
|------|--------|---------|---------|
| Exact | `MAJOR.MINOR.PATCH` | `1.0.0` | Exactly this version. |
| Caret | `^MAJOR.MINOR.PATCH` | `^1.0.0` | Compatible release (same major version). |
| Tilde | `~MAJOR.MINOR.PATCH` | `~1.0.0` | Approximately compatible (same major.minor). |
| Greater-or-equal | `>=MAJOR.MINOR.PATCH` | `>=1.0.0` | This version or newer. |
| Less-than | `<MAJOR.MINOR.PATCH` | `<2.0.0` | Older than this version. |
| Greater | `>MAJOR.MINOR.PATCH` | `>1.0.0` | Newer than this version. |
| Less-or-equal | `<=MAJOR.MINOR.PATCH` | `<=2.0.0` | This version or older. |
| Equal | `==MAJOR.MINOR.PATCH` | `==1.0.0` | Exactly this version. |
| Not-equal | `!=MAJOR.MINOR.PATCH` | `!=1.0.0` | Any version except this one. |
| Compound | constraint `, ` constraint | `>=1.0.0, <2.0.0` | Both constraints must be satisfied. |
| Wildcard | `*`, `MAJOR.*`, `MAJOR.MINOR.*` | `1.*` | Any version matching the prefix. |

Partial versions are allowed: `1.0` is equivalent to `1.0.*`.

## The `[exports]` Section

The `[exports]` section controls which pipes are visible to consumers of the package.

**Default visibility rules:**

- **Concepts are always public.** Concepts are vocabulary — they are always accessible from outside the package.
- **Pipes are private by default.** A pipe not listed in `[exports]` is an implementation detail, invisible to consumers.
- **`main_pipe` must be exported.** If a package declares a `main_pipe`, that pipe MUST appear in the `[exports]` section.

### Exports Table Structure

The `[exports]` section uses nested TOML tables that mirror the domain hierarchy. The domain path maps directly to the TOML table path:

```toml
[exports.legal]
pipes = ["classify_document"]

[exports.legal.contracts]
pipes = ["extract_clause", "analyze_nda", "compare_contracts"]

[exports.scoring]
pipes = ["compute_weighted_score"]
```

Each leaf table contains:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `pipes` | array of strings | Yes | Pipe codes that are public from this domain. Each entry MUST be a valid pipe code (`snake_case`). |

**Validation rules:**

- Domain paths in `[exports]` MUST be valid domain codes.
- Domain paths in `[exports]` MUST NOT start with a reserved domain segment (`native`, `mthds`, `pipelex`).
- A domain MAY have both a `pipes` list and sub-domain tables (e.g., `[exports.legal]` with `pipes` AND `[exports.legal.contracts]`).

### Standalone Bundles (No Manifest)

A `.mthds` file without a `METHODS.toml` manifest is a standalone bundle. It behaves as an implicit local package with:

- No dependencies (beyond native concepts).
- All pipes treated as public (no visibility restrictions).
- No package address (not distributable).

This preserves the "single file = working method" experience for learning, prototyping, and simple projects.

## Package Directory Structure

A package is a directory containing a `METHODS.toml` manifest and one or more `.mthds` bundle files. The directory layout follows a progressive enhancement principle — start minimal, add structure as needed.

**Minimal package:**

```
my_tool/
├── METHODS.toml
└── main.mthds
```

**Full package:**

```
legal_tools/
├── METHODS.toml
├── methods.lock
├── general_legal.mthds
├── contract_analysis.mthds
├── shareholder_agreements.mthds
├── scoring.mthds
├── README.md
└── LICENSE
```

**Rules:**

- `METHODS.toml` MUST be at the directory root.
- `methods.lock` MUST be at the directory root, alongside `METHODS.toml`.
- `.mthds` files MAY be at the root or in subdirectories. A compliant implementation MUST discover all `.mthds` files recursively.
- A single directory SHOULD contain one package. Multiple packages in subdirectories with distinct addresses are possible but outside the scope of this specification.

## Manifest Discovery

When loading a `.mthds` bundle, a compliant implementation SHOULD discover the manifest by walking up from the bundle file's directory:

1. Check the current directory for `METHODS.toml`.
2. If not found, move to the parent directory.
3. Stop when `METHODS.toml` is found, a `.git` directory is encountered, or the filesystem root is reached.
4. If no manifest is found, the bundle is treated as a standalone bundle (no package).

## Complete Manifest Example

```toml
[package]
name          = "nda_analyzer"
address       = "github.com/acme/legal-tools"
display_name  = "Legal Tools"
version       = "0.3.0"
description   = "Legal document analysis and contract review methods."
authors       = ["ACME Legal Tech <legal@acme.com>"]
license       = "MIT"
mthds_version = ">=1.0.0"
main_pipe     = "analyze_nda"

[exports.legal]
pipes = ["classify_document"]

[exports.legal.contracts]
pipes = ["extract_clause", "analyze_nda", "compare_contracts"]

[exports.scoring]
pipes = ["compute_weighted_score"]
```


## methods.lock Format

# methods.lock Format

The `methods.lock` file records the exact resolved versions and integrity hashes for all remote dependencies, enabling reproducible builds. It is auto-generated and SHOULD be committed to version control.

## File Name and Location

The lock file MUST be named `methods.lock` and MUST be located at the root of the package directory, alongside `METHODS.toml`.

## File Encoding and Syntax

`methods.lock` MUST be a valid TOML document encoded in UTF-8.

## Structure

The lock file is a flat TOML document where each top-level table key is a package address, and the value is a table containing the locked metadata for that package.

```toml
["github.com/mthds/document-processing"]
version = "1.2.3"
hash    = "sha256:a1b2c3d4e5f6..."
source  = "https://github.com/mthds/document-processing"

["github.com/mthds/scoring-lib"]
version = "0.5.1"
hash    = "sha256:e5f6a7b8c9d0..."
source  = "https://github.com/mthds/scoring-lib"
```

Because package addresses contain dots and slashes, they MUST be quoted as TOML keys.

## Locked Package Fields

Each entry in the lock file contains:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `version` | string | Yes | The exact resolved version. MUST be valid semver. |
| `hash` | string | Yes | Integrity hash of the package contents. MUST match the pattern `sha256:[0-9a-f]{64}`. |
| `source` | string | Yes | The HTTPS URL from which the package was fetched. MUST start with `https://`. |

## Hash Computation

The integrity hash is a deterministic SHA-256 hash of the package directory contents, computed as follows:

1. Collect all regular files recursively under the package directory.
2. Exclude any path containing `.git` in its components.
3. Sort files by their POSIX-normalized relative path (for cross-platform determinism).
4. For each file in sorted order, feed into the hasher:
   a. The relative path string, encoded as UTF-8.
   b. The raw file bytes.
5. The resulting hash is formatted as `sha256:` followed by the 64-character lowercase hex digest.

## Which Packages Are Locked

- **Remote dependencies** (those without a `path` field in the root manifest) are locked, including all transitive remote dependencies.
- **Local path dependencies** are NOT locked. They are resolved from the filesystem at load time and are expected to change during development.

## When the Lock File Updates

The lock file is regenerated when:

- `mthds pkg lock` is run — resolves all dependencies and writes the lock file.
- `mthds pkg update` is run — re-resolves to latest compatible versions and rewrites the lock file.
- `mthds pkg add` is run — adds a new dependency and may trigger re-resolution.

## Verification

When installing from a lock file (`mthds pkg install`), a compliant implementation MUST:

1. For each entry in the lock file, locate the corresponding cached package directory.
2. Recompute the SHA-256 hash of the cached directory using the algorithm described above.
3. Compare the computed hash with the `hash` field in the lock file.
4. Reject the installation if any hash does not match (integrity failure).

## Deterministic Output

Lock file entries MUST be sorted by package address (lexicographic ascending) to produce deterministic output suitable for clean version control diffs.

An empty lock file (no remote dependencies) MAY be an empty file or absent entirely.


## Namespace Resolution Rules

# Namespace Resolution Rules

This page defines the formal rules for resolving references to concepts and pipes across bundles, domains, and packages.

## Reference Syntax Overview

All references to concepts and pipes in MTHDS follow a uniform three-tier syntax:

| Tier | Syntax | Example (concept) | Example (pipe) |
|------|--------|--------------------|----------------|
| Bare | `name` | `ContractClause` | `extract_clause` |
| Domain-qualified | `domain_path.name` | `legal.contracts.NonCompeteClause` | `legal.contracts.extract_clause` |
| Package-qualified | `alias->domain_path.name` | `acme->legal.ContractClause` | `docproc->extraction.extract_text` |

## Parsing Rules

### Splitting Cross-Package References

If the reference string contains `->`, it is a cross-package reference. The string is split on the first `->`:

- Left part: the package alias.
- Right part: the remainder (a domain-qualified or bare reference).

The alias MUST be `snake_case`. The remainder is parsed as a domain-qualified or bare reference.

### Splitting Domain-Qualified References

For the remainder (or the entire string if no `->` is present), the reference is parsed by splitting on the **last `.`** (dot):

- Left part: the domain path.
- Right part: the local code (concept code or pipe code).

If no `.` is present, the reference is a bare name with no domain qualification.

**Examples:**

| Reference | Domain Path | Local Code | Type |
|-----------|-------------|------------|------|
| `extract_clause` | *(none)* | `extract_clause` | Bare pipe |
| `NonCompeteClause` | *(none)* | `NonCompeteClause` | Bare concept |
| `scoring.compute_score` | `scoring` | `compute_score` | Domain-qualified pipe |
| `legal.contracts.NonCompeteClause` | `legal.contracts` | `NonCompeteClause` | Domain-qualified concept |
| `docproc->extraction.extract_text` | `extraction` (in package `docproc`) | `extract_text` | Package-qualified pipe |

### Disambiguation: Concept vs. Pipe

When parsing a domain-qualified reference, the casing of the local code (the segment after the last `.`) determines whether it is a concept or a pipe:

- `PascalCase` (`[A-Z][a-zA-Z0-9]*`) → concept code.
- `snake_case` (`[a-z][a-z0-9_]*`) → pipe code.

This disambiguation is unambiguous because concept codes and pipe codes follow mutually exclusive casing conventions.

## Domain Path Validation

Each segment of a domain path MUST be `snake_case`:

- Match pattern: `[a-z][a-z0-9_]*`
- Segments are separated by `.`
- No leading, trailing, or consecutive dots

## Resolution Order for Bare Concept References

When resolving a bare concept code (no domain qualifier, no package prefix):

1. **Native concepts** — check if the code matches a native concept code (`Dynamic`, `Text`, `Image`, `Document`, `Html`, `TextAndImages`, `Number`, `Page`, `JSON`, `SearchResult`, `Anything`). Native concepts always take priority.
2. **Current bundle** — check concepts declared in the same `.mthds` file.
3. **Same domain, other bundles** — if the bundle is part of a package, check concepts in other bundles that declare the same domain.
4. **Error** — if not found in any of the above, the reference is invalid.

Bare concept references do NOT fall through to other domains or other packages.

## Resolution Order for Bare Pipe References

When resolving a bare pipe code (no domain qualifier, no package prefix):

1. **Current bundle** — check pipes declared in the same `.mthds` file.
2. **Same domain, other bundles** — if the bundle is part of a package, check pipes in other bundles that declare the same domain.
3. **Error** — if not found, the reference is invalid.

Bare pipe references do NOT fall through to other domains or other packages.

## Resolution of Domain-Qualified References

When resolving `domain_path.name` (no package prefix):

1. Look in the named domain within the **current package**.
2. If not found: **error**. Domain-qualified references do not fall through to dependencies.

This applies to both concept and pipe references.

## Resolution of Package-Qualified References

When resolving `alias->domain_path.name`:

1. Identify the dependency by the alias. The alias MUST match a key in the `[dependencies]` section of the consuming package's `METHODS.toml`.
2. Look in the named domain of the **resolved dependency package**.
3. If not found: **error**.

**Visibility constraints for cross-package pipe references:**

- The referenced pipe MUST be exported by the dependency package (listed in its `[exports]` section or declared as `main_pipe` in its bundle header).
- If the pipe is not exported, the reference is a visibility error.

**Visibility for cross-package concept references:**

- Concepts are always public. No visibility check is needed for cross-package concept references.

## Visibility Rules (Intra-Package)

Within a package that has a `METHODS.toml` manifest:

- **Same-domain references** — always allowed. A pipe in domain `legal.contracts` can reference any other pipe in `legal.contracts` without restriction.
- **Cross-domain references** (within the same package) — the target pipe MUST be exported. A pipe in domain `scoring` referencing `legal.contracts.extract_clause` requires that `extract_clause` is listed in `[exports.legal.contracts]` (or is the `main_pipe` of a bundle in `legal.contracts`).
- **Bare references** — always allowed at the visibility level (they resolve within the same domain).

When no manifest is present (standalone bundle), all pipes are treated as public.

## Reserved Domains

The following domain names are reserved at the first segment level:

| Domain | Owner | Purpose |
|--------|-------|---------|
| `native` | MTHDS standard | Built-in concept types. |
| `mthds` | MTHDS standard | Reserved for future standard extensions. |
| `pipelex` | Reference implementation | Reserved for the reference implementation. |

**Enforcement points:**

- A compliant implementation MUST reject `METHODS.toml` exports that use a reserved domain path.
- A compliant implementation MUST reject bundles that declare a domain starting with a reserved segment when the bundle is part of a package.
- A compliant implementation MUST reject packages at publish time if any bundle uses a reserved domain.

The `native` domain is the only reserved domain with active semantics: it serves as the namespace for native concepts (`native.Text`, `native.Image`, etc.).

## Package Namespace Isolation

Two packages MAY declare the same domain name (e.g., both declare `domain = "recruitment"`). Their concepts and pipes are completely independent — there is no merging of namespaces across packages.

Within a single package, bundles that share the same domain DO merge their namespace. Concept or pipe code collisions within the same package and same domain are errors.

## Conflict Rules

| Scope | Conflict type | Result |
|-------|--------------|--------|
| Same bundle | Duplicate concept code | TOML parse error (duplicate key). |
| Same bundle | Duplicate pipe code | TOML parse error (duplicate key). |
| Same domain, different bundles (same package) | Duplicate concept code | Error at load time. |
| Same domain, different bundles (same package) | Duplicate pipe code | Error at load time. |
| Different domains (same package) | Same concept or pipe code | No conflict — different namespaces. |
| Different packages | Same domain and same concept/pipe code | No conflict — package isolation. |

## Version Resolution Strategy

When resolving dependency versions, a compliant implementation SHOULD use **Minimum Version Selection** (MVS), following Go's approach:

1. Collect all version constraints for a given package address from all dependents (direct and transitive).
2. List all available versions (from VCS tags).
3. Sort versions in ascending order.
4. Select the **minimum** version that satisfies **all** constraints simultaneously.

If no version satisfies all constraints, the resolution fails with an error.

**Properties of MVS:**

- **Deterministic** — the same set of constraints always produces the same result.
- **Reproducible** — no dependency on a "latest" query or timestamp.
- **Simple** — no backtracking solver needed.

## Transitive Dependency Resolution

Dependencies are resolved transitively with the following rules:

- **Remote dependencies** are resolved recursively. If Package A depends on Package B, and Package B depends on Package C, then Package C is also resolved.
- **Local path dependencies** are resolved at the root level only. They are NOT resolved transitively.
- **Cycle detection** — if a dependency is encountered while it is already on the resolution stack, the resolver MUST report a cycle error.
- **Diamond dependencies** — when the same package address is required by multiple dependents with different version constraints, MVS selects the minimum version satisfying all constraints simultaneously.

## Fetching Remote Dependencies

Package addresses map to Git clone URLs by the following rule:

1. Prepend `https://`.
2. Append `.git` (if not already present).

For example: `github.com/acme/legal-tools` → `https://github.com/acme/legal-tools.git`

The resolution chain for fetching a dependency is:

1. **Local path** — if the dependency has a `path` field in `METHODS.toml`, resolve from the local filesystem.
2. **Local cache** — check `~/.mthds/packages/{address}/{version}/` for a cached copy.
3. **VCS fetch** — clone the repository at the resolved version tag using `git clone --depth 1 --branch {tag}`.

Version tags in the remote repository MAY use a `v` prefix (e.g., `v1.0.0`). The prefix is stripped during version parsing.

## Cache Layout

The default package cache is located at `~/.mthds/packages/`. Cached packages are stored at:

```
~/.mthds/packages/{address}/{version}/
```

For example:

```
~/.mthds/packages/github.com/acme/legal-tools/1.0.0/
```

The `.git` directory is removed from cached copies.

## Cross-Package Reference Examples

The following examples illustrate the complete reference resolution for cross-package scenarios.

**Setup:** Package A depends on Package B with alias `scoring_lib`.

Package B (`METHODS.toml`):

```toml
[package]
address = "github.com/mthds/scoring-lib"
version = "0.5.0"
description = "Scoring utilities"

[exports.scoring]
pipes = ["compute_weighted_score"]
```

Package B (`scoring.mthds`):

```toml
domain    = "scoring"
main_pipe = "compute_weighted_score"

[concept.ScoreResult]
description = "A weighted score result"

[pipe.compute_weighted_score]
type        = "PipeLLM"
description = "Compute a weighted score"
inputs      = { item = "Text" }
output      = "ScoreResult"
prompt      = "Compute a weighted score for: $item"

[pipe.internal_helper]
type        = "PipeLLM"
description = "Internal helper (not exported)"
inputs      = { data = "Text" }
output      = "Text"
prompt      = "Process: $data"
```

Package A (`analysis.mthds`):

```toml
domain = "analysis"

[pipe.analyze_item]
type        = "PipeSequence"
description = "Analyze using scoring dependency"
inputs      = { item = "Text" }
output      = "Text"
steps = [
    { pipe = "scoring_lib->scoring.compute_weighted_score", result = "score" },
    { pipe = "summarize", result = "summary" },
]
```

**Resolution of `scoring_lib->scoring.compute_weighted_score`:**

1. `->` detected — split into alias `scoring_lib` and remainder `scoring.compute_weighted_score`.
2. Look up `scoring_lib` in Package A's `[dependencies]` — found, resolves to `github.com/mthds/scoring-lib`.
3. Parse remainder: split on last `.` → domain `scoring`, pipe code `compute_weighted_score`.
4. Look in domain `scoring` of the resolved Package B — pipe found.
5. Visibility check: `compute_weighted_score` is in `[exports.scoring]` pipes — accessible.
6. Resolution succeeds.

**If Package A tried `scoring_lib->scoring.internal_helper`:**

1. Steps 1–4 as above — pipe `internal_helper` is found in Package B's `scoring` domain.
2. Visibility check: `internal_helper` is NOT in `[exports.scoring]` and is NOT `main_pipe` — **visibility error**.

**Cross-package concept reference:**

```toml
[concept.DetailedScore]
description = "An extended score with additional analysis"
refines     = "scoring_lib->scoring.ScoreResult"
```

This refines `ScoreResult` from Package B. Concepts are always public, so no visibility check is needed.

## Validation Rule Summary

This section consolidates the validation rules scattered throughout this specification into a single reference.

### Bundle-Level Validation

1. The file MUST be valid TOML.
2. `domain` MUST be present and MUST be a valid domain code.
3. `main_pipe`, if present, MUST be `snake_case` and MUST reference a pipe defined in the same bundle.
4. Concept codes MUST be `PascalCase`.
5. Concept codes MUST NOT match any native concept code.
6. Pipe codes MUST be `snake_case`.
7. `refines` and `structure` MUST NOT both be set on the same concept.
8. Local concept references (bare or same-domain) MUST resolve to a declared concept in the bundle or a native concept.
9. Same-domain pipe references MUST resolve to a declared pipe in the bundle.
10. Cross-package references (`->` syntax) are deferred to package-level validation.

### Concept Structure Field Validation

1. `description` MUST be present on every field.
2. If `type` is omitted, `choices` MUST be non-empty.
3. `type = "dict"` requires both `key_type` and `value_type`.
4. `type = "concept"` requires `concept_ref` and forbids `default_value`.
5. `type = "list"` with `item_type = "concept"` requires `item_concept_ref`.
6. `concept_ref` MUST NOT be set unless `type = "concept"`.
7. `item_concept_ref` MUST NOT be set unless `item_type = "concept"`.
8. `default_value` type MUST match the declared `type`.
9. If `choices` is set and `default_value` is present, `default_value` MUST be in `choices`.
10. Field names MUST NOT start with `_`.

### Pipe Validation (Type-Specific)

1. **PipeLLM**: All prompt variables MUST have matching inputs. All inputs MUST be used.
2. **PipeStructure**: Exactly one input MUST be declared. The input concept MUST be `Text` or refine `Text`. `output` MUST NOT be `Text` or refine `Text`.
3. **PipeFunc**: `function_name` MUST be present.
4. **PipeImgGen**: `prompt` MUST be present. All prompt variables MUST have matching inputs.
5. **PipeExtract**: Exactly one input MUST be declared. `output` MUST be `"Page[]"`.
6. **PipeCompose**: Exactly one of `template` or `construct` MUST be present. Output MUST NOT use multiplicity.
7. **PipeSequence**: `steps` MUST have at least one entry.
8. **PipeParallel**: At least one of `add_each_output` or `combined_output` MUST be set.
9. **PipeCondition**: Exactly one of `expression_template` or `expression` MUST be present. `outcomes` MUST have at least one entry.
10. **PipeBatch**: `input_list_name` MUST be in `inputs`. `input_item_name` MUST NOT equal `input_list_name` or any `inputs` key.

### Package-Level Validation

1. `[package]` section MUST be present in `METHODS.toml`.
2. `address` MUST match the hostname/path pattern.
3. `version` MUST be valid semver.
4. `description` MUST NOT be empty.
5. All dependency aliases MUST be unique.
6. All dependency aliases MUST be `snake_case`.
7. All dependency addresses MUST match the hostname/path pattern.
8. All dependency version constraints MUST be valid.
9. Domain paths in `[exports]` MUST NOT use reserved domains.
10. All pipe codes in `[exports]` MUST be valid `snake_case`.
11. Cross-package references MUST reference known dependency aliases.
12. Cross-package pipe references MUST target exported pipes.
13. Bundles MUST NOT use reserved domains as their first segment.

### Lock File Validation

1. Each entry's `version` MUST be valid semver.
2. Each entry's `hash` MUST match `sha256:[0-9a-f]{64}`.
3. Each entry's `source` MUST start with `https://`.

## Summary: Reference Resolution Flowchart

Given a reference string `R`:

```
1. Does R contain "->"?
   YES → Split into (alias, remainder).
         Look up alias in [dependencies].
         Parse remainder as domain-qualified or bare ref.
         Resolve in the dependency's namespace.
         For pipes: check export visibility.
   NO  → Continue to step 2.

2. Does R contain "."?
   YES → Split on last "." into (domain_path, local_code).
         Resolve in domain_path within current package.
   NO  → R is a bare name. Continue to step 3.

3. Is R a concept code (PascalCase)?
   YES → Check native concepts → current bundle → same domain.
   NO  → R is a pipe code (snake_case).
         Check current bundle → same domain.

4. Not found? → Error.
```


## CLI I/O Contract

# CLI I/O Contract

This page defines the input/output contract for MTHDS methods when invoked as CLI commands. It specifies what a method writes to stdout, what it reads from stdin, and how errors propagate through pipe chains.

## Output Modes

A method's CLI produces structured JSON on stdout. Two output modes are defined: **compact** (default) and **full** (opt-in via `--with-memory`).

### Compact Output (Default)

The concept's rendered JSON is emitted directly — no envelope, no metadata:

```json
{
  "clauses": [
    { "title": "Non-Compete", "risk_level": "high" },
    { "title": "Termination", "risk_level": "medium" }
  ],
  "overall_risk": "high"
}
```

This is the structured content of the method's main output concept. Standard JSON tools work directly:

```bash
mthds-agent pipelex run method extract-terms | jq '.clauses[] | select(.risk_level == "high")'

# Or using the installed CLI shim:
extract-terms | jq '.clauses[] | select(.risk_level == "high")'
```

When the method has no main output (empty working memory), an empty object `{}` is emitted.

### Full Output (`--with-memory`)

When `--with-memory` is passed, the output includes the main stuff renderings and the full working memory:

```json
{
  "main_stuff": {
    "json": "<concept as JSON string>",
    "markdown": "<concept as Markdown string>",
    "html": "<concept as HTML string>"
  },
  "working_memory": {
    "root": {
      "contract_text": {
        "stuff_code": "...",
        "stuff_name": "contract_text",
        "concept": { "code": "Text" },
        "content": { "text": "The parties agree..." }
      },
      "extracted_terms": {
        "stuff_code": "...",
        "stuff_name": "extracted_terms",
        "concept": { "code": "ContractAnalysis" },
        "content": { "clauses": ["..."], "overall_risk": "high" }
      },
      "main_stuff": {
        "stuff_code": "...",
        "stuff_name": null,
        "concept": { "code": "ContractAnalysis" },
        "content": { "clauses": ["..."], "overall_risk": "high" }
      }
    },
    "aliases": {
      "main_stuff": "extracted_terms"
    }
  }
}
```

The full output preserves all intermediate results and aliases from the pipeline's working memory. This is required when piping output to another method, because the downstream method may need intermediate stuffs for multi-input binding.

### Side Effects

Regardless of output mode, the runtime may write side-effect files to disk:

- **Output JSON** (`live_run.json` / `dry_run.json`) — the full execution result saved alongside the bundle.
- **Graph HTML** (`live_run.html` / `dry_run.html`) — execution graph visualizations (generated by default, disabled with `--no-graph`).

These files are not included in stdout output. Their paths appear in runtime logs on stderr.

## Input Acceptance

A method accepts inputs through three sources, resolved in priority order:

### 1. `--inputs` Flag (Highest Priority)

The `--inputs` / `-i` flag accepts a file path or inline JSON string. If the value starts with `{`, it is parsed as inline JSON; otherwise it is treated as a file path.

```bash
# Inline JSON
mthds-agent pipelex run method my_method --inputs '{"text": {"concept": "native.Text", "content": {"text": "hello"}}}'

# File path
mthds-agent pipelex run method my_method --inputs data.json
```

When a method is installed as a CLI shim (see [CLI Reference](../cli/index.md)), the same commands are available as:

```bash
my_method --inputs '{"text": {"concept": "native.Text", "content": {"text": "hello"}}}'
my_method --inputs data.json
```

When `--inputs` is provided, stdin is ignored entirely. This allows overriding piped data for debugging.

### 2. stdin (Fallback)

When `--inputs` is not provided and stdin is not a TTY (i.e., data is being piped), JSON is read from stdin:

```bash
echo '{"text": {"concept": "native.Text", "content": {"text": "hello"}}}' | mthds-agent pipelex run method my_method
```

When stdin is a TTY (interactive terminal), no stdin reading occurs and the method runs without inputs.

### 3. Auto-detected `inputs.json`

In directory mode, `inputs.json` in the target directory is auto-detected and used as a fallback when no explicit inputs are provided.

## Envelope Detection

When JSON arrives via stdin, the runtime distinguishes between two formats based on the presence of a `working_memory` key at the top level:

### Flat Inputs

No `working_memory` key present. The JSON is treated as direct input bindings — the same format as `--inputs`:

```json
{
  "document": {
    "concept": "native.Document",
    "content": { "url": "/path/to/file.pdf" }
  }
}
```

### Full Envelope

A `working_memory` key is present. This indicates the JSON came from an upstream method's `--with-memory` output. The runtime extracts named stuffs from `working_memory.root` and converts them to input bindings.

The input resolution rules when receiving a full envelope:

1. **Name matching**: Each stuff name in the upstream working memory's `root` is matched against the downstream method's declared input names. Matching entries are bound automatically.

2. **Single-input shortcut**: If the downstream method declares exactly one input, it auto-binds to the upstream's `main_stuff` content. This is the common case for simple chains:

    ```bash
    extract-terms --with-memory | assess-risk
    ```

3. **Error on failure**: If the downstream method's declared inputs cannot be satisfied from the upstream's working memory, a clear error is raised listing what was available upstream vs. what was expected downstream.

## Error Propagation

Errors are emitted as structured JSON on **stderr** with a non-zero exit code:

```json
{
  "error": true,
  "error_type": "PipelineExecutionError",
  "message": "Pipe 'assess_risk' failed: missing required input 'analysis'",
  "hint": "Check 'pipe_stack' to identify which pipe failed",
  "error_domain": "runtime",
  "retryable": false
}
```

In a Unix pipe chain, errors stop execution at the failing step. Use `set -o pipefail` in shell scripts to ensure mid-chain failures propagate:

```bash
set -o pipefail
extract-terms --with-memory \
  | assess-risk --with-memory \
  | generate-report
```

## Examples

### Simple Chain (Single Input)

```bash
# Extract terms, then assess risk, then generate report
extract-terms --inputs '{"document": {"concept": "native.Document", "content": {"url": "contract.pdf"}}}' --with-memory \
  | assess-risk --with-memory \
  | generate-report
```

Each intermediate step uses `--with-memory` to pass the full envelope. The final step omits it to produce compact output.

### Compact Output with jq

```bash
# Extract just the high-risk clauses
extract-terms --inputs data.json \
  | jq '.clauses[] | select(.risk_level == "high")'
```

### Override Piped Input

```bash
# The --inputs flag overrides whatever comes from stdin
echo '{"old": "data"}' | mthds-agent pipelex run method my_method --inputs '{"new": "data"}'
```

The `--inputs` flag always wins — the stdin data is ignored.


## HTTP Runner Protocol

# HTTP Runner Protocol

The MTHDS Protocol is the minimal HTTP contract every MTHDS runner implements. Any server that serves these five routes with the shapes defined here is an MTHDS-compliant runner. A runner is just a runner: it executes methods, validates bundles, and reports what models it can route to and what version it is. It keeps no run store and owns no user, billing, or catalog concepts.

The normative artifact is the OpenAPI document: [`mthds-protocol.openapi.yaml`](openapi/mthds-protocol.openapi.yaml). This page walks through it in prose, then renders it route by route — every parameter, request body, and response schema — in the [route reference](#route-reference) below. Where prose and YAML disagree, the YAML wins.

## The five routes

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/execute` | Execute a method synchronously; the full output comes back in the response. |
| `POST` | `/start` | Start a method asynchronously; returns its `pipeline_run_id` immediately (202). Completion delivery is implementation-defined. |
| `POST` | `/validate` | Parse, validate, and dry-run an MTHDS bundle. |
| `GET` | `/models` | The models this runner can route to. Optional `?type=` filter (`llm` · `extract` · `img_gen` · `search`). |
| `GET` | `/version` | Always public. Protocol and runner versions — the handshake clients use for feature detection. |

All errors are [RFC 7807](https://www.rfc-editor.org/rfc/rfc7807) `application/problem+json` documents. Auth is implementation-defined: a bearer-token slot is reserved, and anonymous access is allowed for self-hosted runners.

## Base URL and versioning

Protocol paths are version-agnostic. The version segment belongs to the server base URL, chosen by the implementation:

```
http://localhost:8081/v1/execute
https://api.example.com/v1/execute
```

The protocol itself is versioned by this standard (`protocol_version` in `/version`); each implementation versions its own mount point. A client written against the protocol composes `{base_url}/{path}` and never inspects the base URL's structure.

## Executing a method

`POST /execute` is blocking: the response carries the method's full output. The request body is a `RunRequest`:

```json
{
  "pipe_code": "analyze_contract",
  "mthds_contents": ["domain = \"legal\"\n..."],
  "inputs": {
    "contract": { "concept": "Document", "content": { "file_path": "..." } }
  }
}
```

At least one of `pipe_code` / `mthds_contents` is required. If `mthds_contents` is provided without `pipe_code`, the first bundle must declare a `main_pipe`. Optional fields: `output_name`, `output_multiplicity`, `dynamic_output_concept_ref`.

The 200 response is a `RunResultExecute` — the completed run, holding two base fields: `pipeline_run_id` (mandatory, server-generated and authoritative) and `pipe_output` (the method's serialized output; always present — a completed run has output). Anything more an implementation returns — a run state, timestamps, output naming, anything else — is an extension field (see [Extension policy](#extension-policy)), declared and documented by that implementation.

The protocol sets no time limit on `/execute`; deployments cap it at their proxy layer. For long-running methods prefer `/start`. Implementations **MAY** answer `202 + RunResultStart` (just the run id, no output yet) with a `Location` header pointing at an implementation-defined status resource when they cannot hold the connection open ([RFC 9110](https://www.rfc-editor.org/rfc/rfc9110#section-15.3.3) asynchronous pattern). Simple runners never emit 202; clients that cannot handle it should use `/start`.

## Starting a method asynchronously

`POST /start` accepts the same `RunRequest` body as `/execute` — the protocol declares no start-only request fields. Anything an implementation accepts on top (a client-supplied run identifier, anything else) is an extension arg (see [Extension policy](#extension-policy)), defined and documented by that implementation.

The response is `202 + RunResultStart` — just the authoritative server-generated `pipeline_run_id` (plus any implementation extension fields). A started run has no output yet.

### No run store, no completion channel

The protocol mandates no run store, and it defines **no completion channel for `/start`**: how a caller learns that an asynchronous run finished — webhooks, polling routes, anything else — is implementation-defined and outside the protocol (see [Extension policy](#extension-policy)). A bare runner is not required to answer "what happened to run X?" after the fact. Clients written against the protocol alone must rely on `/execute`'s response.

## Validating a bundle

`POST /validate` takes `mthds_contents` (always an array, even for a single file) and an optional `allow_signatures` flag (default `false` — strict; when `true`, the validation sweep tolerates unimplemented pipe signatures by minting a mock).

`/validate` is a **diagnostic endpoint**: its job is to return a *verdict* about the submitted bundle, and every verdict it can produce — valid or invalid — rides a **200**, discriminated in the body on the mandatory `is_valid` field.

- **`is_valid: true`** — the bundle is valid. The protocol declares the `is_valid` discriminant plus the runnability facts (`is_runnable`, and `pending_signatures` — refs of pipes still declared as unimplemented signatures); implementations MAY include their own artifacts (parsed structures, graphs, anything else) as additional properties.
- **`is_valid: false`** — the bundle is invalid. The body carries `validation_errors[]` (a non-empty list of structured diagnostics, each at least a `category` and a `message`) plus `is_runnable: false` and an optional `message`. The structural artifacts of a valid report are absent.

A client pattern-matches `is_valid` to learn the verdict — it never inspects a status code or catches an exception body. Non-2xx is reserved for the cases where **no verdict could be produced**: a malformed request body is a `422` problem, auth a `401`/`403`, a server fault a `5xx`. So a non-2xx on `/validate` always means "the endpoint could not produce a verdict," never "your bundle is bad" — which keeps expected validation failures out of the 4xx error budget and never editorializes a verdict into a spurious retry. Signatures are never an error: an unimplemented signature reached during validation is a *runnability fact* (`is_runnable: false` + `pending_signatures`), not a validation failure, and `allow_signatures` only affects the dry-run sweep, not the verdict.

## Discovery

`GET /models` returns the runner's model deck — the models it can route to (`{name, type}` entries), optionally filtered by category. Implementations may add their own routing metadata (aliases, fallback chains, anything else) as additional properties.

`GET /version` is always public (no auth). It returns `protocol_version` (required) and an optional `runner_version` — implementations may add their own identification on top:

```json
{
  "protocol_version": "0.6.0",
  "runner_version": "2.3.0"
}
```

Clients use `/version` as the handshake: it reports the protocol and runner versions, and any additional properties let clients detect vendor extensions before relying on them.

## Extension policy

Implementations may extend the surface — extra routes, extra optional request properties, extra response properties — but **must not change the meaning or shape of the protocol routes**. Both sides of the wire are extension-open: request bodies accept implementation-defined args, and the protocol's response schemas declare only the base fields (`additionalProperties` allowed). A client written against the MTHDS Protocol runs unmodified against any compliant runner; a vendor's superset may accept and return more, but never diverges on the surface defined here.

## Conformance

An implementation claiming conformance states it as: *implements MTHDS Protocol v0.1*. Conformance means: the five routes exist with the request/response shapes of [`mthds-protocol.openapi.yaml`](openapi/mthds-protocol.openapi.yaml), errors are RFC 7807 problems, and `/version` is public.

## Route reference

Everything below is rendered at build time from the normative OpenAPI document — do not edit it by hand; edit [`mthds-protocol.openapi.yaml`](openapi/mthds-protocol.openapi.yaml).

<!--
This Markdown has been generated by essentials-openapi
https://github.com/Neoteroi/essentials-openapi

Most likely, it is not desirable to edit this file by hand!
-->


## Servers

<table>
    <thead>
        <tr>
            <th>Description</th>
            <th>URL</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Example — a self-hosted runner. The version segment belongs to the server base URL; protocol paths are version-agnostic.</td>
            <td>
                <a href="http://localhost:8081/v1" target="_blank" rel="noopener noreferrer">http://localhost:8081/v1</a>
            </td>
        </tr>
        <tr>
            <td>Example — a hosted, protocol-compliant superset.</td>
            <td>
                <a href="https://api.example.com/v1" target="_blank" rel="noopener noreferrer">https://api.example.com/v1</a>
            </td>
        </tr>
    </tbody>
</table>

## <span class="api-tag">run</span>


<hr class="operation-separator" />

### <span class="http-post">POST</span> /execute
Execute a method synchronously and return its full output.

??? note "Description"
    Blocking. The protocol sets no time limit; deployments cap it at their proxy layer. For long-running methods prefer /start. Implementations MAY return 202 + RunResultStart (id only) with a Location header pointing at an implementation-defined status resource when they cannot hold the connection open for the full execution (RFC 9110 asynchronous pattern); clients that cannot handle 202 should use /start instead.



**Input parameters**

<table>
    <thead>
        <tr>
            <th>Parameter</th>
            <th>In</th>
            <th>Type</th>
            <th>Default</th>
            <th>Nullable</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="parameter-name"><code>bearer</code></td>
            <td>header</td>
            <td>string</td>
            <td>N/A</td>
            <td>No</td>
            <td>Token semantics are implementation-defined.</td>
        </tr>
    </tbody>
</table>
<p class="request-body-title"><strong>Request body</strong></p>



=== "application/json"
    
    
    ```json
    {
        "pipe_code": "string",
        "mthds_contents": [
            "string"
        ],
        "inputs": {},
        "output_name": "string",
        "output_multiplicity": null,
        "dynamic_output_concept_ref": "string"
    }
    ```
    <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>

    

    ??? hint "Schema of the request body"
        ```json
        {
            "type": "object",
            "description": "At least one of pipe_code / mthds_contents is required (enforced by the anyOf rule below). If mthds_contents is provided without pipe_code, the first bundle must declare a main_pipe. Extension-open: implementations MAY accept extra top-level properties (extension args).\n",
            "additionalProperties": true,
            "anyOf": [
                {
                    "required": [
                        "pipe_code"
                    ],
                    "properties": {
                        "pipe_code": {
                            "type": "string",
                            "minLength": 1
                        }
                    }
                },
                {
                    "required": [
                        "mthds_contents"
                    ],
                    "properties": {
                        "mthds_contents": {
                            "type": "array",
                            "minItems": 1
                        }
                    }
                }
            ],
            "properties": {
                "pipe_code": {
                    "type": [
                        "string",
                        "null"
                    ],
                    "description": "Code of the pipe to execute (a pipe already registered, or one defined in mthds_contents)."
                },
                "mthds_contents": {
                    "type": [
                        "array",
                        "null"
                    ],
                    "minItems": 1,
                    "items": {
                        "type": "string"
                    },
                    "description": "MTHDS bundle contents to load (always an array, even for a single file; never empty). Implementations bound count and per-file size."
                },
                "inputs": {
                    "type": [
                        "object",
                        "null"
                    ],
                    "description": "Method inputs: map of input name to { concept, content }. Content shapes follow the concept's structure; content validation is deliberately loose here and strict inside the runtime.",
                    "additionalProperties": {
                        "type": "object",
                        "required": [
                            "concept",
                            "content"
                        ],
                        "properties": {
                            "concept": {
                                "type": "string"
                            },
                            "content": {}
                        }
                    }
                },
                "output_name": {
                    "type": [
                        "string",
                        "null"
                    ],
                    "description": "Name of the output slot to return as the main output."
                },
                "output_multiplicity": {
                    "oneOf": [
                        {
                            "type": "boolean"
                        },
                        {
                            "type": "integer"
                        },
                        {
                            "type": "null"
                        }
                    ],
                    "description": "Output multiplicity override (false/true or an explicit count)."
                },
                "dynamic_output_concept_ref": {
                    "type": [
                        "string",
                        "null"
                    ],
                    "description": "Override for the dynamic output concept reference."
                }
            }
        }
        ```

<p class="responses-title"><strong>Responses</strong></p>


=== "200 OK"
    
    === "application/json"
        
        
        ```json
        {
            "pipeline_run_id": "string",
            "pipe_output": {}
        }
        ```
        <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>
        
        

        ??? hint "Schema of the response body"
            ```json
            {
                "type": "object",
                "description": "POST /execute 200 — the completed run. Two base fields: the server-generated authoritative pipeline_run_id and the method's pipe_output (always present — a completed run has output). Extension-open: anything more an implementation returns (a run state, timestamps, output naming) is an extension on top.\n",
                "additionalProperties": true,
                "required": [
                    "pipeline_run_id",
                    "pipe_output"
                ],
                "properties": {
                    "pipeline_run_id": {
                        "type": "string",
                        "description": "The run identifier — server-generated and authoritative."
                    },
                    "pipe_output": {
                        "description": "The method's serialized output (working memory of serialized stuffs).",
                        "type": "object"
                    }
                }
            }
            ```
    
    
=== "202 Accepted"
    
    === "application/json"
        
        
        ```json
        {
            "pipeline_run_id": "string"
        }
        ```
        <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>
        
        

        ??? hint "Schema of the response body"
            ```json
            {
                "type": "object",
                "description": "POST /start 202 (and the optional /execute 202 degrade) — the started run's authoritative pipeline_run_id, nothing else. A started run has no output yet; how it is delivered later (polling, callbacks, anything else) is implementation-defined and outside the protocol. Extension-open.\n",
                "additionalProperties": true,
                "required": [
                    "pipeline_run_id"
                ],
                "properties": {
                    "pipeline_run_id": {
                        "type": "string",
                        "description": "The run identifier — server-generated and authoritative."
                    }
                }
            }
            ```
    
    

    **Response headers**

    | Name | Description | Schema |
    | --- | --- | --- |
    | `Location` | Implementation-defined status resource for this run. |<span class="string-type">string</span> |

    
=== "422 Unprocessable Content"
    <div class="common-response"><p>Refer to the common response description: <a href="#validationproblem" class="ref-link">ValidationProblem</a>.</p></div>
=== "Other responses"
    <div class="common-response"><p>Refer to the common response description: <a href="#problem" class="ref-link">Problem</a>.</p></div>


<hr class="operation-separator" />

### <span class="http-post">POST</span> /start
Start a method asynchronously; returns its pipeline_run_id immediately.

??? note "Description"
    Asynchronous. Returns 202 + RunResultStart immediately (pipeline_run_id only); the runner keeps no run store, and how completion is later delivered (callbacks, polling, anything else) is implementation-defined and outside the protocol. The returned pipeline_run_id is always authoritative (server-generated).



**Input parameters**

<table>
    <thead>
        <tr>
            <th>Parameter</th>
            <th>In</th>
            <th>Type</th>
            <th>Default</th>
            <th>Nullable</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="parameter-name"><code>bearer</code></td>
            <td>header</td>
            <td>string</td>
            <td>N/A</td>
            <td>No</td>
            <td>Token semantics are implementation-defined.</td>
        </tr>
    </tbody>
</table>
<p class="request-body-title"><strong>Request body</strong></p>



=== "application/json"
    
    
    ```json
    {
        "pipe_code": "string",
        "mthds_contents": [
            "string"
        ],
        "inputs": {},
        "output_name": "string",
        "output_multiplicity": null,
        "dynamic_output_concept_ref": "string"
    }
    ```
    <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>

    

    ??? hint "Schema of the request body"
        ```json
        {
            "type": "object",
            "description": "At least one of pipe_code / mthds_contents is required (enforced by the anyOf rule below). If mthds_contents is provided without pipe_code, the first bundle must declare a main_pipe. Extension-open: implementations MAY accept extra top-level properties (extension args).\n",
            "additionalProperties": true,
            "anyOf": [
                {
                    "required": [
                        "pipe_code"
                    ],
                    "properties": {
                        "pipe_code": {
                            "type": "string",
                            "minLength": 1
                        }
                    }
                },
                {
                    "required": [
                        "mthds_contents"
                    ],
                    "properties": {
                        "mthds_contents": {
                            "type": "array",
                            "minItems": 1
                        }
                    }
                }
            ],
            "properties": {
                "pipe_code": {
                    "type": [
                        "string",
                        "null"
                    ],
                    "description": "Code of the pipe to execute (a pipe already registered, or one defined in mthds_contents)."
                },
                "mthds_contents": {
                    "type": [
                        "array",
                        "null"
                    ],
                    "minItems": 1,
                    "items": {
                        "type": "string"
                    },
                    "description": "MTHDS bundle contents to load (always an array, even for a single file; never empty). Implementations bound count and per-file size."
                },
                "inputs": {
                    "type": [
                        "object",
                        "null"
                    ],
                    "description": "Method inputs: map of input name to { concept, content }. Content shapes follow the concept's structure; content validation is deliberately loose here and strict inside the runtime.",
                    "additionalProperties": {
                        "type": "object",
                        "required": [
                            "concept",
                            "content"
                        ],
                        "properties": {
                            "concept": {
                                "type": "string"
                            },
                            "content": {}
                        }
                    }
                },
                "output_name": {
                    "type": [
                        "string",
                        "null"
                    ],
                    "description": "Name of the output slot to return as the main output."
                },
                "output_multiplicity": {
                    "oneOf": [
                        {
                            "type": "boolean"
                        },
                        {
                            "type": "integer"
                        },
                        {
                            "type": "null"
                        }
                    ],
                    "description": "Output multiplicity override (false/true or an explicit count)."
                },
                "dynamic_output_concept_ref": {
                    "type": [
                        "string",
                        "null"
                    ],
                    "description": "Override for the dynamic output concept reference."
                }
            }
        }
        ```

<p class="responses-title"><strong>Responses</strong></p>


=== "202 Accepted"
    
    === "application/json"
        
        
        ```json
        {
            "pipeline_run_id": "string"
        }
        ```
        <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>
        
        

        ??? hint "Schema of the response body"
            ```json
            {
                "type": "object",
                "description": "POST /start 202 (and the optional /execute 202 degrade) — the started run's authoritative pipeline_run_id, nothing else. A started run has no output yet; how it is delivered later (polling, callbacks, anything else) is implementation-defined and outside the protocol. Extension-open.\n",
                "additionalProperties": true,
                "required": [
                    "pipeline_run_id"
                ],
                "properties": {
                    "pipeline_run_id": {
                        "type": "string",
                        "description": "The run identifier — server-generated and authoritative."
                    }
                }
            }
            ```
    
    
=== "422 Unprocessable Content"
    <div class="common-response"><p>Refer to the common response description: <a href="#validationproblem" class="ref-link">ValidationProblem</a>.</p></div>
=== "Other responses"
    <div class="common-response"><p>Refer to the common response description: <a href="#problem" class="ref-link">Problem</a>.</p></div>



## <span class="api-tag">validate</span>


<hr class="operation-separator" />

### <span class="http-post">POST</span> /validate
Parse, validate, and dry-run an MTHDS bundle.

??? note "Description"
    Diagnostic endpoint. Every verdict the validator can produce — valid or invalid — rides a 200, discriminated in the body on is_valid. Non-2xx is reserved for the cases where no verdict could be produced (a malformed request body, auth, a server fault), so a 422 here is a request-shape problem, never "the bundle is invalid".



**Input parameters**

<table>
    <thead>
        <tr>
            <th>Parameter</th>
            <th>In</th>
            <th>Type</th>
            <th>Default</th>
            <th>Nullable</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="parameter-name"><code>bearer</code></td>
            <td>header</td>
            <td>string</td>
            <td>N/A</td>
            <td>No</td>
            <td>Token semantics are implementation-defined.</td>
        </tr>
    </tbody>
</table>
<p class="request-body-title"><strong>Request body</strong></p>



=== "application/json"
    
    
    ```json
    {
        "mthds_contents": [
            "string"
        ],
        "allow_signatures": true
    }
    ```
    <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>

    

    ??? hint "Schema of the request body"
        ```json
        {
            "type": "object",
            "required": [
                "mthds_contents"
            ],
            "properties": {
                "mthds_contents": {
                    "type": "array",
                    "minItems": 1,
                    "items": {
                        "type": "string"
                    },
                    "description": "MTHDS contents to load (always an array, even for a single file)."
                },
                "allow_signatures": {
                    "type": "boolean",
                    "default": false,
                    "description": "When true, the validation sweep tolerates unimplemented pipe signatures (signatures dry-run by minting a mock). Strict by default."
                }
            }
        }
        ```

<p class="responses-title"><strong>Responses</strong></p>


=== "200 OK"
    
    === "application/json"
        
        

        ??? hint "Schema of the response body"
            ```json
            {
                "description": "The 200 response of POST /validate — a produced verdict discriminated on the mandatory is_valid field. A client pattern-matches is_valid to learn the verdict; it never inspects a status code or catches an exception body. Non-2xx is a no-verdict condition (a request-shape problem, auth, a server fault), never an invalid bundle.\n",
                "oneOf": [
                    {
                        "$ref": "#/components/schemas/ValidationReport"
                    },
                    {
                        "$ref": "#/components/schemas/InvalidValidationReport"
                    }
                ]
            }
            ```
    
    
=== "422 Unprocessable Content"
    <div class="common-response"><p>Refer to the common response description: <a href="#validationproblem" class="ref-link">ValidationProblem</a>.</p></div>
=== "Other responses"
    <div class="common-response"><p>Refer to the common response description: <a href="#problem" class="ref-link">Problem</a>.</p></div>



## <span class="api-tag">discovery</span>


<hr class="operation-separator" />

### <span class="http-get">GET</span> /models
The model deck available on this runner.

**Input parameters**

<table>
    <thead>
        <tr>
            <th>Parameter</th>
            <th>In</th>
            <th>Type</th>
            <th>Default</th>
            <th>Nullable</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td class="parameter-name"><code>bearer</code></td>
            <td>header</td>
            <td>string</td>
            <td>N/A</td>
            <td>No</td>
            <td>Token semantics are implementation-defined.</td>
        </tr>
        <tr>
            <td class="parameter-name"><code>type</code></td>
            <td>query</td>
            <td>string</td>
            <td></td>
            <td>No</td>
            <td>Filter the deck by model category.</td>
        </tr>
    </tbody>
</table><p class="responses-title"><strong>Responses</strong></p>


=== "200 OK"
    
    === "application/json"
        
        
        ```json
        {
            "models": [
                {
                    "name": "string",
                    "type": "llm"
                }
            ]
        }
        ```
        <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>
        
        

        ??? hint "Schema of the response body"
            ```json
            {
                "type": "object",
                "description": "The models this runner can route to. Implementations MAY add their own routing metadata (aliases, fallback chains, anything else) as additional properties — on the deck and on each model entry.\n",
                "additionalProperties": true,
                "properties": {
                    "models": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "additionalProperties": true,
                            "properties": {
                                "name": {
                                    "type": "string"
                                },
                                "type": {
                                    "type": "string",
                                    "enum": [
                                        "llm",
                                        "extract",
                                        "img_gen",
                                        "search"
                                    ]
                                }
                            }
                        }
                    }
                }
            }
            ```
    
    
=== "Other responses"
    <div class="common-response"><p>Refer to the common response description: <a href="#problem" class="ref-link">Problem</a>.</p></div>


<hr class="operation-separator" />

### <span class="http-get">GET</span> /version
Protocol and runner versions.<p class="responses-title"><strong>Responses</strong></p>


=== "200 OK"
    
    === "application/json"
        
        
        ```json
        {
            "protocol_version": "string",
            "runner_version": "string"
        }
        ```
        <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>
        
        

        ??? hint "Schema of the response body"
            ```json
            {
                "type": "object",
                "description": "The handshake. The protocol defines protocol_version (required) plus an optional runner_version; implementations MAY add their own identification (a name, an underlying runtime version, anything else) as additional properties.\n",
                "required": [
                    "protocol_version"
                ],
                "additionalProperties": true,
                "properties": {
                    "protocol_version": {
                        "type": "string",
                        "description": "MTHDS Protocol version implemented."
                    },
                    "runner_version": {
                        "type": [
                            "string",
                            "null"
                        ],
                        "description": "Version of the runner serving this protocol (optional)."
                    }
                }
            }
            ```
    
    
=== "Other responses"
    <div class="common-response"><p>Refer to the common response description: <a href="#problem" class="ref-link">Problem</a>.</p></div>




---
## Schemas


### InvalidValidationReport

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>is_runnable</code></td>
            <td><span class="boolean-type">boolean</span></td>
            <td>An invalid bundle is never runnable.</td>
        </tr>
        <tr>
            <td><code>is_valid</code></td>
            <td><span class="boolean-type">boolean</span></td>
            <td>Discriminant — the bundle is invalid.</td>
        </tr>
        <tr>
            <td><code>message</code></td>
            <td><span class="string-type">string</span></td>
            <td>Human-readable summary of the verdict.</td>
        </tr>
        <tr>
            <td><code>pending_signatures</code></td>
            <td>Array&lt;<span class="string-type">string</span>&gt;</td>
            <td>Outstanding signatures (best-effort; empty when no library could be assembled).</td>
        </tr>
        <tr>
            <td><code>validation_errors</code></td>
            <td>Array&lt;<a href="#validationerror" class="ref-link">ValidationError</a>&gt;</td>
            <td>Per-error diagnostics — non-empty on every invalid verdict.</td>
        </tr>
    </tbody>
</table>



### ModelDeck

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>models</code></td>
            <td>Array&lt;<em>Properties: </em><code>name, type</code>&gt;</td>
            <td></td>
        </tr>
    </tbody>
</table>



### Problem

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>detail</code></td>
            <td><span class="string-type">string</span></td>
            <td></td>
        </tr>
        <tr>
            <td><code>instance</code></td>
            <td><span class="string-type">string</span></td>
            <td></td>
        </tr>
        <tr>
            <td><code>status</code></td>
            <td><span class="integer-type">integer</span></td>
            <td></td>
        </tr>
        <tr>
            <td><code>title</code></td>
            <td><span class="string-type">string</span></td>
            <td></td>
        </tr>
        <tr>
            <td><code>type</code></td>
            <td><span class="string-type">string</span>(<span class="uri-format format">uri</span>)</td>
            <td></td>
        </tr>
    </tbody>
</table>



### RunRequest

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>dynamic_output_concept_ref</code></td>
            <td><span class="string-type">string</span> &#124; <span class="null-type">null</span></td>
            <td>Override for the dynamic output concept reference.</td>
        </tr>
        <tr>
            <td><code>inputs</code></td>
            <td></td>
            <td>Method inputs: map of input name to { concept, content }. Content shapes follow the concept's structure; content validation is deliberately loose here and strict inside the runtime.</td>
        </tr>
        <tr>
            <td><code>mthds_contents</code></td>
            <td>Array&lt;<span class="string-type">string</span>&gt;</td>
            <td>MTHDS bundle contents to load (always an array, even for a single file; never empty). Implementations bound count and per-file size.</td>
        </tr>
        <tr>
            <td><code>output_multiplicity</code></td>
            <td></td>
            <td>Output multiplicity override (false/true or an explicit count).</td>
        </tr>
        <tr>
            <td><code>output_name</code></td>
            <td><span class="string-type">string</span> &#124; <span class="null-type">null</span></td>
            <td>Name of the output slot to return as the main output.</td>
        </tr>
        <tr>
            <td><code>pipe_code</code></td>
            <td><span class="string-type">string</span> &#124; <span class="null-type">null</span></td>
            <td>Code of the pipe to execute (a pipe already registered, or one defined in mthds_contents).</td>
        </tr>
    </tbody>
</table>



### RunResultExecute

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>pipe_output</code></td>
            <td></td>
            <td>The method's serialized output (working memory of serialized stuffs).</td>
        </tr>
        <tr>
            <td><code>pipeline_run_id</code></td>
            <td><span class="string-type">string</span></td>
            <td>The run identifier — server-generated and authoritative.</td>
        </tr>
    </tbody>
</table>



### RunResultStart

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>pipeline_run_id</code></td>
            <td><span class="string-type">string</span></td>
            <td>The run identifier — server-generated and authoritative.</td>
        </tr>
    </tbody>
</table>



### ValidateRequest

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>allow_signatures</code></td>
            <td><span class="boolean-type">boolean</span></td>
            <td>When true, the validation sweep tolerates unimplemented pipe signatures (signatures dry-run by minting a mock). Strict by default.</td>
        </tr>
        <tr>
            <td><code>mthds_contents</code></td>
            <td>Array&lt;<span class="string-type">string</span>&gt;</td>
            <td>MTHDS contents to load (always an array, even for a single file).</td>
        </tr>
    </tbody>
</table>



### ValidationError

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>category</code></td>
            <td><span class="string-type">string</span></td>
            <td>Implementation-defined diagnostic category.</td>
        </tr>
        <tr>
            <td><code>message</code></td>
            <td><span class="string-type">string</span></td>
            <td></td>
        </tr>
    </tbody>
</table>



### ValidationReport

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>is_runnable</code></td>
            <td><span class="boolean-type">boolean</span></td>
            <td>Whether the validated library is complete enough to run — false when pipe signatures remain unimplemented (a runnability fact, not an error).
</td>
        </tr>
        <tr>
            <td><code>is_valid</code></td>
            <td><span class="boolean-type">boolean</span></td>
            <td>Discriminant — the bundle is valid.</td>
        </tr>
        <tr>
            <td><code>pending_signatures</code></td>
            <td>Array&lt;<span class="string-type">string</span>&gt;</td>
            <td>Refs of pipes still declared as unimplemented signatures.</td>
        </tr>
    </tbody>
</table>



### ValidationResult
<span>Type: </span>



### VersionInfo

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>protocol_version</code></td>
            <td><span class="string-type">string</span></td>
            <td>MTHDS Protocol version implemented.</td>
        </tr>
        <tr>
            <td><code>runner_version</code></td>
            <td><span class="string-type">string</span> &#124; <span class="null-type">null</span></td>
            <td>Version of the runner serving this protocol (optional).</td>
        </tr>
    </tbody>
</table>



## Common responses

This section describes common responses that are reused across operations.



### Problem
RFC 7807 problem document.

<p class="message-separator"></p>

=== "application/problem+json"
    
    
    ```json
    {
        "type": "string",
        "title": "string",
        "status": 0,
        "detail": "string",
        "instance": "string"
    }
    ```
    <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>

    

    ??? hint "Schema of the response body"
        ```json
        {
            "type": "object",
            "description": "RFC 7807.",
            "properties": {
                "type": {
                    "type": "string",
                    "format": "uri"
                },
                "title": {
                    "type": "string"
                },
                "status": {
                    "type": "integer"
                },
                "detail": {
                    "type": "string"
                },
                "instance": {
                    "type": "string"
                }
            }
        }
        ```






### ValidationProblem
The request failed validation (RFC 7807). On /execute and /start a bad bundle also lands here; on /validate an invalid bundle is a 200 verdict, so there a 422 is request-shape only.


<p class="message-separator"></p>

=== "application/problem+json"
    
    
    ```json
    {
        "type": "string",
        "title": "string",
        "status": 0,
        "detail": "string",
        "instance": "string"
    }
    ```
    <span class="small-note">⚠️</span>&nbsp;<em class="small-note warning">This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.</em>

    

    ??? hint "Schema of the response body"
        ```json
        {
            "type": "object",
            "description": "RFC 7807.",
            "properties": {
                "type": {
                    "type": "string",
                    "format": "uri"
                },
                "title": {
                    "type": "string"
                },
                "status": {
                    "type": "integer"
                },
                "detail": {
                    "type": "string"
                },
                "instance": {
                    "type": "string"
                }
            }
        }
        ```






## Security schemes

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Type</th>
            <th>Scheme</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        
        <tr>
            <td>bearer</td>
            <td>http</td>
            <td>bearer</td>
            <td>Token semantics are implementation-defined.</td>
        </tr>
        
    </tbody>
</table>

## Tags

| Name      | Description                                       |
| --------- | ------------------------------------------------- |
| run       | Execute methods, synchronously or asynchronously. |
| validate  | Static + dry-run validation of MTHDS bundles.     |
| discovery | What this runner is and what it can route to.     |




## CLI Reference

# CLI Reference

The `mthds` CLI is the official command-line tool for working with MTHDS packages. It covers validation, execution, and the full package management lifecycle.

## Core Commands

### `mthds validate`

Validate `.mthds` files, individual pipes, or an entire project.

**Usage:**

```
mthds validate <target>
mthds validate --bundle <file.mthds>
mthds validate --bundle <file.mthds> --pipe <pipe_code>
mthds validate --all
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `target` | A pipe code or a bundle file path (`.mthds`). Auto-detected based on file extension. |

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--pipe` | | Pipe code to validate. Optional when using `--bundle`. |
| `--bundle` | | Bundle file path (`.mthds`). Validates all pipes in the bundle. |
| `--all` | `-a` | Validate all pipes in all loaded libraries. |
| `--library-dir` | `-L` | Directory to search for `.mthds` files. Can be specified multiple times. |

**Examples:**

```bash
# Validate a single pipe by code
mthds validate extract_clause

# Validate a bundle file
mthds validate contract_analysis.mthds

# Validate a specific pipe within a bundle
mthds validate --bundle contract_analysis.mthds --pipe extract_clause

# Validate all pipes in the project
mthds validate --all
```

---

### `mthds run`

Execute a method. Loads the bundle, resolves dependencies, and runs the specified pipe.

**Usage:**

```
mthds run <target>
mthds run --bundle <file.mthds>
mthds run --bundle <file.mthds> --pipe <pipe_code>
mthds run <directory/>
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `target` | A pipe code, a bundle file path (`.mthds`), or a pipeline directory. Auto-detected. |

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--pipe` | | Pipe code to run. If omitted when using `--bundle`, runs the bundle's `main_pipe`. |
| `--bundle` | | Bundle file path (`.mthds`). |
| `--inputs` | `-i` | Path to a JSON file with input data. |
| `--output-dir` | `-o` | Base directory for all outputs. Default: `results`. |
| `--dry-run` | | Run in dry mode (no actual inference calls). |
| `--library-dir` | `-L` | Directory to search for `.mthds` files. Can be specified multiple times. |

**Examples:**

```bash
# Run a bundle's main pipe
mthds run joke_generation.mthds

# Run a specific pipe within a bundle
mthds run --bundle contract_analysis.mthds --pipe extract_clause

# Run with input data
mthds run extract_clause --inputs data.json

# Run a pipeline directory (auto-detects bundle and inputs)
mthds run pipeline_01/

# Dry run (no inference calls)
mthds run joke_generation.mthds --dry-run
```

When a directory is provided as the target, `mthds run` auto-detects the `.mthds` bundle file and an optional `inputs.json` file within it.

---

## Package Commands (`mthds pkg`)

Package commands manage the full lifecycle of MTHDS packages: initialization, dependencies, distribution, and discovery.

### `mthds pkg init`

Initialize a `METHODS.toml` package manifest from `.mthds` files in the current directory.

**Usage:**

```
mthds pkg init [--force]
```

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--force` | `-f` | Overwrite an existing `METHODS.toml`. |

The command scans all `.mthds` files recursively, extracts domain and pipe information, and generates a skeleton `METHODS.toml` with a placeholder address and auto-populated exports. Edit the generated file to set the correct address and refine exports.

**Example:**

```bash
mthds pkg init
# Created METHODS.toml with:
#   Domains: 2
#   Total pipes: 7
#   Bundles scanned: 3
#
# Edit METHODS.toml to set the correct address and configure exports.
```

---

### `mthds pkg list`

Display the package manifest for the current directory.

**Usage:**

```
mthds pkg list
```

Walks up from the current directory to find a `METHODS.toml` and displays its contents: package identity, dependencies, and exports.

---

### `mthds pkg add`

Add a dependency to `METHODS.toml`.

**Usage:**

```
mthds pkg add <address> [--alias NAME] [--version CONSTRAINT] [--path LOCAL_PATH]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `address` | Package address (e.g., `github.com/mthds/document-processing`). |

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--alias` | `-a` | Dependency alias. Auto-derived from the last path segment if not provided. |
| `--version` | `-v` | Version constraint. Default: `0.1.0`. |
| `--path` | `-p` | Local filesystem path to the dependency (for development). |

**Examples:**

```bash
# Add a remote dependency (alias auto-derived as "document_processing")
mthds pkg add github.com/mthds/document-processing --version "^1.0.0"

# Add with a custom alias
mthds pkg add github.com/acme/legal-tools --alias acme_legal --version "^0.3.0"

# Add a local development dependency
mthds pkg add github.com/team/scoring --path ../scoring-lib --version "^0.5.0"
```

---

### `mthds pkg lock`

Resolve dependencies and generate `methods.lock`.

**Usage:**

```
mthds pkg lock
```

Reads the `[dependencies]` section of `METHODS.toml`, resolves all versions (including transitive dependencies), and writes the lock file. The lock file records exact versions and SHA-256 integrity hashes for reproducible builds.

---

### `mthds pkg install`

Fetch and cache all dependencies from `methods.lock`.

**Usage:**

```
mthds pkg install
```

For each entry in the lock file, checks the local cache (`~/.mthds/packages/`). Missing packages are fetched via Git. After fetching, integrity hashes are verified against the lock file.

---

### `mthds pkg update`

Re-resolve dependencies to latest compatible versions and update `methods.lock`.

**Usage:**

```
mthds pkg update
```

Performs a fresh resolution of all dependencies (ignoring the existing lock file), writes the updated lock file, and displays a diff showing added, removed, and updated packages.

---

### `mthds pkg index`

Build and display the local package index.

**Usage:**

```
mthds pkg index [--cache]
```

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--cache` | `-c` | Index cached packages instead of the current project. |

Displays a summary table showing each package's address, version, description, and counts of domains, concepts, and pipes.

---

### `mthds pkg search`

Search the package index for concepts and pipes.

**Usage:**

```
mthds pkg search <query> [options]
mthds pkg search --accepts <concept> [--produces <concept>]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `query` | Search term (case-insensitive substring match). Optional if using `--accepts` or `--produces`. |

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--domain` | `-d` | Filter results to a specific domain. |
| `--concept` | | Show only matching concepts. |
| `--pipe` | | Show only matching pipes. |
| `--cache` | `-c` | Search cached packages instead of the current project. |
| `--accepts` | | Find pipes that accept this concept (type-compatible search). |
| `--produces` | | Find pipes that produce this concept (type-compatible search). |

**Examples:**

```bash
# Text search for concepts and pipes
mthds pkg search "contract"

# Search only pipes in a specific domain
mthds pkg search "extract" --pipe --domain legal.contracts

# Type-compatible search: "What can I do with a Document?"
mthds pkg search --accepts Document

# Type-compatible search: "What produces a NonCompeteClause?"
mthds pkg search --produces NonCompeteClause

# Combined: "What transforms Text into ScoreResult?"
mthds pkg search --accepts Text --produces ScoreResult
```

Type-compatible search uses the [Know-How Graph](../know-how-graph/index.md) to find pipes by their typed signatures. It understands concept refinement: searching for pipes that accept `Text` also finds pipes that accept `NonCompeteClause` (since `NonCompeteClause` refines `Text`).

---

### `mthds pkg inspect`

Display detailed information about a package.

**Usage:**

```
mthds pkg inspect <address> [--cache]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `address` | Package address to inspect. |

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--cache` | `-c` | Look in the package cache instead of the current project. |

Displays the package's metadata, domains, concepts (with structure fields and refinement), and pipe signatures (with inputs, outputs, and export status).

**Example:**

```bash
mthds pkg inspect github.com/acme/legal-tools
```

---

### `mthds pkg graph`

Query the Know-How Graph for concept and pipe relationships.

**Usage:**

```
mthds pkg graph --from <concept_id> [--to <concept_id>] [options]
mthds pkg graph --check <pipe_key_a>,<pipe_key_b>
```

**Options:**

| Option | Short | Description |
|--------|-------|-------------|
| `--from` | `-f` | Concept ID — find pipes that accept it. Format: `package_address::concept_ref`. |
| `--to` | `-t` | Concept ID — find pipes that produce it. |
| `--check` | | Two pipe keys comma-separated — check if the output of the first is compatible with an input of the second. |
| `--max-depth` | `-m` | Maximum chain depth when using `--from` and `--to` together. Default: `3`. |
| `--compose` | | Show an MTHDS composition template for discovered chains. Requires both `--from` and `--to`. |
| `--cache` | `-c` | Use cached packages instead of the current project. |

**Examples:**

```bash
# Find all pipes that accept a specific concept
mthds pkg graph --from "__native__::native.Document"

# Find all pipes that produce a specific concept
mthds pkg graph --to "github.com/acme/legal-tools::legal.contracts.NonCompeteClause"

# Find chains from Document to NonCompeteClause (auto-composition)
mthds pkg graph \
  --from "__native__::native.Document" \
  --to "github.com/acme/legal-tools::legal.contracts.NonCompeteClause"

# Same query, but generate an MTHDS snippet for the chain
mthds pkg graph \
  --from "__native__::native.Document" \
  --to "github.com/acme/legal-tools::legal.contracts.NonCompeteClause" \
  --compose

# Check if two pipes are compatible (can be chained)
mthds pkg graph --check "github.com/acme/legal-tools::extract_pages,github.com/acme/legal-tools::analyze_content"
```

When both `--from` and `--to` are provided, the command searches for multi-step pipe chains through the graph, up to `--max-depth` hops. With `--compose`, it generates a ready-to-use MTHDS `PipeSequence` snippet for each discovered chain.

---



## Editor Support

# Editor Support

The [Pipelex](https://marketplace.visualstudio.com/items?itemName=pipelex.pipelex) extension for VS Code-compatible editors (VS Code, Cursor, Windsurf, Antigravity, and others) provides syntax highlighting, semantic tokens, formatting, and validation for `.mthds` files. It is also available on [Open VSX](https://open-vsx.org/extension/Pipelex/pipelex). It is the recommended way to work with MTHDS.

## Installation

Install the **Pipelex** extension from the VS Code Marketplace:

1. Open your VS Code-compatible editor.
2. Go to Extensions (`Ctrl+Shift+X` / `Cmd+Shift+X`).
3. Search for **Pipelex**.
4. Click **Install**.

The extension activates automatically for `.mthds` files.

## Features

### Syntax Highlighting

The extension provides a full TextMate grammar for `.mthds` files, built on top of TOML highlighting. It recognizes MTHDS-specific constructs: pipe sections, concept sections, prompt templates, Jinja2 variables (`{{ }}`, `@variable`, `$variable`), and HTML content embedded in prompts.

Markdown code blocks tagged as `mthds` or `toml` also receive syntax highlighting when the extension is active.

### Semantic Tokens

Beyond TextMate grammar-based highlighting, the extension provides 7 semantic token types that distinguish MTHDS-specific elements:

| Token type | Applies to | Visual hint |
|------------|-----------|-------------|
| `mthdsConcept` | Concept names (e.g., `ContractClause`, `Text`) | Type color |
| `mthdsPipeType` | Pipe type values (e.g., `PipeLLM`, `PipeSequence`) | Type color, bold |
| `mthdsDataVariable` | Data variables in prompts | Variable color |
| `mthdsPipeName` | Pipe names in references | Function color |
| `mthdsPipeSection` | Pipe section headers (`[pipe.my_pipe]`) | Keyword color, bold |
| `mthdsConceptSection` | Concept section headers (`[concept.MyConcept]`) | Keyword color, bold |
| `mthdsModelRef` | Model field references (`$preset`, `@alias`) | Variable color, bold |

Semantic tokens are enabled by default. To toggle them:

- `pipelex.mthds.semanticTokens` — MTHDS-specific semantic tokens.
- `pipelex.syntax.semanticTokens` — TOML table/array key tokens.

### Formatting

The extension includes a built-in formatter for `.mthds` and `.toml` files. It uses the same engine as the `plxt` CLI (see [Formatting & Linting](formatting-linting.md)). Format on save works out of the box.

Formatting options are configurable in VS Code settings under `pipelex.formatter.*` (e.g., `alignEntries`, `columnWidth`, `trailingNewline`).

### Schema Validation

The extension supports JSON Schema-based validation and completion for TOML files. When the MTHDS JSON Schema is configured (see [MTHDS JSON Schema](json-schema.md)), the editor provides:

- Autocomplete suggestions for field names and values.
- Inline validation errors for invalid fields or types.
- Hover documentation for known fields.

Schema support is enabled by default (`pipelex.schema.enabled`).

### Additional Commands

The extension contributes several commands accessible via the Command Palette:

| Command | Description |
|---------|-------------|
| **TOML: Copy as JSON** | Copy selected TOML as JSON. |
| **TOML: Copy as TOML** | Copy selected text as TOML. |
| **TOML: Paste as JSON** | Paste clipboard content as JSON. |
| **TOML: Paste as TOML** | Paste clipboard content as TOML. |
| **TOML: Select Schema** | Choose a JSON Schema for the current TOML file. |


## Formatting & Linting

# Formatting & Linting

`plxt` is the CLI tool for formatting and linting `.mthds` and `.toml` files. It ensures consistent style across MTHDS projects.

## Installation

`plxt` is distributed as a standalone binary. Install it from the [`pipelex-tools` PyPI package](https://pypi.org/project/pipelex-tools/) (`uv tool install pipelex-tools`), or use the bundled version included with the [VS Code extension](https://marketplace.visualstudio.com/items?itemName=pipelex.pipelex).

## Formatting

Format `.mthds` and `.toml` files in place:

```bash
# Format all .mthds and .toml files in the current directory (recursive)
plxt format .

# Format a single file
plxt format contract_analysis.mthds

# Format and see what changed (check mode — exits non-zero if changes needed)
plxt format --check .
```

The `plxt format` command (also available as `plxt fmt`) aligns entries, normalizes whitespace, and ensures consistent TOML style. Files are modified in place.

## Linting

Lint `.mthds` and `.toml` files for structural issues:

```bash
# Lint all files in the current directory
plxt lint .

# Lint a single file
plxt lint contract_analysis.mthds
```

The `plxt lint` command checks for TOML structural issues and reports errors.

## Configuration

`plxt` reads its configuration from a `.pipelex/plxt.toml` file in the project root or a parent directory. This file controls formatting rules (alignment, column width, trailing commas, etc.) and can define per-file-type overrides.

A basic configuration:

```toml
[formatting]
align_entries      = true
column_width       = 100
trailing_newline   = true
array_trailing_comma = true
```

For the full list of configuration options, see the Pipelex documentation.

## Editor Integration

When the VS Code extension is installed, `plxt` formatting runs automatically on save. The extension uses the same formatting engine, so files formatted via CLI and editor produce identical results.


## MTHDS JSON Schema

# MTHDS JSON Schema

The MTHDS standard includes a machine-readable JSON Schema that describes the structure of `.mthds` files. Tools and editors can use this schema for validation, autocompletion, and documentation.

## What It Covers

The schema defines the complete structure of an `.mthds` bundle:

- **Header fields**: `domain`, `description`, `system_prompt`, `main_pipe`.
- **Concept definitions**: both simple (string) and structured forms, including `structure` fields, `refines`, and all field types (`text`, `integer`, `number`, `boolean`, `date`, `list`, `dict`, `concept`) and the `choices` enum mechanism.
- **Pipe definitions**: all pipe types with their specific fields — `PipeLLM`, `PipeStructure`, `PipeFunc`, `PipeImgGen`, `PipeExtract`, `PipeSearch`, `PipeCompose`, `PipeSequence`, `PipeParallel`, `PipeCondition`, `PipeBatch`.
- **Sub-pipe blueprints**: the `steps`, `branches`, `outcomes`, and `construct` structures used by controllers and PipeCompose.
- **Inline model settings**: the `LLMSetting`, `ImgGenSetting`, `ExtractSetting`, and `SearchSetting` objects that can be used in place of string model references.

## Schema Version

The schema is auto-generated from the MTHDS bundle data model. The current version is noted in the schema's `$comment` field. The hosted schema always corresponds to the latest released version of the MTHDS standard.

## Where to Find It

The schema is distributed with the tools that use it:

- **VS Code extension** — the Pipelex extension bundles the schema and uses it for autocompletion and inline validation.
- **`plxt` CLI** — the `plxt` binary includes the schema for local validation.
- **`pipelex-tools` PyPI package** — the schema is included in the Python distribution.

The schema is also hosted at a stable URL for direct use by editors and other tooling:

**Hosted URL:** [`https://mthds.ai/mthds_schema.json`](https://mthds.ai/mthds_schema.json)

## How to Use It

### With the VS Code Extension

The Pipelex VS Code extension includes the schema and uses it automatically for autocompletion and inline validation of `.mthds` files. No configuration is required.

### With Other Editors

Any editor that supports JSON Schema for TOML can use the MTHDS schema. Configure your editor's TOML language server to associate `.mthds` files with the schema URL `https://mthds.ai/mthds_schema.json`.

### For Tooling

The schema can be used programmatically for:

- Building custom validators for `.mthds` files.
- Generating documentation from the schema structure.
- Implementing autocompletion in non-VS Code editors.

For detailed guidance on building editor support, see [For Implementers: Building Editor Support](../implementers/editor-support.md).


# Guides

## Write Your First Method

# Write Your First Method

This guide walks you through creating a working `.mthds` file from scratch. By the end, you will have a method that generates a short summary from a text input.

## Prerequisites

- A text editor with MTHDS support. Install the [VS Code extension](../tooling/editor-support.md) for the best experience.
- The `plxt` CLI installed for formatting (see [Formatting & Linting](../tooling/formatting-linting.md)).
- The `mthds` CLI installed for validation.

## Step 1: Create a `.mthds` File

Create a new file called `summarizer.mthds` and add a domain header:

```toml
domain      = "summarization"
description = "Text summarization methods"
```

Every bundle starts with a `domain` — a namespace for the concepts and pipes you will define. The domain name uses `snake_case` segments separated by dots.

## Step 2: Define a Concept

Add a concept to describe the kind of data your method produces:

```toml
domain      = "summarization"
description = "Text summarization methods"

[concept]
Summary = "A concise summary of a longer text"
```

This declares a simple concept called `Summary`. It has no internal structure — it is a semantic label that gives meaning to the data your pipe produces.

Concept codes use `PascalCase` (e.g., `Summary`, `ContractClause`, `CandidateProfile`).

## Step 3: Define a Pipe

Add a pipe that takes text input and produces a summary:

```toml
domain      = "summarization"
description = "Text summarization methods"
main_pipe   = "summarize"

[concept]
Summary = "A concise summary of a longer text"

[pipe.summarize]
type        = "PipeLLM"
description = "Summarize the input text in 2-3 sentences"
inputs      = { text = "Text" }
output      = "Summary"
prompt      = """
Summarize the following text in 2-3 concise sentences. Focus on the key points.

@text
"""
```

Here is what each field does:

- `type = "PipeLLM"` — this pipe uses a large language model to generate output.
- `inputs = { text = "Text" }` — the pipe accepts one input called `text`, of the native `Text` type.
- `output = "Summary"` — the pipe produces a `Summary` concept.
- `prompt` — the LLM prompt template. `@text` is shorthand for `{{ text }}`, injecting the input variable.

The `main_pipe = "summarize"` header marks this pipe as the bundle's primary entry point.

## Step 4: Format Your File

Run the formatter to ensure consistent style:

```bash
plxt fmt summarizer.mthds
```

The formatter aligns entries, normalizes whitespace, and ensures your file follows MTHDS style conventions.

## Step 5: Validate

Validate your bundle:

```bash
mthds validate summarizer.mthds
```

If everything is correct, you will see a success message. If there are errors — a misspelled concept reference, an unused input, a missing required field — the validator reports them with specific messages.

## The Complete File

```toml
domain      = "summarization"
description = "Text summarization methods"
main_pipe   = "summarize"

[concept]
Summary = "A concise summary of a longer text"

[pipe.summarize]
type        = "PipeLLM"
description = "Summarize the input text in 2-3 sentences"
inputs      = { text = "Text" }
output      = "Summary"
prompt      = """
Summarize the following text in 2-3 concise sentences. Focus on the key points.

@text
"""
```

This file works as a standalone bundle — no manifest, no package, no dependencies. To run it:

```bash
mthds run summarizer.mthds
```

!!! note "Runtime required"
    Execution requires an MTHDS-compatible runtime. The reference runtime is [Pipelex](https://github.com/Pipelex/pipelex).

## File Naming Conventions

When organizing `.mthds` files in a project:

- Use `snake_case` for file names: `invoice_processing.mthds`, `cv_analysis.mthds`.
- Match the file name to the bundle's domain when practical. A bundle with `domain = "invoice_processing"` lives naturally in `invoice_processing.mthds`.
- Use the `.mthds` extension — it is required by the toolchain for validation and formatting.

## Next Steps

- Add more concepts and pipes to your bundle. See [The Language](../language/bundles.md) for the full set of pipe types and concept features.
- When you are ready to distribute your methods, see [Create a Package](../guides/create-package.md).


## Create a Package

# Create a Package

This guide walks you through turning a standalone bundle into a distributable MTHDS package.

## What You Start With

You have one or more `.mthds` files that work on their own:

```
my-methods/
├── summarizer.mthds
└── classifier.mthds
```

## Step 1: Initialize the Manifest

Run `mthds pkg init` from the package directory:

```bash
cd my-methods
mthds pkg init
```

This scans all `.mthds` files, extracts domains and pipe names, and generates a `METHODS.toml` skeleton:

```toml
[package]
address     = "example.com/yourorg/my_methods"
version     = "0.1.0"
description = "Package generated from 2 .mthds file(s)"

[exports.summarization]
pipes = ["summarize"]

[exports.classification]
pipes = ["classify_document"]
```

## Step 2: Set the Package Address

Edit the `address` field to your actual repository location:

```toml
[package]
address     = "github.com/yourorg/my-methods"
version     = "0.1.0"
description = "Text summarization and document classification methods"
```

The address must start with a hostname (containing at least one dot), followed by a path. It doubles as the fetch location when other packages depend on yours.

## Step 3: Configure Exports

Review the `[exports]` section. The generated manifest exports all pipes found during scanning. Narrow it down to your public API:

```toml
[exports.summarization]
pipes = ["summarize"]

[exports.classification]
pipes = ["classify_document"]
```

Pipes not listed in `[exports]` are private — they are implementation details invisible to consumers. Pipes declared as `main_pipe` in a bundle header are auto-exported regardless of whether they appear here.

Concepts are always public — they do not need to be listed.

## Step 4: Add Metadata

Add optional but recommended fields:

```toml
[package]
address       = "github.com/yourorg/my-methods"
version       = "0.1.0"
description   = "Text summarization and document classification methods"
authors       = ["Your Name <you@example.com>"]
license       = "MIT"
mthds_version = ">=1.0.0"
```

## Step 5: Validate

Verify your package is well-formed:

```bash
mthds validate --all
```

This validates all pipes across all bundles in the package, checking concept references, pipe references, and visibility rules.

## The Result

Your package directory now looks like:

```
my-methods/
├── METHODS.toml
├── summarizer.mthds
└── classifier.mthds
```

You have a distributable package with a globally unique address, versioned identity, and controlled exports. Other packages can now depend on it.

## See Also

- [The Manifest](../packages/manifest.md) — full reference for `METHODS.toml` fields.
- [Exports & Visibility](../packages/exports-visibility.md) — how visibility rules work.
- [Use Dependencies](use-dependencies.md) — how to depend on other packages.


## Use Dependencies

# Use Dependencies

This guide shows how to add dependencies on other MTHDS packages and use their concepts and pipes in your bundles.

## Step 1: Add a Dependency

Use `mthds pkg add` to add a dependency to your `METHODS.toml`:

```bash
mthds pkg add github.com/mthds/document-processing --version "^1.0.0"
```

This adds an entry to the `[dependencies]` section:

```toml
[dependencies]
document_processing = { address = "github.com/mthds/document-processing", version = "^1.0.0" }
```

The alias (`document_processing`) is auto-derived from the last segment of the address. To choose a shorter alias:

```bash
mthds pkg add github.com/mthds/document-processing --alias docproc --version "^1.0.0"
```

```toml
[dependencies]
docproc = { address = "github.com/mthds/document-processing", version = "^1.0.0" }
```

## Step 2: Resolve and Lock

Generate the lock file to pin exact versions:

```bash
mthds pkg lock
```

Then install the dependencies into the local cache:

```bash
mthds pkg install
```

## Step 3: Use Cross-Package References

In your `.mthds` files, reference the dependency's concepts and pipes using the `->` syntax:

```toml
domain = "analysis"

[pipe.analyze_document]
type        = "PipeSequence"
description = "Extract pages from a document and analyze them"
inputs      = { document = "Document" }
output      = "AnalysisResult"
steps = [
    { pipe = "docproc->extraction.extract_text", result = "pages" },
    { pipe = "process_pages", result = "analysis" },
]
```

The reference `docproc->extraction.extract_text` reads as: "from the package aliased as `docproc`, get the pipe `extract_text` in the `extraction` domain."

Cross-package concept references work the same way:

```toml
[concept.DetailedPage]
description = "An enriched page with additional metadata"
refines     = "docproc->extraction.ExtractedPage"
```

## Step 4: Validate

```bash
mthds validate --all
```

Validation checks that:

- The alias `docproc` exists in `[dependencies]`.
- The pipe `extract_text` exists in the `extraction` domain of the resolved dependency.
- The pipe is exported by the dependency (listed in its `[exports]` or declared as `main_pipe`).

## Using Local Path Dependencies

During development, you can point a dependency to a local directory instead of fetching it remotely:

```bash
mthds pkg add github.com/mthds/document-processing --path ../document-processing --version "^1.0.0"
```

```toml
[dependencies]
docproc = { address = "github.com/mthds/document-processing", version = "^1.0.0", path = "../document-processing" }
```

Local path dependencies are resolved from the filesystem at load time. They are not resolved transitively and are excluded from the lock file.

## Updating Dependencies

To update all dependencies to their latest compatible versions:

```bash
mthds pkg update
```

This performs a fresh resolution, writes an updated `methods.lock`, and shows a diff of what changed.

## See Also

- [Dependencies](../packages/dependencies.md) — full reference for dependency fields and version constraints.
- [Cross-Package References](../packages/cross-package-references.md) — the `->` syntax explained.
- [Version Resolution](../packages/version-resolution.md) — how Minimum Version Selection works.


## Discover Methods

# Discover Methods

This guide shows how to search for and discover existing MTHDS methods — by text, by domain, or by typed signature.

## Searching by Text

The simplest search is a text query:

```bash
mthds pkg search "contract"
```

This searches concepts and pipes for the term "contract" (case-insensitive substring match) and displays matching results in tables showing package, name, domain, description, and export status.

To narrow results:

```bash
# Show only concepts
mthds pkg search "contract" --concept

# Show only pipes
mthds pkg search "contract" --pipe

# Filter by domain
mthds pkg search "extract" --domain legal.contracts
```

## Searching by Type ("I Have X, I Need Y")

MTHDS enables something that text-based discovery cannot: **type-compatible search**. Instead of searching by name, you search by what data types a pipe accepts or produces.

### "What can I do with X?"

Find all pipes that accept a given concept:

```bash
mthds pkg search --accepts Document
```

This returns every pipe whose input type is `Document` or a concept that `Document` refines. Because the search understands the concept refinement hierarchy, it finds pipes you might not discover through text search alone.

### "What produces Y?"

Find all pipes that produce a given concept:

```bash
mthds pkg search --produces NonCompeteClause
```

### Combining Accepts and Produces

Find pipes that bridge two types:

```bash
mthds pkg search --accepts Document --produces NonCompeteClause
```

## Exploring the Know-How Graph

For more advanced queries — multi-step chains, compatibility checks, auto-composition — use the `mthds pkg graph` command.

### Finding Chains

When no single pipe transforms X into Y, the graph can find multi-step chains:

```bash
mthds pkg graph \
  --from "__native__::native.Document" \
  --to "github.com/acme/legal-tools::legal.contracts.NonCompeteClause"
```

This might discover a chain like:

```
1. extract_pages -> analyze_content -> extract_clause
```

With `--compose`, it generates a ready-to-use MTHDS snippet:

```bash
mthds pkg graph \
  --from "__native__::native.Document" \
  --to "github.com/acme/legal-tools::legal.contracts.NonCompeteClause" \
  --compose
```

### Checking Compatibility

Before wiring two pipes together, verify they are type-compatible:

```bash
mthds pkg graph --check "pkg_a::extract_pages,pkg_a::analyze_content"
```

This reports whether the output of the first pipe matches any input of the second.

## Searching Cached Packages

By default, search and graph commands operate on the current project. To search across all cached packages (everything you have installed):

```bash
mthds pkg search "scoring" --cache
mthds pkg graph --from "__native__::native.Text" --cache
```

## Inspecting a Package

To see the full contents of a specific package — its domains, concepts, and pipe signatures:

```bash
mthds pkg inspect github.com/acme/legal-tools
```

This displays detailed tables for every domain, concept (including structure fields and refinement), and pipe (including inputs, outputs, and export status).

## Building the Index

Before searching, you may want to build or refresh the package index:

```bash
# Index the current project
mthds pkg index

# Index all cached packages
mthds pkg index --cache
```

The index is built automatically when you run search or graph commands, but building it explicitly lets you verify what packages are available.

## See Also

- [The Know-How Graph](../know-how-graph/index.md) — how typed signatures enable semantic discovery.
- [Cross-Package References](../packages/cross-package-references.md) — how to use discovered pipes in your bundles.
- [Use Dependencies](use-dependencies.md) — how to add a discovered package as a dependency.
- [The Registry](../packages/registry.md) — query remote registries for packages beyond your local cache.
- [Registry Search](../packages/registry-search.md) — type-aware search semantics and concept compatibility rules.


# For Implementers

## Building a Runtime

# Building a Runtime

This page describes how to build a runtime that loads, validates, and executes MTHDS bundles and packages. The specification defines *what* must hold; this page describes *how* the reference implementation achieves it, as guidance for alternative implementations.

## High-Level Architecture

A compliant MTHDS runtime has four main subsystems:

1. **Parser** — reads `.mthds` TOML files into an in-memory bundle model.
2. **Loader** — discovers manifests, resolves dependencies, assembles a library of bundles.
3. **Validator** — checks all structural, naming, reference, and visibility rules.
4. **Executor** — runs pipes by dispatching to operator backends (LLM, function, image generation, extraction, composition) and orchestrating controllers.

The first three are specified by the standard; the fourth is implementation-specific (the standard defines *what* a pipe does, not *how*).

## Parsing .mthds Files

A `.mthds` file is valid TOML. Parse it with any compliant TOML parser, then validate the resulting structure against the MTHDS data model.

**Recommended approach:**

1. Parse the TOML into a generic dictionary.
2. Extract header fields (`domain`, `description`, `system_prompt`, `main_pipe`).
3. Extract the `concept` table — a mix of simple declarations (string values) and structured declarations (sub-tables with `description`, `structure`, `refines`).
4. Extract `pipe` sub-tables. Each pipe has a `type` field that determines the discriminated union variant (one of the supported pipe types).
5. Validate all fields against the rules in the [Specification](../spec/mthds-format.md).

The reference implementation uses Pydantic's discriminated union on the `type` field to dispatch pipe parsing:

```
PipeBlueprintUnion = PipeFuncBlueprint
                   | PipeImgGenBlueprint
                   | PipeComposeBlueprint
                   | PipeLLMBlueprint
                   | PipeStructureBlueprint
                   | PipeExtractBlueprint
                   | PipeSearchBlueprint
                   | PipeBatchBlueprint
                   | PipeConditionBlueprint
                   | PipeParallelBlueprint
                   | PipeSequenceBlueprint
```

This means an invalid `type` value is rejected at parse time, before any field-level validation occurs.

## Manifest Discovery

When loading a bundle, the runtime must locate the package manifest (`METHODS.toml`) by walking up the directory tree:

```
function find_manifest(bundle_path):
    current = parent_directory(bundle_path)
    while true:
        if "METHODS.toml" exists in current:
            return parse_manifest(current / "METHODS.toml")
        if ".git" directory exists in current:
            return null  // stop at repository boundary
        parent = parent_directory(current)
        if parent == current:
            return null  // filesystem root
        current = parent
```

If no manifest is found, the bundle is treated as a standalone bundle: all pipes are public, no dependencies are available beyond native concepts, and the bundle is not distributable.

## Loading a Package

Loading a package involves these steps in order:

1. **Parse the manifest** — read `METHODS.toml` and validate all fields (address, version, dependencies, exports). Reject immediately on any parse or validation error.
2. **Discover bundles** — recursively find all `.mthds` files under the package root.
3. **Parse all bundles** — parse each `.mthds` file into a bundle blueprint. Collect parse errors.
4. **Resolve dependencies** — for each dependency in the manifest:
    - If it has a `path` field, resolve from the local filesystem (non-transitive).
    - If it is remote, resolve via VCS (transitive, with cycle detection and diamond handling).
5. **Build the library** — assemble all parsed bundles (local and dependency) into a library structure indexed by domain and package.
6. **Validate references** — check that all concept and pipe references resolve correctly, following the [Namespace Resolution Rules](../spec/namespace-resolution.md).
7. **Validate visibility** — check that cross-domain and cross-package pipe references respect export rules.

## Working Memory

Controllers orchestrate pipes through **working memory** — a key-value store that accumulates results as a pipeline executes.

When a `PipeSequence` runs, each step's output is stored under its `result` name. Subsequent steps can consume any previously stored value. The final step's output (or the value matching the sequence's `output` concept) becomes the sequence's output.

Working memory is scoped to a pipeline execution. Each top-level `mthds run` invocation starts with a fresh working memory containing only the declared inputs.

## Concept Refinement at Runtime

Concept refinement establishes a type-compatibility relationship. When a pipe declares `inputs = { doc = "ContractClause" }`, any concept that refines `ContractClause` (directly or transitively) is an acceptable input.

A runtime must build and query a refinement graph:

```
function is_compatible(actual_concept, expected_concept):
    if actual_concept == expected_concept:
        return true
    if actual_concept is a native concept and expected_concept == "Anything":
        return true
    parent = refinement_parent(actual_concept)
    if parent is null:
        return false
    return is_compatible(parent, expected_concept)
```

The refinement graph is built during loading by following `refines` fields across all loaded concepts (including cross-package refinements).

## Output Validation

A compliant runtime validates the output of every pipe against the declared output concept's structure at every intermediate step — not just at the method level. This ensures that errors surface at the step that produces incorrect output, not downstream where the symptoms are harder to trace.

**Recommended approach:**

1. After a pipe produces output, resolve the output concept's definition (including its `structure` fields if any).
2. Validate the produced data against the concept's type and field constraints — required fields, field types (`text`, `integer`, `boolean`, `list`, `dict`, `number`, `date`, `concept`), and any `choices` enums.
3. If validation fails, report the error with the pipe code and step index, and halt execution of the current pipeline.

Validation libraries such as Pydantic (Python) or Zod (TypeScript) are natural fits for implementing these checks. Beyond mapping MTHDS concept structures to schema definitions, these libraries also support custom validation logic — expressed in Python or TypeScript — that goes beyond what the MTHDS standard defines.

## Model References

MTHDS defines several forms of model reference (`$` preset, `@` alias, `~` waterfall, and bare handle) that method authors use in the `model` field of `PipeLLM`, `PipeStructure`, `PipeImgGen`, `PipeExtract`, and `PipeSearch`. See [Model References](../language/model-references.md) for the full description and examples.

A runtime must resolve each form to a concrete model configuration. The recommended approach:

1. **Parse the prefix** — inspect the first character of the `model` string to determine the reference kind (`$`, `@`, `~`, or no prefix).
2. **Look up in a registry** — resolve the name (after stripping the prefix) against the appropriate registry:
    - `$` → preset registry (returns a model handle plus parameters such as temperature, max tokens, quality).
    - `@` → alias registry (returns a model handle).
    - `~` → waterfall registry (returns an ordered list of model handles to try in sequence).
    - No prefix → treat the string as a direct model handle.
3. **Return the model configuration** — pass the resolved handle (and any associated parameters) to the operator backend.

A compliant runtime may implement model references differently — or not at all, treating the `model` field as a direct model identifier. The standard requires only that the field be a string.

## Template Blueprint (Advanced PipeCompose)

When the `template` field of a `PipeCompose` pipe is a table (rather than a plain string), it is a **template blueprint** with additional rendering options:

| Field | Type | Description |
|-------|------|-------------|
| `template` | string | The Jinja2 template source. Required. |
| `category` | string | Determines which Jinja2 filters and rendering rules apply. Values: `basic`, `expression`, `html`, `markdown`, `mermaid`, `llm_prompt`, `img_gen_prompt`. |
| `templating_style` | object or null | Controls tag style and text formatting during rendering. |
| `extra_context` | object or null | Additional variables injected into the template rendering context beyond the pipe's declared inputs. |

The `category` field influences which Jinja2 filters are available. For example, `html` templates get HTML-specific filters, while `llm_prompt` templates get prompt-specific filters. The reference implementation registers different filter sets per category.

**Shorthand preprocessor:** A compliant runtime SHOULD expand the `$`, `@`, and `@?` shorthand patterns into their Jinja2 equivalents before template rendering. See the [specification](../spec/mthds-format.md#template-mode) for the normative expansion rules. The preprocessor runs on the `template` field of PipeCompose, the `prompt` and `system_prompt` fields of PipeLLM, and the `prompt` field of PipeImgGen and PipeSearch.

**Filter registration per category:** The reference implementation registers the following Jinja2 filters based on the template category:

| Category | Registered Filters |
|----------|-------------------|
| `basic` | `format`, `tag` |
| `expression` | *(none)* |
| `html` | `format`, `tag`, `escape_script_tag` |
| `markdown` | `format`, `tag`, `escape_script_tag` |
| `mermaid` | *(none)* |
| `llm_prompt` | `format`, `tag`, `with_images` |
| `img_gen_prompt` | `format`, `tag`, `with_images` |

See [Pipes — Operators: Template Mode](../language/pipes-operators.md#template-mode) for the user-facing reference on template categories, filters, and shorthand syntax.

A compliant runtime must support the plain string form of `template`. The table form with `category`, `templating_style`, and `extra_context` is an advanced feature that implementations may support progressively.

## Exposing a Runner over HTTP

A runtime becomes a network-accessible **runner** by implementing the [MTHDS Protocol](../spec/protocol.md) — the minimal HTTP contract of five routes: `POST /execute`, `POST /start`, `POST /validate`, `GET /models`, `GET /version`. The normative artifact is the protocol's [OpenAPI document](../spec/openapi/mthds-protocol.openapi.yaml).

The protocol deliberately excludes everything that is not "run a method": no run store, no completion channel for async runs (webhooks or polling are implementation extensions), no users, no billing, no catalog. Implementations may extend the surface with extra routes or optional request properties, but must not change the protocol routes' shapes — see the [extension policy](../spec/protocol.md#extension-policy).

The reference implementation is `pipelex-api` (MIT, self-hostable Docker image), which implements the protocol and extends it with stateless `/build/*` authoring helpers.

`/validate` is a diagnostic endpoint: a produced verdict — valid or invalid — always rides a 200, discriminated on `is_valid` (the protocol also declares the runnability facts `is_runnable` / `pending_signatures`; a non-2xx means *no verdict could be produced*, not an invalid bundle). Beyond those declared fields a runner returns its own verification artifacts as implementation-defined additional properties on the valid arm. The reference implementation (Pipelex) returns `pipe_io_contracts` — a map keyed by namespaced `pipe_ref` (`<domain>.<pipe_code>`) giving each pipe's resolved input and output contracts (the concept and multiplicity of every input and of the output). The map contains entries only for pipes defined in the validated package, not its transitive dependencies (whose pipes resolve under the package-qualified `alias->domain.pipe_code` form), so the `<domain>.<pipe_code>` keys cannot collide across packages. The name is documented here only to encourage convergence among implementations; it is not part of the protocol, and other runners may expose their own artifacts under their own keys. On the invalid arm (`is_valid: false`) the protocol requires a non-empty `validation_errors[]` (one diagnostic per failure); each item's `category` is implementation-defined — the reference implementation (Pipelex) narrows it to its own closed vocabulary.


## Validation Rules

# Validation Rules

This page consolidates all validation rules from the [Specification](../spec/mthds-format.md) into an ordered checklist for implementers. Rules are grouped by the stage at which they should be enforced.

## Stage 1: TOML Parsing

Before any MTHDS-specific validation, the file must be valid TOML.

- The file MUST be valid UTF-8-encoded TOML.
- A `.mthds` file MUST have the `.mthds` extension.
- `METHODS.toml` MUST be named exactly `METHODS.toml`.
- `methods.lock` MUST be named exactly `methods.lock`.

## Stage 2: Bundle Structural Validation

After parsing TOML into a dictionary, validate the bundle structure:

1. `domain` MUST be present.
2. `domain` MUST be a valid domain code: one or more `snake_case` segments (`[a-z][a-z0-9_]*`) separated by `.`.
3. `main_pipe`, if present, MUST be `snake_case` and MUST reference a pipe defined in the same bundle.
4. Concept codes MUST be `PascalCase` (`[A-Z][a-zA-Z0-9]*`).
5. Concept codes MUST NOT match any native concept code (`Dynamic`, `Text`, `Image`, `Document`, `Html`, `TextAndImages`, `Number`, `Page`, `JSON`, `SearchResult`, `Anything`).
6. Pipe codes MUST be `snake_case` (`[a-z][a-z0-9_]*`).
7. `refines` and `structure` MUST NOT both be set on the same concept.

## Stage 3: Concept Field Validation

For each field in a concept's `structure`:

1. `description` MUST be present.
2. If `type` is omitted, `choices` MUST be non-empty.
3. `type = "dict"` requires both `key_type` and `value_type`.
4. `type = "concept"` requires `concept_ref` and forbids `default_value`.
5. `type = "list"` with `item_type = "concept"` requires `item_concept_ref`.
6. `concept_ref` MUST NOT be set unless `type = "concept"`.
7. `item_concept_ref` MUST NOT be set unless `item_type = "concept"`.
8. `default_value` type MUST match the declared `type`.
9. If `choices` is set and `default_value` is present, `default_value` MUST be in `choices`.
10. Field names MUST NOT start with `_`.

## Stage 4: Pipe Type-Specific Validation

Each pipe type has specific rules:

**PipeLLM:**

- All prompt and system_prompt variables MUST have matching inputs.
- All inputs MUST be referenced in prompt or system_prompt.

**PipeStructure:**

- `inputs` MUST contain exactly one entry.
- The single input concept MUST be `Text` or a concept that refines `Text`.
- `output` MUST NOT be `Text` and MUST NOT be a concept that refines `Text`.
- `output` MAY use multiplicity (`Foo`, `Foo[]`, `Foo[N]`).

**PipeFunc:**

- `function_name` MUST be present and non-empty.

**PipeImgGen:**

- `prompt` MUST be present.
- All prompt variables MUST have matching inputs.

**PipeExtract:**

- `inputs` MUST contain exactly one entry.
- `output` MUST be `"Page[]"`.

**PipeSearch:**

- `prompt` MUST be present.
- All prompt variables MUST have matching inputs.
- `output` MUST be `SearchResult` or a concept that refines `SearchResult`.

**PipeCompose:**

- Exactly one of `template` or `construct` MUST be present.
- `output` MUST NOT use multiplicity brackets (`[]` or `[N]`).
- All template/construct variables MUST have matching inputs.

**PipeSequence:**

- `steps` MUST have at least one entry.
- `nb_output` and `multiple_output` MUST NOT both be set on the same step.
- `batch_over` and `batch_as` MUST either both be present or both be absent.
- `batch_over` and `batch_as` MUST NOT be the same value.

**PipeParallel:**

- At least one of `add_each_output` or `combined_output` MUST be set.

**PipeCondition:**

- Exactly one of `expression_template` or `expression` MUST be present.
- `outcomes` MUST have at least one entry.

**PipeBatch:**

- `input_list_name` MUST be in `inputs`.
- `input_item_name` MUST NOT be empty.
- `input_item_name` MUST NOT equal `input_list_name`.
- `input_item_name` MUST NOT equal any key in `inputs`.

## Stage 5: Reference Validation (Bundle-Level)

Within a single bundle:

- Bare concept references MUST resolve to: a native concept, a concept in the current bundle, or a concept in the same domain (same package).
- Bare pipe references MUST resolve to: a pipe in the current bundle, or a pipe in the same domain (same package).
- Domain-qualified references MUST resolve within the current package.
- Cross-package references (`->` syntax) are deferred to package-level validation.

## Stage 6: Manifest Validation

For `METHODS.toml`:

1. `[package]` section MUST be present.
2. `address` MUST match the pattern `^[a-zA-Z0-9._-]+\.[a-zA-Z0-9._-]+/[a-zA-Z0-9._/-]+$`.
3. `version` MUST be valid semver.
4. `description` MUST NOT be empty.
5. All dependency aliases MUST be unique and `snake_case`.
6. All dependency addresses MUST match the hostname/path pattern.
7. All dependency version constraints MUST be valid.
8. Domain paths in `[exports]` MUST be valid domain codes.
9. Domain paths in `[exports]` MUST NOT use reserved domains (`native`, `mthds`, `pipelex`).
10. All pipe codes in `[exports]` MUST be valid `snake_case`.

## Stage 7: Package-Level Validation

After loading all bundles and resolving dependencies:

1. Bundles MUST NOT declare a domain starting with a reserved segment.
2. Cross-package references MUST reference known dependency aliases.
3. Cross-package pipe references MUST target exported pipes.
4. Exported pipes MUST exist in the scanned bundles.
5. Same-domain concept and pipe code collisions across bundles are errors.

## Stage 8: Lock File Validation

For `methods.lock`:

1. Each entry's `version` MUST be valid semver.
2. Each entry's `hash` MUST match `sha256:[0-9a-f]{64}`.
3. Each entry's `source` MUST start with `https://`.



## Package Loading

# Package Loading

This page details the dependency resolution algorithm, library assembly, and namespace isolation mechanics.

## Dependency Resolution Algorithm

Dependency resolution is a recursive process that handles local paths, remote fetching, cycle detection, and diamond dependencies.

```
function resolve_all_dependencies(manifest, package_root):
    local_resolved = []
    remote_deps = []

    for dep in manifest.dependencies:
        if dep.path is not null:
            local_resolved.append(resolve_from_filesystem(dep, package_root))
        else:
            remote_deps.append(dep)

    resolved_map = {}        // address -> resolved dependency
    constraints = {}         // address -> list of version constraints
    resolution_stack = set() // for cycle detection

    resolve_transitive_tree(remote_deps, resolution_stack, resolved_map, constraints)

    return local_resolved + values(resolved_map)
```

**Key rules:**

- **Local path dependencies** are resolved directly from the filesystem. They are NOT resolved transitively — only the root package's local paths are honored.
- **Remote dependencies** are resolved transitively. If Package A depends on Package B, and B depends on Package C, then C is also resolved.
- **Cycle detection** uses a DFS stack set. If an address is encountered while already on the stack, the resolver reports a cycle error.

## Diamond Dependency Handling

Diamond dependencies occur when the same package is required by multiple dependents with different version constraints.

```
function resolve_diamond(address, all_constraints, available_tags):
    parsed_constraints = [parse_constraint(c) for c in all_constraints]
    for version in sorted(available_tags, ascending):
        if all(constraint.matches(version) for constraint in parsed_constraints):
            return version
    error("No version satisfies all constraints")
```

This is Minimum Version Selection applied to multiple constraints simultaneously. The resolver:

1. Collects all version constraints from every dependent that requires the package.
2. Lists available version tags from the remote repository (cached to avoid repeated network calls).
3. Sorts versions in ascending order.
4. Selects the first version that satisfies ALL constraints.

When a diamond re-resolution picks a different version than previously resolved, the stale sub-dependency constraints contributed by the old version are recursively removed before re-resolving.

## VCS Fetching

Remote packages are fetched via Git with a three-tier resolution chain:

1. **Local cache check** — look in `~/.mthds/packages/{address}/{version}/`.
2. **VCS fetch** — if not cached, clone the repository:
    - Map address to clone URL: prepend `https://`, append `.git`.
    - List remote tags: `git ls-remote --tags {url}`.
    - Filter tags that parse as valid semver (strip optional `v` prefix).
    - Select version via MVS.
    - Clone at the selected tag: `git clone --depth 1 --branch {tag}`.
3. **Cache storage** — store the cloned directory under `~/.mthds/packages/{address}/{version}/`, removing the `.git` directory.

Cache writes use a staging directory with atomic rename for safety against partial writes.

## Library Assembly

After resolving all dependencies, the runtime assembles a **library** — the complete set of loaded bundles indexed by domain and package:

```
Library:
    local_bundles:     domain -> list of bundle blueprints
    dependency_bundles: (alias, domain) -> list of bundle blueprints
    exported_pipes:    (alias, domain) -> set of pipe codes
    main_pipes:        (alias, domain) -> pipe code
```

The library provides the lookup context for namespace resolution. When a pipe reference like `scoring_lib->scoring.compute_weighted_score` is encountered:

1. Find the dependency by alias `scoring_lib`.
2. Look up domain `scoring` in the dependency's bundles.
3. Find the pipe `compute_weighted_score`.
4. Verify it is exported (in the `[exports]` list or declared as `main_pipe`).

## Namespace Isolation

Packages isolate namespaces completely. Two packages declaring `domain = "recruitment"` have independent concept and pipe namespaces. The isolation boundary is the package, not the domain.

Within a single package, bundles sharing the same domain merge into a single namespace. Collisions (duplicate concept or pipe codes within the same domain of the same package) are errors.

The reference implementation enforces isolation through the library structure: lookups are always scoped to a specific package (identified by alias for dependencies, or "current package" for local references).

## Visibility Checking Algorithm

The visibility checker runs after library assembly:

```
function check_visibility(manifest, bundles):
    exported_pipes = build_export_index(manifest)
    main_pipes = build_main_pipe_index(bundles)

    errors = []

    // Check reserved domains
    for bundle in bundles:
        if bundle.domain starts with reserved segment:
            errors.append(reserved domain error)

    // Check intra-package cross-domain references
    for bundle in bundles:
        for (pipe_ref, context) in bundle.collect_pipe_references():
            if pipe_ref is special outcome ("fail", "continue"):
                skip
            if pipe_ref is cross-package (contains "->"):
                validate alias exists in dependencies
            else:
                ref = parse_pipe_ref(pipe_ref)
                if ref is qualified and not same domain as bundle:
                    if ref.pipe_code not in exported_pipes[ref.domain]:
                        if ref.pipe_code != main_pipes[ref.domain]:
                            errors.append(visibility error)

    return errors
```

The checker runs three passes:

1. **Reserved domain check** — ensures no bundle uses `native`, `mthds`, or `pipelex` as the first domain segment.
2. **Intra-package visibility** — ensures cross-domain pipe references target exported or main_pipe pipes.
3. **Cross-package alias validation** — ensures `->` references use aliases declared in `[dependencies]`.

## See Also

- [Specification: Namespace Resolution Rules](../spec/namespace-resolution.md) — the formal resolution algorithm.
- [The Package System: Version Resolution](../packages/version-resolution.md) — how MVS works.


## Building Editor Support

# Building Editor Support

This page describes how to build editor support for `.mthds` files — syntax highlighting, semantic tokens, schema validation, and formatting.

## TextMate Grammar

The primary mechanism for syntax highlighting is a TextMate grammar layered on top of TOML. The grammar recognizes MTHDS-specific constructs within the TOML structure.

**Scope hierarchy:**

The base scope is `source.mthds` (extending `source.toml`). Key MTHDS-specific scopes include:

- `meta.pipe-section.mthds` — `[pipe.<name>]` table headers
- `meta.concept-section.mthds` — `[concept.<name>]` table headers
- `entity.name.type.mthds` — concept codes in `PascalCase`
- `entity.name.function.mthds` — pipe codes in references
- `string.template.mthds` — prompt template strings
- `variable.other.jinja.mthds` — Jinja2 variables (`{{ }}`, `@var`, `$var`)

**Key patterns to recognize:**

1. **Pipe sections** — table headers matching `[pipe.<snake_case>]` or `[pipe.<snake_case>.<subfield>]`.
2. **Concept sections** — table headers matching `[concept.<PascalCase>]` or `[concept.<PascalCase>.structure]`.
3. **Pipe type values** — string values that match the pipe type names (`PipeLLM`, `PipeSequence`, etc.) in the `type` field of pipe sections.
4. **Prompt templates** — multi-line strings containing Jinja2 syntax and `@variable` / `$variable` shorthand.
5. **Cross-package references** — strings containing `->` (the arrow separator for package-qualified references).
6. **Model references** — string values with `$` or `@` prefixes in the `model` field.

**Implementation approach:**

The reference implementation's TextMate grammar is structured as a set of injection grammars that layer on top of the TOML base grammar. This allows TOML syntax to remain correct while MTHDS-specific constructs receive additional semantic coloring.

## Semantic Token Types

Beyond TextMate grammar-based highlighting, an LSP-aware extension can provide semantic tokens for more precise highlighting. The reference implementation defines 7 MTHDS-specific semantic token types:

| Token Type | Description | Applied To |
|------------|-------------|------------|
| `mthdsConcept` | Concept names | `ContractClause`, `Text`, `Image`, concept references in `inputs`, `output`, `refines` |
| `mthdsPipeType` | Pipe type values | `PipeLLM`, `PipeSequence`, etc. in the `type` field |
| `mthdsDataVariable` | Data variables in prompts | `@variable_name`, `$variable_name`, `{{ variable }}` |
| `mthdsPipeName` | Pipe names in references | Pipe codes in `steps[].pipe`, `branch_pipe_code`, `outcomes`, etc. |
| `mthdsPipeSection` | Pipe section headers | The entire `[pipe.my_pipe]` header |
| `mthdsConceptSection` | Concept section headers | The entire `[concept.MyConcept]` header |
| `mthdsModelRef` | Model field references | Values in the `model` field (e.g., `$writing-factual`, `@default-text-from-pdf`) |

**Detection algorithm for semantic tokens:**

The semantic token provider parses the TOML document and walks the AST to identify MTHDS-specific elements. For each token, it determines the type based on:

1. **Context** — is this value inside a `[pipe.*]` section or a `[concept.*]` section?
2. **Field name** — is this the `type` field, the `model` field, a prompt field, an `inputs`/`output` field?
3. **Value pattern** — does the value match `PascalCase` (concept), `snake_case` (pipe), or have a `$`/`@` prefix (model ref)?

## Using the MTHDS JSON Schema

The MTHDS JSON Schema (`mthds_schema.json`) provides machine-readable validation for `.mthds` files. It is a standard JSON Schema document that describes the complete bundle structure.

**What the schema covers:**

- Header fields (`domain`, `description`, `system_prompt`, `main_pipe`)
- Concept definitions (simple and structured forms)
- All pipe types with their specific fields
- Sub-pipe blueprints (`steps`, `branches`, `outcomes`, `construct`)
- Field types and their constraints

**How to use it:**

1. **For validation** — feed the parsed TOML (as JSON) through a JSON Schema validator. This catches structural errors (wrong field types, missing required fields) without implementing MTHDS-specific validation logic.
2. **For autocompletion** — use the schema's `properties` and `enum` values to suggest field names and valid values.
3. **For hover documentation** — use the schema's `description` fields to show documentation on hover.

**Generating the schema:**

The reference implementation auto-generates the schema from the Pydantic data model (`PipelexBundleBlueprint`) using the `pipelex-dev generate-mthds-schema` command. This ensures the schema stays in sync with the implementation. Alternative implementations can use the published schema directly.

**Configuring schema association:**

In the `plxt.toml` configuration, associate `.mthds` files with the schema:

```toml
[[rule]]
include = ["**/*.mthds"]

[rule.schema]
url = "https://mthds.ai/mthds_schema.json"
```

## LSP Integration Points

The reference implementation includes an LSP server, available standalone via `plxt lsp stdio`. It is built on a fork of [taplo](https://github.com/tamasfe/taplo), extended with MTHDS-specific semantic tokens, validation, and navigation. It currently provides formatting, document symbols, folding, semantic tokens, schema-based validation and completion, and basic within-bundle go-to-definition. The LSP is bundled with the [`pipelex-tools`](https://pypi.org/project/pipelex-tools/) CLI and with the Pipelex VS Code extension ([source](https://github.com/Pipelex/vscode-pipelex), [Marketplace](https://marketplace.visualstudio.com/items?itemName=pipelex.pipelex), [Open VSX](https://open-vsx.org/extension/Pipelex/pipelex)).

The following integration points describe the full scope of MTHDS-aware language server capabilities. Each bullet notes the current coverage in the reference implementation:

- **Diagnostics** — run validation (Stages 2–7 from the [Validation Rules](validation-rules.md) page) and report errors as LSP diagnostics. *(Reference implementation: schema-level validation only.)*
- **Completion** — suggest pipe type names, native concept codes, field type names, concept codes from the current bundle, and pipe codes for references. *(Reference implementation: schema-based suggestions for field names and values.)*
- **Hover** — show concept descriptions, pipe signatures, and field documentation. *(Reference implementation: schema-based field documentation.)*
- **Go to Definition** — navigate from a concept/pipe reference to its definition (may span files for domain-qualified or cross-package references). *(Reference implementation: within-bundle navigation only.)*
- **Find References** — find all usages of a concept or pipe across bundles. *(Not yet implemented in the reference implementation.)*
- **Rename** — rename a concept or pipe code across all references in the package. *(Not yet implemented in the reference implementation.)*

## See Also

- [Tooling: Editor Support](../tooling/editor-support.md) — user-facing editor documentation.
- [Tooling: MTHDS JSON Schema](../tooling/json-schema.md) — user-facing schema documentation.


# About

## Roadmap

# Roadmap

The MTHDS standard is at version `1.0.0`. This page outlines planned and potential directions for future development.

## Near-Term

- **Registry reference implementation.** A reference implementation for the registry index, enabling `mthds pkg search` to query remote registries in addition to local packages.
- **Package signing.** Optional signed manifests for enterprise use, enabling verifiable authorship and integrity beyond SHA-256 content hashes.
- **Cross-package concept refinement validation at install time.** The specification allows validation of concept refinement across packages at both install time and load time. The current reference implementation validates at load time only. Install-time validation would detect breaking changes earlier.

## Medium-Term

- **Know-How Graph web interface.** A web-based explorer for the Know-How Graph, enabling visual navigation of concept hierarchies and pipe chains across the public ecosystem.
- **Proxy/mirror support.** Configurable proxy for package fetching, supporting speed, reliability, and air-gapped environments (similar to Go's `GOPROXY`).

## Long-Term

- **Conditional concept fields.** Allow concept structure fields to be conditionally present based on the values of other fields.
- **Runtime interoperability standard.** A specification for how different MTHDS runtimes can exchange concept instances, enabling cross-runtime pipe invocation.

## Contributing to the Roadmap

The roadmap is shaped by community needs. If you have a use case that the standard does not yet support, open an issue in the MTHDS standard repository. Proposals that include concrete `.mthds` examples demonstrating the need are especially helpful.


## Contributing to MTHDS

# Contributing to MTHDS

MTHDS is an open standard. Contributions are welcome — whether they are bug reports, specification clarifications, tooling improvements, or new packages.

## Ways to Contribute

### Report Issues

If you find an inconsistency in the specification, a bug in a tool, or an edge case that is not documented, open an issue in the MTHDS standard repository. Include:

- What you expected to happen.
- What actually happened.
- A minimal `.mthds` or `METHODS.toml` example that demonstrates the issue.

### Propose Specification Changes

Specification changes follow a structured process:

1. **Open a discussion** describing the problem and your proposed solution. Include concrete `.mthds` examples showing before/after.
2. **Draft the change** as a pull request against the specification. Normative changes use RFC 2119 language (`MUST`, `SHOULD`, `MAY`).
3. **Review** by the maintainers and community. Changes to the specification require careful consideration of backward compatibility.
4. **Merge and release** as a new minor or major version of the standard.

### Build Packages

The ecosystem grows through packages. Publish packages that solve real problems in your domain. Well-documented packages with clear concept hierarchies and typed pipe signatures make the Know-How Graph more useful for everyone.

### Build Tools

The standard is tool-agnostic. If you build an MTHDS-related tool — an alternative runtime, an editor extension, a registry implementation, a visualization tool — share it with the community.

## Coding Standards for the Reference Implementation

The reference implementation (Pipelex) has its own coding standards and contribution guidelines. See the Pipelex repository for details.

## License

The MTHDS standard specification is open. Implementations may use any license. The reference implementation's license is specified in its repository.


## Contributing

--8<-- "CONTRIBUTING.md"


## Code of Conduct

--8<-- "CODE_OF_CONDUCT.md"


## License

--8<-- "LICENSE"


## Changelog

--8<-- "CHANGELOG.md"