Span Chain — find where two recorded traces diverge

Q: Which OpenTelemetry input does it accept?

Span Chain supports an OTLP/HTTP JSON subset on /v1/traces, with a Bearer key. Protobuf isn’t accepted, so put an OpenTelemetry Collector in front and export with encoding: json and no compression. Each resource needs service.instance.id, which becomes the run id.

Q: How do I self-host it?

With Docker Compose: clone the repository, copy .env.example to .env, set POSTGRES_PASSWORD, SECRET_KEY_BASE and GF_API_KEY, then run docker compose up --build. Database migrations run on first start. The code is MIT licensed.

Record a run.
Compare two.

Span Chain is a self-hosted trace recorder⁰¹ and structural regression-diff layer⁰² for AI agents.

01trace recorder

Record a run

The spans an agent run sends are stored on your own server, in order, run by run. Walk the run span by span afterwards.

7f3a · agent.run9 spans · 4.32 s

span0 s12345 s

span tree · time from run startillustrative data

02structural regression-diff

Compare two runs

Two records side by side. The diff points to the span where the runs parted and shows what differed in the agent’s configuration.

A 7f3aB 9c015 differences

span0 s12345 st 0.00 stype

config_diff · temperature 0.2 → 0.7illustrative data

License

MIT

Ingest

Span Chain supports an OTLP/HTTP JSON subset.

Compare

Compare agent configuration — structure and config.

Playback

Trace playback / cassette re-ingest. Not agent replay.

Limits of the record

Span Chain stores received spans in a SHA-256 hash chain. Re-verification can detect edits, reordering, and gaps in the middle of that stored history. It cannot detect spans that were never received, spans removed from the end, or a chain rebuilt by the server operator.

Case	What it means	Diagram
Capture	spans that were never received
Tail	spans removed from the end
Operator-rebuild	a chain rebuilt by the server operator

What goes into the record.

Every span a run sends becomes one row in an append-only ledger. The row’s SHA-256 hash covers the span, its place in the run and the hash of the row before it.

01one entry

One span, one ledger row

The span is stored as canonical JSON. Its position and its parent go into the hash with it, so an edited or moved row no longer matches.

omp-01a10c2e…7f3a · seq 4ledger_entries

run_idomp-01a10c2e-5b7d-7c41-9a2e-3f0d8b6c7f3ahashed

epoch_id · seq0 · 4hashed

event_typeexecute_toolhashed

parent_span_id5e0b9c21a7d4f6e3hashed

payload{"attributes":{"gen_ai.tool.name":"bash","status":"ok",…},"ended_at":"…","name":"execute_tool",…}hashed

prev_hash4b07e2d1…f3c9e91ahashed

hashsha256 → c3d95f0a…7e8a02f4result

projectionsspan_id · trace_id · started_at · ended_at · statuscopies

projections are copied out of the payload for fast readsillustrative data

02one run

Each run is its own chain

Entries of a run link through prev_hash from seq 0. Run-level fields come from OTLP resource attributes.

omp-01a10c2e…7f3a9 entries

run_idservice.instance.idomp-<session uuid>

eval_idgf.eval_idomp-sessions-2026-10

modelgf.agent.modelclaude-sonnet-4-5

statusattributes.statusok | error, per span

prev_hash links · seq 0 → 8illustrative data

How a run enters the record

Send. Chain. Check again.

Standard OpenTelemetry in, a hash chain per run in the middle, re-verification and a diff on the way out.

01send

Export spans over OTLP/HTTP JSON

Span Chain supports an OTLP/HTTP JSON subset. The resource attribute service.instance.id names the run; gf.eval_id groups the runs you want to compare. Protobuf isn’t accepted, so a Collector in front exports JSON.

POST /v1/traces
Authorization: Bearer $GF_API_KEY
Content-Type: application/json

02chain

Append each span to its run

Each run gets its own process and is appended in order. Every entry gets a seq and a SHA-256 hash that includes the previous entry’s hash.

hash = sha256(seq : prev_hash : event_type
  : payload : parent_span_id
  : run_id : epoch_id)

03check again

Re-verify a run, diff two runs

verify_ledger walks a run’s chain again from its first entry; a background sweep does the same for recent runs. Two runs with the same eval_id can be diffed span by span.

GET /api/runs/:run_id/verify
GET /eval/:eval_id?run_a=…&run_b=…

Views over the ledger.

Runs, the spans of one run, a diff of two runs and the chain after re-verification. All four exist in the repo today: /trail and /eval/:id in Phoenix LiveView, run detail and verify also in the React UI. The data is illustrative, shaped like OMP sessions recorded through a local OpenTelemetry Collector.

Span Chain illustrative mockup

/traillive

Runs

latest 50 runs · newest first

run · modelspanserrorsstartedduration

/trail/omp-01a10c2e…7f3a/trail/:run_id

← All runs

omp-01a10c2e…7f3a

9 ledger rows · claude-sonnet-4-5 · eval omp-sessions-2026-10

event_typeseqmsstatus0 s12.5 s

/trail — the latest runs; one that is still arriving updates live. One OMP session is one run, a task subagent arrives as its own run.

/trail/:run_id — the span tree of one run, built from parent_span_id, with seq, duration and the ok / error status.

Span Chain illustrative mockup

/eval/omp-sessions-2026-10?run_a=…7f3a&run_b=…9c01/eval/:id

← Back to eval

…7f3a vs …9c01

A: 9 spans · 12,480 ms | B: 9 spans · 13,120 ms | 5 differences

⚙ Agent config diffmodel"claude-sonnet-4-5" → "claude-opus-4-1"

spantyperun Arun BΔ

execute_tool · editspan_removed310 ms——

execute_tool · grepspan_added—420 ms—

chatduration_diff2,380 ms3,310 ms39%

execute_tool · bashduration_diff2,300 ms1,580 ms31%

GET /api/runs/…7f3a/verifyverify

Chain re-verified

run omp-01a10c2e…7f3a · walked from seq 0

{
  "run_id": "omp-01a10c2e-…-7f3a",
  "verified": true,
  "span_count": 9,
  "error": null
}

verifiedverify_ledger → OK · 9/9 spans verified

Re-verification can’t see spans that were never received, spans removed from the end, or a chain rebuilt by the server operator. Limits of the record

/eval/:id — two runs that share an eval_id, diffed span by span. Config differences come first; the marked row is the deviation point.

/api/runs/:id/verify — walks the run’s chain again from the first entry. The React UI shows the result as a stamp on the run page.

Next to LangSmith and Langfuse.

What each tool documents about itself, checked against its own docs, pricing pages and GitHub on 2026-10-17. Span Chain’s column follows the same rule: only what the repo does today.

Aspect	LangSmith	Langfuse	Span Chain
What it is	A framework-agnostic platform for testing, deploying, monitoring and improving agents.	An open-source AI engineering platform to debug, analyze and iterate on agent applications.	A self-hosted trace recorder and structural regression-diff layer for AI agents.
License	SDKs under MIT. The self-hosted platform runs with a license key from LangChain.	Core under MIT; `ee/` directories under a commercial license.	MIT.
Self-hosting	An add-on to the Enterprise plan, on Kubernetes; hybrid deployment too.	All core features, free. Docker Compose for local use, Kubernetes (Helm) for production.	Docker Compose on your own server.
OpenTelemetry	OTLP/HTTP endpoint at `/otel`.	OTLP/HTTP, JSON or protobuf, at `/api/public/otel`.	An OTLP/HTTP JSON subset at `/v1/traces`.
Comparing runs	Experiment comparison: regressions on feedback scores, a text diff of outputs, traces side by side.	Experiments against a baseline: scores, outputs, cost and latency.	A span-by-span diff of two runs: added and removed spans, durations off by more than 20%, agent config, the deviation point.
Evals & prompts	Offline and online evals, LLM-as-judge, versioned prompts, Playground.	Online and offline evals, LLM-as-judge, prompt management, Playground.	None. Runs are grouped by eval_id to be diffed, not scored.
Re-running	Open an LLM run in the Playground with its provider and model.	Open a generation in the Playground; sessions show a replay of the whole interaction.	Trace playback / cassette re-ingest. Not agent replay.
Audit & integrity	Audit logs of administrative and configuration actions (Enterprise).	Audit logs of actions on resources (Enterprise); data retention policies.	Received spans in a SHA-256 hash chain per run; re-verification can detect edits, reordering and gaps in the middle of that history. Limits

Span Chain doesn’t replace LangSmith or Langfuse. Keep them for evals and prompts; Span Chain adds a record you can re-verify later and a span-by-span diff of two runs.

Sources: docs.langchain.com, langchain.com/pricing, langfuse.com docs and pricing, github.com/langfuse/langfuse, checked 2026-10-17. Products change; this table reflects their documentation on that date.

Logs are claims.
Not evidence.

Auditing AI agents · GhostFactory. What a session should capture, and where vendor claims stop being session facts.

Latest2026-10-14

We audited an agent’s report. Then we had to audit the audit.

At the end of a session an agent writes a summary, and that summary is what I act on. So I wanted to know how much of a summary actually holds: every factual claim, checked against git history and the tool outputs recorded in the transcript.

Read the post 2026-09-26

The toggle did not stop the pack. “Destroyed” is still a vendor claim.

A coding agent packs your workspace under ~/.zcode, including .git. Toggles that look like off switches do not stop capture. “Destroyed” is still a vendor claim, not a session fact.

Read 2026-09-24

Changelog said AGENTS.md. The session never recorded it.

Two developers pull the same commit and the same AGENTS.md. One session loads the file; the other skips it with no warning. That is a record problem, not a philosophy problem.

Read 2026-09-18

How to Audit Claude Code Sessions: A Tamper-Evident Trail for Every Tool Call

Claude Code reads files, writes code and runs shell commands on your behalf. Here is how to audit Claude Code sessions into a hash-chained, tamper-evident trail you control — using two hooks and one Python script.

Read 2026-08-06

How to Audit AI Agents: From Mutable Logs to Tamper-Evident History

When an AI agent fails, the first question is why. The second, increasingly asked by security and compliance teams, is: can you prove it?

Read

spanchain.dev/blogAll posts

Frequently asked.

Short answers. The repository and its docs have the long ones.

What is Span Chain?

Span Chain is a self-hosted trace recorder and structural regression-diff layer for AI agents. It stores the spans your agent runs send, keeps each run in a SHA-256 hash chain, and shows where two recorded runs diverge.

What can re-verification detect, and what can’t it?

Span Chain stores received spans in a SHA-256 hash chain. Re-verification can detect edits, reordering, and gaps in the middle of that stored history. It cannot detect spans that were never received, spans removed from the end, or a chain rebuilt by the server operator.

How is it different from LangSmith or Langfuse?

They are platforms for tracing, evaluation and prompt work, and Span Chain doesn’t replace them. It adds a record of each run you can re-verify and a span-by-span diff of two runs. The comparison above lists what each tool documents.

Does it replay agent runs?

No. Trace playback / cassette re-ingest. Not agent replay. A recorded run can be saved as a cassette and re-ingested under a new run_id; that builds a fresh chain and diffs it against the original. No model is called and the agent isn’t run again.

Which OpenTelemetry input does it accept?

Span Chain supports an OTLP/HTTP JSON subset on /v1/traces, with a Bearer key. Protobuf isn’t accepted, so put an OpenTelemetry Collector in front and export with encoding: json and no compression. Each resource needs service.instance.id, which becomes the run id.

What does an OMP session look like in the record?

Through a local Collector, as in the examples on this page: one OMP session is one run, omp- plus the session uuid. Advisor spans fold into that run; a task subagent becomes its own run. Runs land in the eval omp-sessions-YYYY-MM, with the model from gf.agent.model and an ok / error status per span.

How do I self-host it?

With Docker Compose: clone the repository, copy .env.example to .env, set POSTGRES_PASSWORD, SECRET_KEY_BASE and GF_API_KEY, then run docker compose up --build. Database migrations run on first start. The code is MIT licensed.

Find where two recorded traces diverge.

Record a run.Compare two.