Dissecting Claude Code: A Graph-Powered Deep Dive into an AI Coding Agent's Architecture

Thu, 02 Apr 2026 00:00:00 +0000

We indexed 1,906 source files, 9,848 functions, and 22,678 call edges using CodeGraph — a code intelligence skill powered by NeuG, an open-source embedded graph database — to reveal what lies beneath the surface of one of the most ambitious AI coding tools ever built.

The Numbers at a Glance

Metric	Count
Source Files	1,906
Functions	9,848
Call Edges	22,678
Import Edges	15,743
Modules	306
Classes	128

Claude Code is a TypeScript monolith running on Bun + Ink (React for terminals). main.tsx alone exceeds 4,700 lines, and REPL.tsx tops 5,000 lines with 288 outgoing function calls — the highest fan-out of any component in the system.

1. Architecture: Five Layers, One Gravity Well

CodeGraph’s layer discovery reveals an onion-ring architecture with src/utils as the gravitational center. The five layers, from top to bottom:

Entry Points — REPL.tsx (288 calls, 236 imports), main.tsx (275 calls, 163 imports), and print.ts (164 calls, CLI pipeline). These are the three doors into Claude Code.

UI Layer — 131 files including the 113-file components/ directory, PromptInput (105 outgoing calls), the Buddy virtual pet system, and Ink’s custom terminal renderer (52 methods).

Logic Layer — 120 hooks, 80+ slash commands, 42 tools (from the 151-function BashTool to the 1-function SleepTool), and 14 keybinding files.

Service Layer — Analytics (428 callers), MCP protocol (41 files), Permissions (41 files), Bridge remote control (33 files), the Swarm multi-agent system, and 7 task types.

Foundation — The utils/ directory with 299 files absorbing 5,951 incoming calls — 6x more than the next highest module. It’s not a utility folder; it’s the bedrock of reality for this codebase.

The REPL: A Component That Imports the Universe

REPL.tsx imports from 236 files — more than any other file in the codebase. It connects to hooks, components, services, bridge, swarm, voice, cost-tracking, MCP, and keybindings. Yet it has 0 callers (fan-in = 0). It’s the root UI leaf that nobody calls into; it only calls out.

2. Bridge Functions: The Nervous System

A bridge function is one called from the most distinct modules — the glue holding disparate subsystems together. CodeGraph identified these top bridges:

Function	File	Modules Spanned	Total Callers
`logForDebugging`	debug.ts	90	911
`logEvent`	analytics/index.ts	94	428
`logError`	log.ts	77	413
`set`	fileStateCache.ts	64	290
`jsonStringify`	slowOperations.ts	63	248
`isEnvTruthy`	envUtils.ts	47	194
`getGlobalConfig`	config.ts	44	163
`getCwd`	cwd.ts	43	135

logForDebugging is called from 90 out of 306 modules — nearly 1 in 3 modules feeds debug info through this single function. It’s the all-seeing eye of Claude Code.

Notice the file name slowOperations.ts housing jsonStringify (248 callers, 63 modules). The team explicitly wraps JSON operations in a module named “slow” — a candid acknowledgment that serialization is a performance bottleneck they want to track.

The top 3 bridge functions alone account for 1,752 call edges across 261 module-boundaries. Removing any one would fracture the entire system.

3. The Permission Gateway: A Single Chokepoint

One of the most architecturally significant findings: every tool execution flows through a single function — checkPermissionsAndCallTool. It has only 1 caller (streamedCheckPermissionsAndCallTool) but calls 30 different functions spanning telemetry, permission rules, classifier checks, error handling, and analytics.

This is a textbook Policy Enforcement Point (PEP) — a single chokepoint where all 42 tools must pass through for authorization. The downstream functions include:

checkRuleBasedPermissions — evaluates the rule engine
startSpeculativeClassifierCheck — the “YOLO mode” classifier
resolveHookPermissionDecision — hook-based overrides
logEvent + logOTelEvent — dual telemetry (analytics + OpenTelemetry)
startToolSpan / endToolSpan — tracing boundaries
formatZodValidationError — schema validation

This architecture makes it trivially easy to add security policies, audit logging, and rate limiting without modifying any individual tool.

The 151:1 Complexity Ratio

Not all tools are created equal. BashTool has 151 functions (command parsing, sandboxing, permission verification, timeout management, output capture, cross-platform compatibility). SleepTool has 1 function. That’s a 151:1 complexity range across a unified interface.

Tool	Functions	Category
BashTool	151	Shell execution
PowerShellTool	101	Windows shell
AgentTool	91	Multi-agent orchestration
LSPTool	35	Language server
FileEditTool	31	File operations
GrepTool	8	Search
SleepTool	1	Wait

4. The Buddy System: A Gacha Game Inside a Dev Tool

Perhaps the most delightful discovery: Claude Code contains a complete virtual pet system with gacha mechanics, ASCII art animations, and RPG stats.

How Companion Generation Works

Hash the User ID with a salt (friend-2026-401) using hashString()
Seed a Mulberry32 PRNG with the hash value
Roll deterministically for species, rarity, eyes, hat, and stats
Cache the result (the buddy renders every 500ms, can’t re-roll each frame)

The “bones” (species, rarity, stats) are regenerated from hash on every read and never persisted to disk. Only the “soul” (user-chosen name and personality) is saved. This means:

Anti-cheat: Users can’t edit their config to fake a Legendary rarity
Schema-safe: Changes to the species array won’t break saved companions
Deterministic: Same user = same companion forever

The Hex Obfuscation Trick

All 18 species names are encoded as hex character codes to dodge a build-time string filter:

const duck = String.fromCharCode(0x64, 0x75, 0x63, 0x6b)  // "duck"

One species name collides with an internal model codename listed in excluded-strings.txt, so the team encoded ALL species uniformly.

Gacha Rarity Distribution

Rarity	Probability
Common	60%
Uncommon	25%
Rare	10%
Epic	4%
Legendary	1%

Each companion has 5 RPG stats: DEBUGGING, PATIENCE, CHAOS, WISDOM, SNARK. The ASCII sprites are 5 lines tall, 12 columns wide, with 2-3 frames for idle animation. Frame -1 in the 15-frame idle sequence triggers a rare blink animation.

Here’s an actual duck sprite from the codebase:

    __
  <(· )___
   (  ._>
    `--´

CodeGraph’s impact analysis traces the call chain: getCompanion() → CompanionSprite → PromptInput (105 calls) → REPL (288 calls) → Terminal UI.

5. Dream Task: The AI That Dreams While You Code

Claude Code has a background memory consolidation system that literally runs while you work — the AI “dreams” about past sessions to improve its memory.

The Four Gates (Cheapest First)

Before dreaming begins, four gates must pass — ordered by computational cost:

Time Gate (cost: 1 stat() call) — Has 24 hours elapsed since the last dream?
Scan Throttle (cost: memory timestamp check) — Has 10 minutes passed since the last scan?
Session Gate (cost: readdir) — Are there 5+ new sessions to consolidate?
Lock Gate (cost: flock) — Is the file-based mutex free?

This ordering ensures cheap rejections happen first. Most API turns exit at gate 1 with a single stat call.

The Four Phases of Dreaming

Once the gates pass, a restricted subagent forks with read-only Bash permissions (only ls, grep, cat, head, tail):

Orient — ls the memory directory to understand current state
Gather — grep recent session transcripts for patterns and reusable knowledge
Consolidate — Write/update memory files with distilled knowledge
Prune — Keep the memory index under 25KB, entries under 150 characters

The Filesystem Retry Trick

When a dream is killed mid-flight, it calls rollbackConsolidationLock(priorMtime) to rewind the lock file’s modification time. This makes the Time Gate pass again on the next API turn — a retry mechanism built into the filesystem itself.

6. Swarm: Multi-Agent Orchestration

The most architecturally ambitious module: a three-tier backend abstraction for spawning and managing teams of AI agents.

Backend Detection Priority Chain

When a teammate is spawned, the system probes the environment:

Priority	Backend	Lines	Detection Method
P1	TmuxBackend	765	`ORIGINAL_USER_TMUX` env var
P2	ITermBackend	370	`TERM_PROGRAM` + `it2 session list`
P3	InProcessBackend	340	Always available (fallback)

The PaneBackendExecutor adapter wraps tmux/iTerm2 into the same TeammateExecutor interface that InProcessBackend implements directly. This lets the system degrade gracefully: tmux → iTerm2 → in-process.

Why AsyncLocalStorage?

When agents are backgrounded (Ctrl+B), multiple agents run concurrently in the same Node.js process. AppState is a single shared mutable object. If Agent A’s telemetry reads it, it might get Agent B’s context.

AsyncLocalStorage isolates each async execution chain, giving each agent its own identity bubble — without requiring process isolation.

Task ID Design

Each task gets a unique ID: a single-letter prefix + 8 random base-36 characters:

b = bash, a = agent, r = remote, t = teammate, w = workflow, m = monitor, d = dream

This gives instant type identification from the ID string alone, with 36^8 ≈ 2.8 trillion combinations.

7. Cost Tracker: Follow the Money

The cost tracking system handles 7 pricing tiers with dynamic tier selection for Opus 4.6:

Model	Input/1M	Output/1M
Haiku 3.5	$0.80	$4
Sonnet	$3	$15
Opus 4.5	$5	$25
Opus 4/4.1	$15	$75
Opus 4.6 (fast)	$30	$150

getOpus46CostTier(fastMode) dynamically selects the tier based on response_speed — fast mode costs 2x.

The Billing Access Gate

The display logic reveals thoughtful product thinking:

Subscribers (Max/Pro): Costs are hidden (flat rate — showing costs would be confusing)
API-key users with admin/billing roles: Costs are displayed (they’re paying per-token)

This is controlled by hasConsoleBillingAccess(), which checks DISABLE_COST_WARNINGS env, subscriber status, auth tokens, and org/workspace roles.

What Gets Tracked

totalCostUSD — cumulative dollar amount
totalAPIDuration — time spent waiting for API responses
totalToolDuration — time spent executing tools
linesAdded / linesRemoved — code change volume
modelUsage — per-model token breakdown
fpsMetrics — terminal rendering performance

All of this persists to project config and survives session restarts via restoreCostStateForSession().

8. The Loneliest 1,265

CodeGraph found 1,265 isolated functions — functions with zero callers AND zero callees. Out of 9,848 total, that’s 12.8% sitting in complete isolation.

Where do they live?

File	Total Functions	Why Isolated
`bootstrap/state.ts`	212	Getters/setters called dynamically
`sessionStorage.ts`	153	Dynamic property access
`yoga-layout/index.ts`	134	FFI binding layer

bootstrap/state.ts alone has 212 functions — the densest file in the codebase. It’s a global state machine that vends getters and setters for everything from session IDs to model strings. Static analysis can’t see dynamic property access, so they appear “isolated” even though they’re heavily used at runtime.

9. The Class Hierarchy: OOP at the Boundaries

With 128 classes in a primarily functional codebase, where does OOP live?

Class	Methods	Purpose
`Node`	91	Yoga layout FFI binding
`Cursor`	59	Terminal cursor management
`Ink`	52	Custom React terminal renderer
`YogaLayoutNode`	50	Flexbox layout engine
`WebSocketTransport`	30	WebSocket client
`TmuxBackend`	20	Tmux pane management
`CCRClient`	24	API transport layer

The top 4 classes are all rendering infrastructure. The business logic (tools, permissions, cost tracking, buddy) is overwhelmingly functional — closures, hooks, and module-level state. Classes appear only at the boundaries: transport protocols, terminal abstractions, and system backends.

10. Patterns Worth Stealing

Deterministic Generation Without Persistence

The buddy system regenerates bones from hash(userId) every time. Only soul persists. This eliminates schema migration headaches AND prevents config-file cheating.

Gate Ordering by Cost

The dream task checks: in-memory flag → stat a file → list a directory → acquire a lock. Each gate is more expensive than the last, so cheap rejections happen first.

Single-Letter Task ID Prefixes

bash, agent, remote, teammate, workflow, monitor, dream — instant type identification from the ID string alone.

Honest Function Names

getFeatureValue_CACHED_MAY_BE_STALE, slowOperations.ts, writeFileSyncAndFlush_DEPRECATED — names that force callers to understand the tradeoffs.

AsyncLocalStorage for Concurrent Identity

When multiple agents share a process, don’t use global state for identity. Use AsyncLocalStorage to give each async execution chain its own context.

Single Permission Gateway

Route all 42 tools through one function. Adding a new security policy means changing one file, not 42.

Conclusion

Claude Code is not just an AI coding assistant — it’s a miniature operating system for AI agents. Through CodeGraph’s structural analysis, we’ve uncovered:

A utils foundation with 5,951 incoming calls acting as bedrock
A permission system where all 42 tools pass through a single gateway
A virtual pet with gacha mechanics and anti-cheat architecture
An AI that dreams about its memories during idle hours
A three-tier multi-agent orchestration system spanning tmux, iTerm2, and in-process
1,265 isolated functions (12.8%) sitting in silence

The graph never lies. Behind every great product is an architecture that tells its own story — you just need the right tools to read it.

Following 94 Million Relationships Down the Rabbit Hole

Tue, 24 Mar 2026 00:00:00 +0000

We ran an experiment: take a real question, point it at a massive academic knowledge graph, and keep asking follow-ups to see how deep LLM-driven analysis can actually go.

The question: “What impact has the Russia-Ukraine conflict had on Russian academia?”

The data comes from OpenAIRE, one of Europe’s largest open research data platforms: 22 million entities, 94 million relationships, spanning publications, authors, institutions, funding sources, and journals. Too large to feed into any LLM directly — but perfect for structured analysis.

Here’s how the investigation unfolded.

Question 1: Did publication output actually decline?

The obvious starting point. We matched Publication ↔ Organization patterns across the graph and aggregated by country and year to track Russia’s publication trend.

Year	Publications	YoY Change	Cumulative
2020	9,544	-	0%
2021	8,976	-6.0%	-6.0%
2022	8,548	-4.8%	-10.4%
2023	7,732	-9.5%	-19.0%
2024	4,686	-39.4%	-50.9%

A mild decline before the war, accelerating sharply after 2022, nearly halved by 2024.

But is this Russia-specific, or a global trend? We sliced the same analytical structure differently — global publication volume actually increased in 2022. Comparing 2021→2024: Russia dropped 47.8%, far exceeding the US and Germany at 27.6%, while China grew 14.8%.

This isn’t a global downturn. It’s a cliff — and it’s Russia’s alone.

Question 2: Did the funding dry up?

When publication output drops, the first instinct is to follow the money.

We extended our analysis by adding a Funding dimension on top of the existing structure — not starting over, but incrementally expanding what we’d already built.

The result: the European Commission (EC) had been a major funder of Russian research. After 2022, EC completely cut off funding for projects involving Russian institutions — while continuing to fund 26 Ukrainian institutions. This aligns perfectly with the EC’s official policy announcements.

The money did dry up. But does that explain everything?

Question 3: Are journals shutting the door too?

We pivoted to a different angle, adding a Publisher dimension to the structure from Question 1: which journals changed their stance toward Russian papers around 2022?

Finding: EDP Sciences dramatically reduced publications from Russian institutions after 2022. Further investigation revealed that EDP Sciences had publicly declared support for Ukraine. And crucially, their publication volume from other countries didn’t see a comparable drop — this wasn’t a general contraction, it was selective.

Funding was being cut. Publishing channels were narrowing.

Question 4: What happened to the collaboration network?

The previous questions looked at individual factors. We zoomed out to the big picture — querying Author(country=RU) → Publication → Organization to map how Russia’s collaboration with each country changed over time.

Partner Country	2021	2024	Change	Country Type
Turkey	988	16	-98.4%	Non-Western, no sanctions
Germany	656	46	-93.0%	Western, sanctions
USA	3,614	271	-92.5%	Western, sanctions
China	570	112	-80.4%	Non-Western, no sanctions

Wait — the steepest drop is Turkey?

-98.4%. A non-Western country. A country that imposed no sanctions on Russia. And yet the collaboration decline is worse than with the US or Germany.

This breaks a seemingly natural assumption: that the decline is primarily sanctions-driven. If sanctions were the main cause, Western countries should show the steepest drops. But the data tells a different story.

Sanctions are part of the picture, but the deeper shift is systemic: the entire international academic network is distancing itself from Russia. It’s not that certain countries stopped cooperating — Russia is being structurally marginalized from the global research fabric.

Why could we keep asking?

Looking back at these four steps, each finding naturally raised the next question. This “follow the thread” experience feels intuitive — analysis should work this way.

But if you’ve used traditional data analysis tools, you know reality is different. The typical workflow looks like this:

Ask question → get result → want to follow up → realize the data model doesn’t have that dimension → ask a data engineer to redesign the schema → rebuild the wide table → rerun the query → finally see the result → want to follow up again → redesign again → …

Every follow-up costs as much as the first question. Three questions means three full analysis cycles from scratch. So most analyses stop at the surface — not because people don’t want to go deeper, but because going deeper is prohibitively expensive.

Text-to-SQL and NL2BI tools solve the problem of “how to ask the first question more easily” — having an LLM translate natural language into queries. That’s genuinely useful, but it doesn’t solve the follow-up problem. The SQL gets generated, but the underlying data model is still rigid, intermediate results are still discarded, and every new question still requires full recomputation.

What we set out to solve is exactly this: make follow-up questions dramatically cheaper than the first one.

Step	Question	Operation	What was reused
1	Did output decline?	Match Pub↔Org on graph, aggregate by year+country	—
2	Did funding dry up?	Add Funding dimension	Pub-Org structure from Step 1
3	Are journals biased?	Add Publisher dimension	Pub-Org structure from Step 1
4	How much did collaborations drop?	Match Author→Pub→Org, aggregate by country	Underlying data structures

The analytical structure built in Step 1 was reused across all subsequent steps. Each follow-up wasn’t “start over” — it was “take one more step from where we left off.”

How it works under the hood

Keeping the follow-up chain going requires three things simultaneously: the analytical structure must be dynamically extensible, intermediate results must persist, and computation over massive data must be fast enough for interactive use. We tackle each with a specific design.

Hypergraph Data Model: a schema that grows with your questions

Traditional analytical data models (star/snowflake schemas) are designed upfront. You must decide on all dimensions before asking any questions — miss one, and you start over.

We designed a Hypergraph-based data model with a composable operator algebra:

Source: perform pattern matching on the raw data graph to produce a hypergraph — the starting point
Join: add new dimensions to an existing hypergraph — the ability to extend on follow-up
View: materialize intermediate results — the system’s “memory” of what’s been computed
DrillDown / RollUp / Slice / Dice: multidimensional slicing and aggregation on hypergraphs

These operators form a closed algebra: freely composable and chainable. This is what powers the “add a dimension and keep going” experience in the case study above — the schema isn’t predefined, it grows as your curiosity leads.

Unbiased Sampling: sub-second responses over 94 million relationships

Whether you can sustain a chain of follow-ups depends heavily on how long each step takes. If every question takes five minutes to run, you’ll lose patience by Step 3.

But the core operation behind these analyses is subgraph matching — exhaustively computing this over a large-scale graph is prohibitively expensive. Our approach: unbiased sampling. Instead of enumerating all matches, we uniformly sample a subset and use statistical methods to estimate aggregates. “Unbiased” means the estimates don’t systematically skew high or low.

For analytical tasks, this trade-off is remarkably effective: what matters is “did something change, which direction, who changed most” — not whether every number is exact to the last unit. In practice, COUNT estimates have an average error rate of just 0.27% — trend-level conclusions stay perfectly intact while computation cost drops by one to two orders of magnitude.

NeuG: co-located storage and computation

NeuG is our graph storage engine. The key design decision: sampling-based subgraph matching algorithms are implemented directly in the storage layer, so computation executes where the data lives — no need to shuttle data to a separate system.

Many systems don’t bottleneck on the operators themselves, but on data movement. NeuG’s principle: the best optimization is not moving data at all.

For the formal definitions of the hypergraph data model, operator design, and theoretical guarantees of the sampling algorithms, see our paper: A Hypergraph-Based Framework for Exploratory Business Intelligence.

Benchmarks

Beyond the real-world case study above, we ran systematic evaluations on LDBC Social Network Benchmark datasets (SF1, SF3, SF10) across 13 analytical queries of varying complexity.

On the largest scale, SF10 (~10GB), ExBI was the only system to complete all 13 queries. Neo4j completed 7; MySQL completed 2.

Performance:

vs. Neo4j: 16.21x average speedup, up to 146.25x
vs. MySQL: 46.67x average speedup, up to 230.53x

Speed doesn’t come at the cost of accuracy — COUNT average error rate is 0.27%, MAX error is near zero.

For the full experimental design and analysis, see the paper.

Try it yourself

We integrated this analytical capability into OpenClaw by adding NeuG-related data loading and analysis skills. The case study above was conducted entirely in this environment.

NeuG: https://github.com/alibaba/neug
NeuG data loading and analysis skills: https://github.com/Louyk14/neug/tree/main/skills
Docker (code + OpenAIRE dataset, ready to use): https://hub.docker.com/r/shunyangli/neugbi
Paper: A Hypergraph-Based Framework for Exploratory Business Intelligence
Full analysis report: Impact of the Russia-Ukraine War on Russian Academia

We X-Rayed OpenClaw with a Graph Database — Here's the Technical Debt We Found

Thu, 19 Mar 2026 00:00:00 +0000

When a codebase grows to 21,000 functions connected by 35,000 call edges, code review alone isn’t enough. We ran OpenClaw through NeuG’s graph database and surfaced structural problems that traditional tools can’t see.

Why a Graph Perspective?

OpenClaw is a popular AI gateway project — 18,000+ commits, 6,000+ TypeScript files, and 21,000+ functions, with new versions shipping almost daily. In our previous post, we looked at OpenClaw from the user’s perspective and found that the heartbeat mechanism was consuming far more tokens than anyone expected.

This time, we go deeper. From a developer’s perspective: what structural debt has fast iteration quietly accumulated?

Traditional code review and static analysis tools catch a lot, but they struggle to answer structural questions:

Which functions are “time bombs” — where a bug would cause maximum blast radius?
Which code is dead weight, confusing new contributors who think it’s still active?
Which module is the system’s structural hub — touch it, and half the codebase shakes?

The answers to these questions live in relationships, not in lines of code.

The Approach: Turn the Codebase into a Graph

We built a code knowledge graph using NeuG (graph database) and zvec (vector database):

Nodes: functions, classes, modules, files
Edges: call relationships, import dependencies, commit history

Here’s what the graph looks like at scale:

Type	Count
Source files	6,062
Functions	21,057
Call edges	35,761
Import edges	25,883
Classes	233
Modules	318

21,000 functions connected by 35,000 call edges. Once the graph is built, hard structural questions become Cypher queries.

The full analysis pipeline is packaged as CodeGraph Skill — open source and fully reproducible.

Finding 1: High-Risk Functions

Defining Risk Score

We define a risk score for each function to quantify its potential blast radius:

Risk Score = fan_in × fan_out

fan-in: how many functions call it (dependency pressure)
fan-out: how many functions it calls (dependency complexity)

The product represents the blast radius — how much of the system breaks if this function has a bug.

Detecting fan-out is straightforward; most IDE tools can do it. But fan-in requires a full-codebase scan — exactly where a graph database shines. This is one of the built-in capabilities of our CodeGraph Skill: it ships with a hotspots() method that handles the full-graph traversal for you, so finding the highest-risk functions across the entire codebase is a single call:

functions = cs.hotspots(topk=30)

Top 5 by risk score:

Function	File	fan-in	fan-out	Risk Score
startGatewayServer	src/gateway/server.impl.ts	10	103	1030
createConfigIO	src/config/io.ts	18	56	1008
runEmbeddedPiAgent	src/agents/pi-embedded-runner/run.ts	14	67	938
loadOpenClawPlugins	src/plugins/loader.ts	20	36	720
runCronIsolatedAgentTurn	src/cron/isolated-agent/run.ts	11	60	660

These functions sit at the core of critical execution paths. startGatewayServer bootstraps the entire service; createConfigIO generates all configuration; runEmbeddedPiAgent orchestrates embedded Pi Agent execution.

Here’s an interesting observation about createConfigIO: configuration parsing rarely gets prioritized for test coverage — it “seems safe.” But with 18 callers and 56 outgoing calls, by structural risk metrics it’s one of the most dangerous functions in the entire codebase.

Filtering for the Real High-Risk Core

We wrote a custom Cypher query to filter out low-level utility functions (high fan-in, low fan-out) and focus on functions that are simultaneously heavily depended upon and highly complex:

MATCH (f:Function)-[:CALLS]->(g:Function)
WITH f, COUNT(DISTINCT g) as fan_out
MATCH (f)<-[:CALLS]-(h:Function)
WITH f, fan_out, COUNT(DISTINCT h) as fan_in
WHERE fan_in > 10 AND fan_out > 10
RETURN f.name, fan_in, fan_out
LIMIT 100

The result was surprising: only 17 functions in the entire codebase meet both criteria. Every one of them is a critical, high-risk node:

Function	File	fan-in	fan-out	Risk Score
startGatewayServer	src/gateway/server.impl.ts	10	103	1030
createConfigIO	src/config/io.ts	18	56	1008
runEmbeddedPiAgent	src/agents/pi-embedded-runner/run.ts	14	67	938
loadOpenClawPlugins	src/plugins/loader.ts	20	36	720
runCronIsolatedAgentTurn	src/cron/isolated-agent/run.ts	11	60	660
getReplyFromConfig	src/auto-reply/reply/get-reply.ts	20	24	480
runMessageAction	src/infra/outbound/message-action-runner.ts	21	22	462
loadPluginManifestRegistry	src/plugins/manifest-registry.ts	22	16	352
createOpenClawTools	src/agents/openclaw-tools.ts	10	24	240
fetchWithSsrFGuard	src/infra/net/fetch-guard.ts	24	10	240
resolveCommandSecretRefsViaGateway	src/cli/command-secret-gateway.ts	15	14	210
fetchRemoteMedia	src/media/fetch.ts	18	11	198
loadModelCatalog	src/agents/model-catalog.ts	18	11	198
resolveApiKeyForProvider	src/agents/model-auth.ts	13	15	195
start	src/gateway/client.ts	12	13	156
resolveAuthProfileOrder	src/agents/auth-profiles/order.ts	11	11	121
createClackPrompter	src/wizard/clack-prompter.ts	10	11	110

These 17 functions are the structural pressure points of the system. Any change to them deserves extra scrutiny.

Finding Zombie Functions

We plotted the fan-in distribution across all functions as a histogram. NeuG’s Python SDK makes this straightforward — query results flow directly into pandas or matplotlib. The distribution reveals that most functions have fan-in and fan-out values below 2, meaning OpenClaw’s code is largely linear: most functions sit in a simple “one-in, one-out” position in the call chain.

What stands out: over 20% of all functions have fan-in = 0 — they are never called by anything. After filtering out legitimate entry points and framework hooks, more than 2,000 functions remain that are true zombie code.

Here’s a concrete example. assertPublicHostname in src/infra/net/ssrf.ts has zero callers in the current codebase. A git bisect reveals:

Commit 5bd550: the function is introduced and actively used
Commit b62355: its logic is migrated to resolvePinnedHostname
The old function is never deleted — likely out of caution — and silently becomes a zombie

The real harm isn’t the wasted lines. It’s what happens next: a new contributor finds the function, assumes it’s still relevant, builds something on top of it, and then spends hours debugging a dead code path.

In a project with OpenClaw’s pace of change, a 20% zombie function rate is a significant cognitive tax on everyone who reads the code.

Finding 2: The Over-Coupled Module

We analyzed the 20 most recent bug reports. Of 57 root-cause function candidates, 24 — 42% — pointed to src/agents.

To understand why, we ran a Cypher query to inspect agents’ call relationships with the rest of the system:

MATCH (m1:Module {name: 'agents'})<-[:BELONGS_TO]-(f1:File)-[:DEFINES_FUNC]->(func1:Function)
MATCH (func1)-[:CALLS]->(func2:Function)<-[:DEFINES_FUNC]-(f2:File)-[:BELONGS_TO]->(m2:Module)
WHERE m2.name <> 'agents'
RETURN m2.name, count(*) as call_count
ORDER BY call_count DESC
LIMIT 10

The numbers below show bidirectional call counts between modules (a→b = calls from module a into module b):

agents <-> reply              a→b=19    b→a=117
agents <-> infra              a→b=108   b→a=15
agents <-> pi-embedded-runner a→b=19    b→a=88
agents <-> tools              a→b=38    b→a=55
agents <-> plugins            a→b=45    b→a=40
agents <-> gateway            a→b=9     b→a=60
agents <-> src                a→b=35    b→a=31
agents <-> models             a→b=1     b→a=63
agents <-> sessions           a→b=46    b→a=5
agents <-> auth-profiles      a→b=24    b→a=12

In total, 30+ modules have bidirectional dependencies with agents. The reply module calls into agents 117 times; pi-embedded-runner calls it 88 times; models calls it 63 times. The vast majority of the system’s business logic flows through this one module.

We visualized the reply->agents call relationships using neug-ui:

agents is the structural hub of the system. A single-line change there can ripple across half the codebase. The fact that 42% of bugs trace back to this module isn’t bad luck — it’s an architectural consequence.

What Graph Analysis Reveals That Other Tools Don’t

Code review catches function-level issues. Static analysis handles syntax and import-level concerns. Graph analysis fills the gap between them — cross-file, cross-module structural problems:

Traditional tools can tell you:

Which line has a syntax error
Which function has high cyclomatic complexity

NeuG can tell you:

How many functions are affected by a single change
Which code is dead but still confusing contributors
Which module is the structural load-bearing wall of the system

When a codebase outgrows what human reviewers can hold in their heads, structural problems can only be found through structural queries.

Three Recommendations for OpenClaw

Based on this analysis, the three highest-leverage improvements:

Decouple src/agents: With 30+ modules in bidirectional dependency, start by defining clear interface boundaries between agents and its heaviest callers (reply, pi-embedded-runner, models). Even partial decoupling would significantly reduce blast radius.
Prioritize test coverage for high-risk functions: Functions like createConfigIO and startGatewayServer are structurally critical but likely undertested. A bug there propagates everywhere. Cover them first.
Clean up zombie functions: 2,000+ dead functions are actively misleading contributors. Running a graph-based dead code analysis and removing confirmed zombies is a high-ROI cleanup with low risk.

Analysis powered by NeuG graph database and CodeScope code analysis engine. Full index completed in approximately 4 minutes.

OpenClaw Users: Why Is Your Token Budget Disappearing Out of Nowhere?

Wed, 18 Mar 2026 00:00:00 +0000

Turns out, heartbeat has been quietly eating your tokens all along.

Background

OpenClaw has become one of the hottest open-source AI gateway projects. With 18,000+ commits and a rapidly growing community, it connects messaging platforms — Telegram, Discord, Slack, MS Teams, Lark, Matrix, and more — to various LLM providers. For many teams building AI-powered chatbots and agents, OpenClaw has become the go-to solution.

But as adoption grows, so do user complaints — and one issue stands out: token consumption is way higher than expected. Users report mysterious token budget drains, conversation histories that grow without bound, and system prompts bloating to 29K characters. Some hit the 200K token limit repeatedly, even with lightContext: true configured. These aren’t edge cases — they’re common pain points affecting real users and real budgets.

What’s going on? Is it a misconfiguration? A model issue? Or something deeper in the codebase itself?

To find out, we analyzed the entire OpenClaw codebase using NeuG (graph database) + zvec (vector index), producing a code knowledge graph with 21,057 functions and 35,761 call edges. What we discovered was surprising: one of the biggest token consumers is hiding in plain sight — the heartbeat mechanism.

In this article, we start from the end-user perspective and trace how a supposedly “lightweight” timer became a token-eating machine. Follow-up articles will cover the extension developer and core maintainer perspectives.

The Problem: Session History That Never Stops Growing

If you are using OpenClaw and experiencing any of the following:

Conversation history that grows indefinitely without being truncated
A system prompt that bloats unexpectedly (one user reported 29K characters)
Repeatedly hitting the model’s 200K token limit
lightContext: true configured in HEARTBEAT.md, but with no noticeable effect

This is likely not a misconfiguration or a model issue. One easily overlooked cause is OpenClaw’s heartbeat mechanism.

What Is Heartbeat Secretly Doing?

OpenClaw’s heartbeat is a periodic timer. By design, it is supposed to be a lightweight tick that keeps the agent alive. We traced its call chain using CodeScope, and the results were surprising.

CodeScope provides two retrieval methods: text search (string matching) and semantic search (vector similarity). For a well-named codebase like OpenClaw, keyword search is usually sufficient to locate the core code. We query all functions containing heartbeat:

MATCH (f:Function)
WHERE f.name CONTAINS 'heartbeat' OR f.name CONTAINS 'Heartbeat'
RETURN f.name, f.file_path
ORDER BY f.file_path

The query returns 68 functions spread across 10 modules — infra, auto-reply, gateway, config, cron, and others. We visualized the results using neug-ui:

The infra module has the heaviest heartbeat presence, and most of it is concentrated in heartbeat-runner.ts.

This is where conventional approaches stop. Once you have the 68 functions and the core file, the usual next step is to feed the code to an LLM and ask it to explain. That process consumes tokens itself, and what you get back is a text description — not a queryable structure. You cannot directly ask: “Who calls whom?”, “How deep does the call chain go?”, “Which function is the structural center?”

Graph analysis takes a different approach: instead of reading code content, we model the function-to-function call relationships as a graph, turning structural questions into graph queries.

Step 2: Find the Core Entry Point via the Call Graph

We build a connectivity graph from the call relationships among the 68 functions and group them by connected component. The largest component contains 45 functions, indicating they form a tightly coupled core module. The remaining 19 components average just 1.2 functions each — mostly standalone interface or configuration functions.

We then ran centrality analysis on the largest component: runHeartbeatOnce connects 27 functions within 3 hops. The second-ranked function connects only 6. The gap is decisive: runHeartbeatOnce is the core entry point of the entire heartbeat mechanism. Results below:

Step 3: Expand the Call Chain — What Does It Actually Touch?

We expand the multi-hop outgoing edges of runHeartbeatOnce to find all context-loading functions it directly or indirectly calls:

// Find context-loading functions in the runHeartbeatOnce call chain
MATCH (f:Function {name: 'runHeartbeatOnce'})-[:CALLS*1..3]->(t:Function)
WHERE t.name CONTAINS 'Context'
   OR t.name CONTAINS 'Workspace'
   OR t.name CONTAINS 'Session'
   OR t.name CONTAINS 'Prompt'
RETURN t.name, t.file_path
ORDER BY t.file_path

Results:

Every heartbeat tick triggers all of these functions — loading the workspace directory, building a full session context, generating a complete prompt. runHeartbeatOnce has 15 direct callees, each involved in parsing heartbeat configuration and assembling context. This is the structural root cause of heartbeat being far heavier than its design intended.

Why Doesn’t `lightContext: true` Help?

You may have set lightContext: true in HEARTBEAT.md, expecting heartbeat to load only a lightweight context.

lightContext is a configuration flag, but it cannot change the structure of the call chain. As long as the dependency tree of heartbeat remains intact, every tick must load the workspace, build the session context, and traverse multiple modules. This is a path determined by the code structure — it cannot be bypassed by a flag.

We also analyzed the modification frequency of the most recent 200 commits: 4 out of 21 functions in the context-pruning module (responsible for context trimming) have been repeatedly modified recently. This indicates the OpenClaw team is aware of the context bloat problem and is actively working on a fix — but a stable solution is not yet available.

CodeGraph: Reproduce This Analysis Yourself

We have packaged the entire analysis workflow into a CodeGraph Skill, published inside NeuG. You can use it to reproduce every query in this article.

Prerequisites

CodeScope requires Python 3.10+ and PyTorch 2.4+.

pip install codegraph-ai

Environment variables:

# Use HuggingFace offline mode if you have already downloaded the models
export HF_HUB_OFFLINE="1"
# Set the database directory
export CODESCOPE_DB_DIR="/path/to/your/project/.codegraph"

Building the Index

Index the OpenClaw repository:

# Build the index (first run)
codegraph init --repo /path/to/openclaw --lang auto --commits 100
# Check index status
codegraph status --db $CODESCOPE_DB_DIR

Sample output:

============================================================
CodeScope Index Summary
============================================================
  Files:      7083  (+ 5774 external)
  Functions:  24173  (+ 0 historical)
  Call edges: 41269
  Vectors:    24173
  Imports:    28877
  Classes:    255
  Modules:    380

Once indexed, you can query the graph via CLI or Python SDK.

CLI

Check the database status:

codegraph status --db $CODESCOPE_DB_DIR

Sample output:

============================================================
CodeScope Index Status: /path/to/.codegraph
============================================================

Graph:
  File        :     12,857
  Function    :     24,173
  Class       :        255
  Module      :        380
  Commit      :        100

Edges:
  CALLS       :     41,269
  TOUCHES     :        605
  MODIFIES    :          0

Backfill: 0/100 commits have MODIFIES edges
  100 commits still need backfill

Vectors: 24,173 function embeddings

Natural language query:

codegraph query "Who calls runHeartbeatOnce?" --db $CODESCOPE_DB_DIR

Output:

Question type: structural
Retrieved 6 evidence items in 76ms:

[1] (caller) executeJobCore (src/cron/service/timer.ts) — hop=1, relevance=0.000
[2] (caller) createGatewayReloadHandlers (src/gateway/server-reload-handlers.ts) — hop=2, relevance=0.000
[3] (caller) startGatewayServer (src/gateway/server.impl.ts) — hop=2, relevance=0.000
[4] (caller) executeJob (src/cron/service/timer.ts) — hop=2, relevance=0.000
[5] (caller) executeJobCoreWithTimeout (src/cron/service/timer.ts) — hop=2, relevance=0.000
[6] (caller) buildGatewayCronService (src/gateway/server-cron.ts) — hop=1, relevance=0.000

Generate a full architecture report with multi-dimensional analysis:

codegraph analyze --db $CODESCOPE_DB_DIR --output architecture-report.md

Python SDK

For more complex queries — such as the multi-hop call chain traversal used in this analysis — call the CodeGraph Python API directly, using runHeartbeatOnce as an example:

import os
os.environ['HF_HUB_OFFLINE'] = '1'
from codegraph.core import CodeScope

cs = CodeScope(os.environ['CODESCOPE_DB_DIR'])

rows = list(cs.conn.execute('''
    MATCH (f:Function {name: 'runHeartbeatOnce'})-[:CALLS*1..3]->(t:Function)
    WHERE t.name CONTAINS 'Context'
       OR t.name CONTAINS 'Workspace'
       OR t.name CONTAINS 'Session'
       OR t.name CONTAINS 'Prompt'
    RETURN t.name, t.file_path
    ORDER BY t.file_path
'''))
for r in rows:
    print(f"{r[0]} @ {r[1]}")

cs.close()

We will continue to publish more analysis angles on OpenClaw. If you want to explore further or try your own queries, give CodeGraph a try.

What You Can Do Now: Four Practical Mitigations

1. Treat heartbeat as a heavyweight operation

Do not assume heartbeat is lightweight. When estimating token budgets, count every heartbeat tick as a real consumption event. Do not rely on lightContext: true to save tokens.

2. Reduce heartbeat trigger frequency

If your use case does not strictly depend on real-time heartbeat behavior, lowering the trigger frequency is the most direct and effective way to reduce token consumption. Fewer ticks = fewer full context loads.

3. Manually cap session history length

Do not rely on OpenClaw to auto-truncate conversation history. Explicitly set a maximum history count in your configuration to prevent unbounded accumulation.

4. Monitor for periodic token spikes

Enable token usage monitoring on the LLM provider side. If you observe periodic, clock-like token spikes (rather than growth correlated with conversation length), heartbeat is likely triggering full context loads on schedule.

Summary

We dissected heartbeat across three analytical layers:

Function distribution: 68 heartbeat-related functions spread across 10+ modules, with the heaviest concentration in heartbeat-runner.ts
Call graph centrality: runHeartbeatOnce is the core entry point — 15 direct callees, 27 functions reachable within 3 hops
Call chain expansion: every tick loads the workspace directory and builds full session context, with a load comparable to processing a complete user message

Together, these findings explain one observation: heartbeat is designed as a lightweight timer, but its actual execution path is structurally equivalent to a full message-processing pipeline. lightContext: true cannot override this. Until an official fix is available, reducing heartbeat trigger frequency and manually capping session history length are the most effective mitigations.

This analysis is based on a scan of the OpenClaw codebase using NeuG graph database and zvec vector index.

AI-Powered Open Source Development Management: NeuG's Exploration and Practice

Thu, 12 Feb 2026 00:00:00 +0000

NeuG is a lightweight, high-performance embedded graph database open-sourced by the GraphScope team. It can be installed with a simple pip install neug and is designed for local analytics and real-time transaction processing. In this article, we share how we leverage AI capabilities to manage NeuG’s development workflow. In the future, we will further explore how NeuG’s graph computing capabilities can empower AI in return.

We invite you to follow and star NeuG repository!

For more technical details, please refer to the NeuG official documentation.

Pain Points in Open Source Development: Lack of Comprehensive Project Management

Managing open source software projects is one of the biggest pain points in the development process. If an engineer only focuses on writing code without an efficient management and tracking mechanism, the final product will struggle to be delivered on time. This is because open source projects inherently possess dynamic uncertainty — requirements can change at any moment.

In recent months, AI Coding capabilities have been rapidly improving. However, we have found that the imbalance between development and management has actually worsened: developers can leverage the latest and most powerful models to rapidly generate large volumes of code, but this seemingly functional yet poorly readable and architecturally weak code only adds to the project management burden. As code accumulates, it may well be that only the large language model itself can deconstruct such a complex repository.

In reality, we don’t lack coding capability. On the contrary, in open source projects, designing and breaking down solutions and tracking development progress are far more important. Taking our team’s NeuG development as an example, the upfront PRD design and discussion can take up a third or even more of the entire development cycle. The recently popular “vibe-coding” is really only suitable for personal projects built from scratch — for serious development scenarios like databases, it is virtually inapplicable.

Based on these reflections, we tried introducing AI tools into NeuG development in the role of “manager” rather than “developer,” covering the entire lifecycle from requirements analysis and task decomposition to GitHub synchronization and tracking.

Spec-Driven: The Best Development Paradigm for NeuG

In our early exploration, we still followed a “vibe-coding”-style workflow, letting AI directly manage requirements. We iterated through multiple versions:

Developers write PRD documents, and the Coding Agent parses the context to decompose tasks, syncing them to GitHub.
The Coding Agent parses the Markdown structure of PRD documents before decomposing tasks.
The Coding Agent converts PRD documents into a standardized intermediate format before parsing and decomposing tasks.

Throughout this exploration, we continuously encountered common pain points such as detail loss from overly long contexts and inconsistent outputs. However, we gradually realized that a structured context might be the paradigm best suited for NeuG development. Many AI products also adopt similar structured contexts — for example, DingTalk’s meeting minutes allow pre-selecting templates, and some project documentation tools offer fully customized templates. After testing, we found that while enforcing output format alignment makes some content read a bit stiffly (for example, some development tasks simply don’t need corresponding unit tests), the completeness of such documents effectively reduces omissions — trimming is always easier than patching.

Meanwhile, a wealth of Spec-Driven tools have been open-sourced, and we were among the first to try them. We found that the paradigm of “write project specifications first, then generate project code” highly aligned with our needs. Taking GitHub’s official speckit as an example, here’s a brief explanation of what Spec-Driven involves and what each step does:

Specify. In this phase, users provide a requirements description. This phase doesn’t involve specific technology stacks but focuses on product-level analysis: Who will use it? What features does it provide? What problems does it solve? What are the inputs and outputs? What is the interaction flow?
Plan. In this phase, the AI tool generates a comprehensive development plan. This phase focuses on designing how to incorporate the requirement into the existing codebase, which modules are affected, and what constraints must be maintained. It therefore requires comprehensive code, architecture, and constraint information, with strict confirmation from development experts.
Tasks. In this phase, the AI tool further decomposes the Plan into several executable tasks. Each task is a minimal, independently runnable code unit, making it convenient for the AI tool to test after code completion. Simply put, each task can be viewed as a standalone function.
Implement. In this phase, the AI tool incrementally completes each task from the Tasks phase. Thanks to the well-prepared Specify, Plan, and Tasks, the AI tool can generate more accurate code that integrates better with the existing context.

Additionally, these Spec-Driven tools interact with the Coding Agent through slash commands. This invocation style is completely transparent and flexible, and users can freely adjust the prompts in Markdown files to create their own specifications. As we encountered the problem of overly verbose generated content, we began migrating our daily development habits into prompts, which significantly improved results. Below, we use a simple example to illustrate how NeuG leverages the Spec-Driven development paradigm to improve efficiency.

Use Case: Adding Multi-threaded Transaction Tests to NeuG

Background: Transaction functionality is one of the most important — and most error-prone — modules in a database. NeuG’s transaction engine implements lock-free read-insert parallelism and serializable isolation based on MVCC (see NeuG Transaction Mechanism Explained for details). To ensure this module executes correctly, we need to add various transaction tests. We attempted to use the Spec-Driven approach to let the large language model handle the entire process.

Specify Phase: Task Expansion and Analysis

In the Specify phase, users can provide a brief requirements description along with some draft documentation (no format or content requirements), and the model independently organizes, expands, and generates a complete PRD document.

In NeuG’s previous development workflow, PRD document design was a crucial step. However, writing a complete PRD document was extremely tedious and difficult to account for all edge cases and special scenarios. During the model’s document generation process, we enforced a document structure where a requirement is split into multiple modules, each module is further decomposed into multiple tasks, and each task must include corresponding tests and verification to ensure document completeness.

Even when a task genuinely doesn’t need testing or verification (for example, cycle detection can directly call a third-party library), we still ask the model to list relevant content whenever possible. If some redundant content truly needs to be removed, we can simply edit it ourselves or prompt the model to modify the document. From practical experience, “trimming excess” is always easier than “filling gaps.”

Plan Phase: Persisting Implementation Details

The Plan phase is primarily responsible for further refining technical details and technology choices that weren’t covered in the Specify phase.

During NeuG’s product development, we found that many design decisions are difficult to reflect intuitively in code, and the related technical documentation often lacks maintenance, directly leading to collaboration and handoff difficulties. As a solution, we chose to persist extensive technical details during the Plan phase. This content effectively serves as supplementary context for the Coding Agent’s code generation and provides convenience when producing technical reports later.

As shown in the figure below, we needed to standardize transaction generation ratios and data formats, and determine transaction execution strategies for multi-threaded scenarios. Additionally, transaction testing relies on specific testing workflows, requiring detailed descriptions of dependency graph construction, dependency analysis, and cycle detection algorithms. This content significantly improves the accuracy and consistency of subsequent code generation.

Tasks Phase: Decomposing Tasks, Syncing to GitHub

In the Tasks phase, the Coding Agent generates several Modules and Tasks based on the previous documents and syncs them to GitHub Issues. Our hierarchical structure perfectly aligns with GitHub’s Sub-Issue feature, making it convenient to track the entire development progress of a requirement end-to-end.

In NeuG’s past development, these Issues were created and linked manually — not only time-consuming and labor-intensive, but when a new requirement was inserted, older requirements were often overlooked. Through the Task phase tracking, we can not only generate Issues with one click but also persistently track the completion status of all Issues, preventing development tasks from stalling midway.

Furthermore, we customized our approach to NeuG’s development habits by creating separate Markdown files for each Module, storing detailed information about each task’s assignee, labels, and implementation details. This isolation ensures that when a requirement spans multiple parts of the project and requires collaboration from multiple developers, conflicts are avoided.

GitHub Sync Operations

In the Tasks phase, we need to sync tasks to GitHub for tracking. Specifically, multiple Module Issues are created under the main Issue, and each Module Issue further contains multiple Task Issues. The overall structure is consistent with the documentation and can be displayed very intuitively on GitHub.

We also designed a set of synchronization rules tailored to our team’s development habits: initially, the entire Module Content is synced to the Issue body, but no specific Sub-Issues are created. Only when a specific task is about to be developed is the corresponding Issue created, and upon completion, the PR is merged and the Issue is closed. The reason for this design is that the overall plan continuously adjusts as development progresses — new requirements may be inserted or existing ones modified. Fixing the entire workflow at once would lead to high modification costs later.

The figure below shows a development snapshot of the transaction testing requirement described above (displayed content differs slightly). We planned two Modules, each requiring 3 Tasks to complete. Currently, Tasks 101, 102, 201, and 202 have been completed, while the remaining two Tasks involve complex multi-query scenarios that require further support before implementation.

To accommodate this usage pattern, we designed two separate Commands: sync-module and sync-task. They accept a Module ID or Task ID and generate the corresponding Issue, automatically completing the Sub-Issue linking. When the content of these Modules or Tasks is modified, the same two commands can be reused to update without needing to recreate.

Implement Phase: Preserving Traditional Development

In this phase, we don’t mandate the use of /speckit.implement. Instead, we allow the task to be handed off to the corresponding developer for implementation. It doesn’t matter which tool they use to generate code (or even “old-school manual coding”), because the comprehensive context and specifications already provide sufficient constraints to naturally produce high-quality code.

We also recognize the importance of testing, requiring that all Tasks must be supplemented with corresponding CI tests after implementation, ensuring they can execute correctly in an isolated test environment. This is also an area where current large language models excel, so we won’t elaborate further.

GitHub Integration: A More Automated Issue/PR Experience

Inspired by the Commands functionality, we further integrated more GitHub operations into the Coding Agent. Currently, we have implemented two core capabilities: Issue creation and PR creation. In these two Commands, we incorporated two very important paradigms:

Context Collection

Context is critically important for Bug Issues. While Issue and PR creation already encourage users to fill in context thoroughly using templates, this step is extremely tedious. Our developers often need to copy terminal error messages, relevant code snippets, and error reproduction commands multiple times — and the useful information in these logs is minimal with poor readability.

We introduced the create-issue and create-pr commands. Users can directly select relevant content, and the model reads the corresponding terminal or file content, automatically parsing and generating organized content for submission. Developers only need to review the final generated result, which is extremely convenient.

Below is a simple example: by referencing error messages from the terminal and relevant source code, we prompt the model to submit an Issue and automatically configure the Parent Issue, Assignee, and Project information. In the era of large language models, this information doesn’t need to match exactly — a rough description is sufficient for the model to infer the best match.

`/create-issue` [bash:15-20][main.cpp:150:210] Check the error messages and executed code, create an issue of type bug, link to parent issue #xx, assign to "@user-xxx", and add to project "Project-xxx".

Creation Confirmation

When Git performs dangerous operations like merge or rebase, it typically displays a confirmation file for the user to review. Inspired by this, we believe that creating Issues and PRs can similarly incorporate a confirmation flow to verify Issue configurations.

Specifically, before creating an Issue or PR, the Agent provides a file for user interaction to confirm the information to be submitted, with a structure roughly as follows:

Issue Type: Bug
Issue Title: [BUG] <title>
Assignee: <assignee>
Labels: bug, <additional labels>
Project: NeuG v0.1
Parent Issue: #<parent-issue-id>

**Describe the bug**
A clear and concise description of what the bug is.

**Execution Logs**
The commands that can reproduce the issue.
1. Command 1: ...
2. Command 2: ...
...

**Expected behavior**
A clear and concise description of what you expected to happen.

**Error Message**
The error message in the terminal.

In this temporary file, the configuration parameters for creating the Issue are listed in detail, along with the specific body content. Through this file, users can clearly confirm the Issue’s type, assignee, labels, project, parent issue, and other information. Only after user confirmation does the Agent create and execute the corresponding command. This semi-structured template also ensures greater consistency in subsequently generated commands, significantly reducing unexpected situations caused by users’ unfamiliarity with commands.

Agent Skills Support

Just as we were writing this blog post, another new paradigm — Agent Skills — was rolling out broadly. We won’t discuss the comparison between Commands and Skills here, but from a product design perspective, the advantage of Agent Skills lies in encouraging users to store large scripts, templates, and other information in separate folders, while keeping only concise but critical descriptions in the core file. Moreover, Agent Skills is fully compatible with Commands operations, invoked explicitly through slash commands. Therefore, we promptly migrated our commands to the Skills format, moving templates, scripts, and other content from prompts into independent files.

.cursor
└── skills
    ├── speckit.specify
    │   ├── templates
    │   │   └── spec-template.md
    │   └── SKILL.md
    │
    ├── speckit.plan
    │   ├── templates
    │   │   └── plan-template.md
    │   └── SKILL.md
    │
    ├── speckit.tasks
    │   ├── templates
    │   │   ├── tasks-metadata-template.md
    │   │   └── tasks-module-template.md
    │   └── SKILL.md
    │
    ├── sync-modules
    │   └── SKILL.md
    │
    ├── sync-tasks
    │   └── SKILL.md
    │
    ├── create-issue
    │   ├── scripts
    │   │   └── gh-update.sh
    │   ├── templates
    │   │   ├── bug-issue.md
    │   │   └── feature-issue.md
    │   └── SKILL.md
    │
    └── create-pr
        ├── templates
        │   └── pull-request.md
        └── SKILL.md

Conclusion

Throughout our experience with Spec tools, it became evident that using Coding Agents effectively involves a wealth of techniques — from the overall product application scenarios to the crafting of individual prompts, as well as various interaction paradigms and generation logic. These details are what truly unlock the model’s capabilities, and they explain why the same programming tools produce entirely different results in different people’s hands. In the future, we will continue exploring applications of large language model programming tools across various project development scenarios, letting AI truly unleash productivity.

Opensourcing NeuG

Wed, 11 Feb 2026 00:00:00 +0000

We are pleased to announce the open source release of NeuG (pronounced “new-gee”), a lightweight, high-performance embedded graph database designed for local analytics and real-time transaction processing.

GitHub: https://github.com/alibaba/neug
Documentation: https://graphscope.io/neug/en/overview/introduction/

Why NeuG

The GraphScope team has spent the past few years building large-scale graph computing engines. Along the way, we received consistent feedback from our community: many users don’t need distributed clusters. What they want is a graph database they can pip install, explore data in Jupyter notebooks, run graph queries in Python scripts, and quickly spin up as a service when needed for production.

This reminded us of what DuckDB did for relational data processing—an embeddable analytics database that doesn’t require deploying servers or configuring connection pools. Just import duckdb and start working. We wanted to bring that same experience to graph data.

NeuG is the result of that effort. It reuses the storage and query engine from GraphScope Flex (which achieved strong results on the LDBC SNB Interactive benchmark), but with a redesigned interface layer that allows it to be embedded directly as a Python library. We also kept the db.serve() interface for switching to service mode when concurrent access is needed.

NeuG is suited for scenarios where you need to process graph data locally and quickly: data science exploration, ML feature engineering, AI application prototyping, and lightweight applications where deployment simplicity matters.

Key Features

Lightweight & Embeddable

Lightweight, all dependencies managed via third_party submodules
Embeddable design, currently supporting Python applications with more languages and platforms coming soon
Get started with pip install neug

import neug

db = neug.Database("/path/to/data")
conn = db.connect()

result = conn.execute("""
    MATCH (a:Person)-[:KNOWS]->(b:Person)
    RETURN a.name, b.name
""")

Dual-Mode Architecture (HTAP)

NeuG provides two operational modes through a single lightweight core:

Mode	Use Case	Characteristics
Embedded Mode	Offline analytics, ML/AI pipelines	Import as a Python library, ideal for Jupyter notebooks, batch ETL, graph algorithm development
Service Mode	Online transactions, concurrent access	Call `db.serve()` to start a network service with multi-session ACID transactions

This design allows NeuG to adapt flexibly from prototyping to production deployment without changing your technology stack.

Cypher-Native

Industry-standard Cypher query language
Built on GOpt’s unified intermediate representation, designed for future ISO/GQL compatibility

Extensible by Design

Extension system inspired by PostgreSQL/DuckDB
Keep the core lean; add graph algorithms, vector search, and custom procedures through an extensible framework

ACID Transactions

Embedded mode: Single-connection serial access with data consistency guarantees
Service mode: Multi-session concurrent transactions with read-write isolation

Performance

NeuG is built on the GraphScope Flex engine, which achieved industry-leading results on the LDBC SNB Interactive benchmark using Cypher queries:

Metric	Result
Throughput	80,000+ QPS
Scale Factor	SF300 (~1 billion edges)
Audit Status	Officially audited by LDBC

These results validate NeuG’s capability for high-concurrency transactional workloads. The full audit report is available on the LDBC website.

Use Cases

NeuG v0.1 is particularly suited for:

AI/LLM Application Development: Knowledge graph storage and querying for RAG systems and AI Agents
Data Science & Research: Graph exploration in Jupyter Notebooks—social network analysis, relationship mining, pattern discovery
ML Feature Engineering: Computing graph-structural features (node centrality, community structure, path patterns) as machine learning inputs
Prototype to Production: Use embedded mode for rapid iteration during development, switch to service mode for deployment—no technology stack changes required
Edge Computing & Local-First Applications: Run graph queries in resource-constrained environments without network connectivity

Quick Start

# Installation
pip install neug

# Verify installation
python -c "import neug; print('NeuG is ready!')"

import neug

# Create database and load sample data
db = neug.Database("./my_graph")
conn = db.connect()
db.load_builtin_dataset("tinysnb")

# Execute queries
result = conn.execute("""
    MATCH (a:person)-[:knows]->(b:person)-[:knows]->(c:person),
          (a)-[:knows]->(c)
    RETURN a.fName, b.fName, c.fName
""")

for record in result:
    print(f"{record} are mutual friends")

# Switch to service mode
conn.close()
db.serve(port=8080)

Coming Soon (v0.2)

v0.2 focuses on enhanced AI scenario support and data ecosystem integration:

Feature	Description	Use Case
Node.js Binding	TypeScript/JavaScript language binding	AI Agent development, web backend integration
Graph Algorithm Extensions	Leiden community detection and more	Knowledge graph clustering, entity grouping in GraphRAG
Vector DB Extension	Graph + vector hybrid retrieval	GraphRAG, semantic search + relationship reasoning
Data Lake Integration	S3/OSS + Parquet format	Large-scale offline analytics, data platform integration
Foreign Tables	Direct querying of external data sources	Interoperability with PostgreSQL, DuckDB, and relational ecosystems

Technology Stack

Query Language: Cypher (ISO/GQL compatibility planned)
Query Optimization: GOpt unified optimization framework
Storage Engine: GraphScope Flex
Language Bindings: Python (C++ API available, Node.js in development)
Platform Support: Linux, macOS (x86_64, ARM64)

License

NeuG is released under the Apache License 2.0.

Contributing

We welcome community participation and contributions:

Star & Watch: https://github.com/alibaba/neug

Acknowledgments

NeuG is developed by the GraphScope team at Alibaba, incorporating years of expertise in large-scale graph computing. We thank all developers who have contributed to this project.

Making graph data simple.

GitHub: https://github.com/alibaba/neug
Documentation: https://graphscope.io/neug/en/overview/introduction/

NeuG's High-Performance MVCC for Graph Workloads

Wed, 11 Feb 2026 00:00:00 +0000

This article provides an in-depth look at NeuG’s transaction mechanism, which powers its high-throughput Transactional Processing (TP) capabilities while maintaining serializable isolation guarantees.

NeuG Repository: https://github.com/alibaba/neug

Executive Summary

NeuG is an embedded graph database supporting both Transactional Processing (TP) and Analytical Processing (AP) workloads through distinct execution models.

Operational Models & Concurrency

Embedded mode: Analytical queries run in embedded mode with concurrency managed at the Connection level. Within a given AP Connection, all queries (reads or writes) are executed serially. This model does not utilize the fine-grained parallel MVCC mechanism detailed for TP.
Service mode: High-throughput TP operations are achieved via a single Write Connection managed by an internal service layer. This connection spawns multiple worker threads that execute in parallel, utilizing the MVCC and version management system described herein to enable lock-free read-insert parallelism and serializable isolation.

Scope: This document details the transaction mechanism powering the TP execution model. AP mode uses a simpler, connection-serialized approach for consistency.

Core Architecture & Design Philosophy

The transaction design is driven by the need for maximal throughput on common graph operations. Recognizing that a dominant pattern is highly concurrent reads mixed with blind inserts, the system categorizes transactions to eliminate unnecessary synchronization. A global versioning system creates immutable snapshots for readers, while inserters proceed in parallel, coordinated only when modifying the same adjacency list.

Transaction Classification

NeuG categorizes transactions into three types to optimize concurrency:

Type	Purpose	Key Property	Concurrency
Read	Graph traversal, point lookup	Pure read; sees a versioned snapshot	Parallel with all Inserts via MVCC
Insert	Add vertices/edges	Blind write; does not read existing graph	Parallel with all Reads; fine-grained locks for edge adjacency lists
Update	Complex CRUD	Sees its own writes; can abort/rollback	Currently exclusive; future two-phase design

Focus: This article details the mechanisms for Read and Insert transactions within the TP worker pool.

Version Management: The Foundation of Consistency

NeuG maintains a global, monotonically increasing version timeline through two counters:

Global Write Version (GWV): Atomic counter assigning unique IDs to write transactions
Global Read Version (GRV): The highest version where all prior writes are guaranteed committed

Transaction Version Assignment

Read Transaction: Starts with the current GRV as its snapshot version. GRV remains unchanged.
Insert Transaction: Acquires the current GWV as its write version; GWV is atomically incremented.

GRV Advancement Logic

GRV advances only when a contiguous prefix of the version sequence is complete. A bit-set tracks completion within a sliding window.

The example above demonstrates how GRV advances only when a contiguous prefix of transaction versions have completed. Although v₄ was already committed in STATE 1, GRV could not advance beyond v₀ because v₁ was still pending. Once v₁ commits, the continuous sequence v₀-v₂ becomes complete, allowing GRV to advance to v₂.

MVCC Storage & Visibility

NeuG implements a lightweight MVCC model optimized for graph workloads:

Single-Version Model: Each vertex/edge stores exactly one write_version (its creation version)
Visibility Rule: A Read transaction with version Tₓ sees an element if and only if:
```
element.write_version ≤ Tₓ
```

This single-version approach simplifies storage and memory management while providing snapshot isolation for readers.

Concurrency and Isolation Guarantees

Read-Insert Parallelism

Lock-free parallelism between reads and inserts is achieved because:

No Read-Write Conflicts: Insert transactions perform zero reads on existing graph data
Non-Blocking Snapshot Reads: Reads operate on an immutable version snapshot

Write-Write Coordination

Vertex Insertion: Uniqueness is the caller’s precondition (assumed non-existent)
Edge Insertion: Fine-grained, per-vertex adjacency locks prevent corruption when concurrent transactions add edges from the same source vertex. Lock atomicity prevents deadlocks.

Serializable Isolation Guarantee

Reads: A single version snapshot is inherently serializable
Inserts: Linearized by GWV assignment and persisted via WAL (see below). No internal read-dependencies ensure a deterministic, serializable order.

Insert Transaction Lifecycle & Durability

The lifecycle of an Insert transaction follows a carefully designed protocol to ensure both performance and durability:

1. START & VERSION
   └── Assign GWV (e.g., v₇); atomically increment GWV to v₈

2. PREPROCESSING (Read-Only Phase)
   └── Validate IDs, schema, encode properties
   └── No graph modifications

3. COMMIT PERSISTENCE (Write-Ahead Log - WAL)
   └── Flush all mutation data to durable WAL
   └── This point defines durability
   └── On abort: log a minimal abort marker

4. ATOMIC APPLY
   └── Acquire necessary per-vertex adjacency locks
   └── Apply all vertex/edge inserts atomically, stamping with version v₇
   └── Update indexes

5. COMPLETION
   └── Mark transaction v₇ as complete in tracking bit-set
   └── Advance GRV if possible (e.g., from v₆ to v₇)

Recovery

After a crash, the system restores the last checkpoint and replays all committed WAL entries to guarantee durability. This design ensures that no committed transaction is lost while minimizing recovery time.

Performance Characteristics

The design choices in NeuG’s transaction mechanism deliver several performance benefits:

High Read Throughput

Zero Lock Contention: Read transactions never block on locks
No Version Chain Traversal: Single-version model eliminates multi-version lookup overhead
Snapshot Isolation: Readers see a consistent view without coordination

Efficient Insert Processing

Parallel Execution: Multiple insert transactions execute simultaneously
Fine-Grained Locking: Only edges sharing the same source vertex require coordination
Minimal Synchronization: No read-write conflicts to manage

Predictable Latency

Lock-Free Reads: Query latency is independent of write workload
Deterministic Version Assignment: Clear ordering of all transactions
WAL-Based Durability: Fast commit with background persistence

Design Trade-offs

Advantages

Optimal for Read-Heavy Workloads: Zero overhead for the common case of concurrent reads
Simple Implementation: Single-version storage reduces complexity
Strong Guarantees: Serializable isolation without complex protocols
Predictable Performance: Lock-free reads provide consistent latency

Current Limitations

Update Transactions: Currently serialized; future work will enable greater concurrency
Delete Operations: Handled through Update transactions
Long-Running Transactions: May delay GRV advancement temporarily

Use Cases

NeuG’s transaction mechanism is particularly well-suited for:

Real-Time Graph Analytics

Social Network Analysis: Concurrent queries on user relationships while ingesting new connections
Recommendation Systems: Read-heavy workloads with periodic batch inserts
Fraud Detection: Real-time pattern matching on continuously growing transaction graphs

Knowledge Graph Applications

RAG Systems: High-throughput entity lookups while incrementally building the knowledge base
AI Agents: Parallel knowledge retrieval with concurrent fact insertion
Semantic Search: Snapshot-consistent queries during graph expansion

Edge Computing

Local-First Applications: Embedded TP capabilities without distributed coordination
IoT Data Processing: Continuous sensor data ingestion with real-time queries
Mobile Applications: ACID guarantees in resource-constrained environments

Future Work

The NeuG team is actively working on several enhancements to the transaction system:

Two-Phase Update Transactions

Refining the Update transaction model to allow greater concurrency through:

Optimistic concurrency control
Conflict detection and resolution
Selective retry mechanisms

Distributed Transactions

Extending the mechanism for distributed deployment:

Multi-node version coordination
Distributed WAL replication
Cross-shard transaction support

Advanced Optimizations

Adaptive Concurrency Control: Dynamic adjustment based on workload patterns
Version Garbage Collection: Automatic cleanup of old versions
Intelligent Lock Management: Reducing lock contention for hot vertices

Implementation Details

For developers interested in the technical implementation:

Version Counter Design

class VersionManager:
    def __init__(self):
        self.gwv = AtomicCounter(0)  # Global Write Version
        self.grv = AtomicCounter(0)  # Global Read Version
        self.completion_bits = SlidingBitSet()
    
    def assign_write_version(self):
        return self.gwv.fetch_and_increment()
    
    def get_read_version(self):
        return self.grv.load()
    
    def mark_complete(self, version):
        self.completion_bits.set(version)
        self.try_advance_grv()

Adjacency Lock Protocol

class EdgeInserter:
    def insert_edge(self, src, dst, properties):
        # Acquire lock on source vertex's adjacency list
        with self.vertex_locks[src]:
            # Stamp with transaction version
            edge = Edge(src, dst, properties, self.tx_version)
            self.adjacency_lists[src].append(edge)

WAL Format

Each WAL entry contains:

Transaction ID (version)
Operation type (INSERT/UPDATE/ABORT)
Payload (vertices/edges with properties)
Checksum for integrity

Comparison with Other Approaches

vs. Traditional RDBMS MVCC

Feature	NeuG	Traditional RDBMS
Version Storage	Single version per element	Multi-version chains
Read Overhead	Zero lock contention	May need version traversal
Write Model	Optimized for blind inserts	General-purpose updates
Isolation Level	Serializable	Typically snapshot isolation

vs. Graph-Specific Systems

Feature	NeuG	Neo4j	JanusGraph
Concurrency Model	MVCC	Lock-based	Optimistic locking
Read Parallelism	Lock-free	Lock-based	Lock-free
Transaction Scope	Local (embedded)	Local	Distributed
Performance Focus	TP workloads	Mixed OLTP	OLAP-oriented

Conclusion

NeuG’s transaction mechanism delivers high-throughput serializable isolation for graph workloads by:

Categorizing transactions into Read and Insert types
Isolating them via MVCC and versioned snapshots
Synchronizing writes with fine-grained locking and a global version order
Guaranteeing durability with a WAL-based commit protocol

This design provides an optimal balance between performance, simplicity, and correctness for embedded graph database workloads. The system achieves lock-free read scalability while maintaining strong consistency guarantees—making it ideal for modern AI and analytics applications that require both high throughput and transactional integrity.

Learn More

GitHub: https://github.com/alibaba/neug
Documentation: https://graphscope.io/neug/en/overview/introduction/
Previous Post: Opensourcing NeuG

Building robust graph transactions for the embedded era.

New Breakthrough in Declarative Graph Benchmarking: GraphScope Flex Shatters LDBC SNB Interactive Benchmark World Record

Thu, 12 Jun 2025 15:03:00 +0000

The Linked Data Benchmark Council (LDBC) has released the latest results of its SNB Interactive Benchmark (via Declarative Queries), where GraphScope Flex achieved a historic breakthrough with a throughput exceeding 80,000 QPS (queries per second) – nearly twice the performance of the previous record holder!

LDBC is an internationally recognized authority in graph data and computing. Its Social Network Benchmark (SNB) simulates a Facebook-like social graph, covering CRUD operations, shortest-path queries, multi-hop traversals, and more, making it the industry’s gold standard for transactional online query performance evaluation. SNB supports two implementation approaches:

Imperative: Requires graph experts to manually write and optimize queries in programming languages like C++.
Declarative: Uses graph query languages like Cypher, with the system automatically optimizing execution.

In this benchmark, GraphScope Flex exclusively used declarative Cypher queries for all test cases, demonstrating not only its leadership in declarative query optimization and execution but also its full-stack technical prowess – from underlying storage to high-level optimization.

Full-Stack Optimization: Synergistic Advancements in Storage and Compute

GraphScope Flex’s record-breaking performance is built on innovations across storage, compute, and scheduling:

Efficient Storage Structure: Employs a compact adjacency list format with vertex/edge compression and memory layout optimizations, enabling larger graph datasets on the same hardware.
Intelligent Memory Management: Enhances traversal locality via prefetching and caching, significantly reducing memory access latency.
High-Concurrency Transaction Support: Leverages MVCC (Multi-Version Concurrency Control) for high-throughput queries while maintaining low-latency data modification.
End-to-End Vertical Optimization: Coordinated optimizations across storage, compute, and scheduling layers ensure peak performance.

A Milestone in Declarative Graph Query Optimization

Powered by its robust infrastructure, GraphScope Flex employs its in-house GOpt Optimization Framework to intelligently transform declarative queries into high-performance execution plans:

Rule-Based Optimization (RBO): Pluggble heuristic rules (e.g., filter pushdown, field trimming, join fusion) optimize query plans, minimizing intermediate results and boosting efficiency.
Automatic Type Inference: Dynamically deduces implicit type constraints in patterns, validating them against the data graph to avoid invalid traversals and improve cardinality estimation.
Cost-Based Optimization (CBO): Uses high-order statistics and backend-registered cost models to search for optimal execution plans via branch-and-bound strategies, balancing data characteristics and system-specific implementations.

The research behind GOpt has been accepted by SIGMOD 2025. Additionally, we’ve published a detailed paper on arXiv outlining GraphScope’s optimizations for the LDBC SNB benchmark (paper link).

Leading in Both Declarative and Imperative SNB Leaderboard

GraphScope currently holds the performance record for imperative queries in the LDBC SNB Interactive benchmark. This latest breakthrough in declarative queries further cements its technical leadership – its declarative performance even surpasses other players’ records in imperative scenarios. This achievement proves that GraphScope Flex’s auto-optimized declarative queries are not only highly efficient and user-friendly, but also exceed manually tuned imperative solutions, dramatically lowering the barrier to high-performance graph querying.

Acknowledgments

The GraphScope team gratefully acknowledges the research contributions of Prof. Cheng Li (University of Science and Technology of China) and Prof. Ying Zhang (Zhejiang Gongshang University). Their rigorous prototype validation and performance benchmarking were instrumental in enhancingthe system’s performance during this benchmark evaluation.

Why DuckDB Is Such A Good Database Product

Sat, 26 Apr 2025 00:00:00 +0000

TL;DR

Although databases are ancient, they still provide guidance for LLMs development.
DuckDB’s Success Formula: Insight into trends + relentless focus on core competencies + extremely good product.
Everyone agrees on “big data,” but not everyone needs “big data.”
“Big Data is Dead”, and large models are a better computational paradigm.
Be meticulous in initial technology selection, let data guide decisions, and avoid blind trust in authority.
Technology choices must always serve the product.
What appears “small and elegant” is actually the result of accumulated efforts in good products.

Introduction

It’s 2025, who still cares about databases? IT IS anyone who takes data seriously. Since their inception in the 1970s, relational databases have remained the backbone of data management. Despite waves of NoSQL, NewSQL, and other trends, the dominance of traditional databases and SQL remains unshaken. At SIGMOD 2023, Don Chamberlin, co-creator of SQL, delivered a keynote titled “49 Years of Queries”, reflecting on the evolution of relational databases and SQL over nearly half a century. Chamberlin emphasized that the longevity of database systems stems from E. F. Codd’s foundational theories, such as:

Database Normalization: Teaches structured data organization to eliminate redundancy.
Natural Language Queries: A vision pursued since 1974 with Codd’s “Rendezvous” project, which led to SQL.
Relational Algebra Closure: Ensures uniform data formats for chained operations.

Fast forward 55 years: large models dominate the AI landscape, unstructured data processing explodes, yet database theory remains deeply influential. Normalization principles guide large language model (LLM) training, inspiring slogans like “Data Quality is All You Need”. Natural language queries finally thrive with LLMs (though perhaps too enthusiastically). Developers of agent systems now recognize the importance of closure and standardization in ensuring stable, end-to-end execution.

In the ever-shifting tech world, databases stand out as a rare bastion of stability—flashy trends come and go, but foundational principles endure. For database products, “mastering the fundamentals” (correctness, reliability, efficiency) remains paramount. This doesn’t imply stagnation. Instead, strong foundations empower us to navigate trends confidently. Enter DuckDB, a database that marries trend awareness with technical depth. This article explores how DuckDB’s team built a competitive product by balancing innovation and foundation.

Market Trends

In 2023, MotherDuck founder Jordan Tigani’s blog “Big Data is Dead” sparked debate while crystallizing the rationale behind DuckDB’s embedded, lightweight, single-process design.

Debunking “Big Data Myths”—Three Truths

Most organizations don’t have “big” data. Investor surveys reveal that even large B2B companies manage mere terabytes, with many operating on gigabytes. Internal data from SingleStore and others shows core datasets often fit in single-digit gigabytes. For most, data scale isn’t the bottleneck.

(Source: Big Data is Dead)

Storage-compute imbalance. Modern architectures decouple storage and compute, but storage grows linearly while compute demand lags. Most analytics focus on recent data, leaving historical datasets underutilized. This “storage obsession” incurs maintenance costs for rarely accessed data.

(Source: Big Data is Dead)

Rapid data value decay. Business data often loses relevance within weeks. Historical data serves audits or model training, not daily analytics. “Big data” is a game for the 1%—most users operate within traditional single-machine capabilities.

(Source: Big Data is Dead)

The Scale-Up Revolution

The “big data” era emerged when scaling out (distributed clusters) was cheaper than scaling up (powerful single machines). Today, hardware advancements flip this equation:

Memory/storage leaps: DDR5 RAM hits 70GB/s bandwidth; NVMe SSDs approach DRAM latency.
Unified architectures: Apple’s M-series chips fuse CPU/GPU/NPU memory, slashing data movement overhead.

Result? Tasks once requiring Hadoop/Spark clusters now run efficiently on single machines. TPC-DS benchmarks show DuckDB outperforming Spark by 3-8x on 100GB datasets with 10x energy efficiency.

Large Models vs. Big Data

Another significant reason for the decline of big data analytics lies in the gradual transition from “big data” to “large models.” As Large Language Models (LLMs) become the new technological cornerstone, the importance of traditional “big data” stacks is being re-evaluated. Large pre-trained models essentially act as compression and distillation of massive datasets. Insights that previously required labor-intensive analysis of terabytes of data are now embedded within these models. When people increasingly obtain information by querying models like GPT-4 instead of executing complex queries on logs or data warehouses, the underlying data processing and computational paradigms inevitably simplify.

In this landscape, embedded databases like DuckDB thrive. Projects like DeepSeek’s smallpond leverage DuckDB for lightweight, high-performance data processing.

DuckDB’s Ascent

Born in defiance of the ‘big data’ hype, DuckDB started with a simple insight: nobody was building an embedded analytical database. The chart below shows exactly what the creators saw missing:

(Source：https://dl.acm.org/doi/10.1145/3299869.3320212)

Born in 2018, gaining explosive traction by 2022, DuckDB has by 2025 positioned itself as the de facto leader in the analytical processing database arena - an unprecedented rise for an embedded database:

(Source: OSS Insight)

Product Excellence

DuckDB’s success isn’t just about riding the right trends - its killer product features deserve equal credit. While we could write volumes about its capabilities, let‘s highlight the most impressive ones.

Zero-Copy Data Access

DuckDB queries external files (CSV, Parquet) without importing data. Its vectorized engine and columnar format enable efficient scans, even with partial HTTP Range requests. For example:

SELECT * FROM 'data.parquet';

Queries run directly on Pandas DataFrames or Arrow data, eliminating data duplication. This “Swiss Army knife” versatility aligns perfectly with data scientists’ workflows.

Queries run directly on Pandas DataFrames or Arrow data, eliminating data duplication. This “Swiss Army knife” versatility aligns perfectly with data scientists’ workflows.

HTTPFS Extension

Drawing inspiration from PostgreSQL’s design, DuckDB was architected with extensibility as a first-class citizen from day one. Among its extensions, HTTPFS stands out as a stroke of genius. This extension enables direct access to HTTP(S) endpoints and cloud object storage (like S3) - meaning DuckDB can query remote data as if it were local tables, using nothing more than a URL.

For example:

SELECT count(*)
FROM 'https://example.com/dataset/file.parquet';

DuckDB automatically fetches only the required file segments via HTTP. For columnar formats like Parquet, it goes further by selectively retrieving just the needed columns, dramatically improving efficiency.

This capability effectively complements the “big data” paradigm: users maintain large datasets in the cloud while pulling only relevant subsets for local analysis, even joining them with existing local datasets.

In this quintessential “edge-cloud integration” scenario, DuckDB delivers ad-hoc cloud data analysis without complex middleware – streamlining data engineering workflows. Wait, isn’t this exactly how “datalakes” were supposed to work all along?

The Ingenious HTAP Strategy of Seamlessly Integrating OLTP/OLAP

The “Pitfalls” of HTAP

HTAP, or Hybrid Transactional/Analytical Processing, aims to provide both TP and AP capabilities within a single database. This presents an opportunity for both TP-oriented and AP-oriented databases to expand their business scenarios, making HTAP a fiercely contested battleground in the database market. But is HTAP really that great? Whether extending from TP to HTAP or vice versa, one risks falling into the seemingly sweet “trap” of HTAP:

Storage-Compute Dissonance: OLTP relies on row-based storage (Row Store) for high-concurrency transactions, while OLAP requires column-based storage (Column Store) to accelerate aggregate computations. Forced integration necessitates maintaining two storage engines (e.g., row-store replicas + column-store replicas), leading to skyrocketing data synchronization overhead and consistency maintenance costs.
Resource Contention and Performance Degradation: OLTP’s short transactions (millisecond-level responses) compete with OLAP’s long queries (minute-level computations) for CPU, memory, and I/O resources.
Technological Conflict: OLTP emphasizes ACID transactions and row-level locks, whereas OLAP only requires relaxed eventual consistency. For instance, full-table scans in analytical queries may trigger row-store lock contention, causing transactional operations to stall.

Why Not Leverage the “Big Players”?

DuckDB’s philosophy is: Instead of forcing a single system to handle both, let systems excelling in transactions and those specializing in analysis each play to their strengths, then bridge them. Thus, PostgreSQL and DuckDB—the two most scalable systems in the TP and AP domains—naturally converge.

On one hand, DuckDB offers the Postgres Scanner extension, which can directly connect to a PostgreSQL database and map its tables as virtual views within DuckDB for querying. This allows DuckDB to act as an analytical accelerator, reading PostgreSQL’s live data without additional replication. This approach avoids the hassles of dual writes and asynchronous synchronization. During queries, data stored in PostgreSQL is efficiently read via its binary protocol, while analytical logic is executed within DuckDB.

SELECT * FROM postgres_scan('dbname=myshinydb', 'public', 'mytable');

On the other hand, the PostgreSQL ecosystem has seen a wave of extensions enhancing DuckDB’s AP capabilities, including the official pg_duckdb by DuckDB’s team.

In pg_duckdb’s implementation, DuckDB aligns with PostgreSQL’s logical timestamp synchronization (Logical Timestamp Alignment), ensuring analytical queries are based on the latest consistent snapshot of the transactional database, avoiding dirty reads and phantom reads. Additionally, DuckDB leverages its Postgres Scanner to directly access PostgreSQL’s row-store tables, then uses its vectorized engine to execute OLAP queries, boosting AP performance without data replication. For example, in TPC-H benchmarks, third-party tests showed pg_duckdb achieving over 1000x performance gains in complex analytical queries compared to native PostgreSQL.

PostgreSQL and DuckDB’s extensibility decouples the technical contradictions of OLTP and OLAP: PostgreSQL focuses on OLTP (e.g., order processing), DuckDB specializes in OLAP (e.g., real-time reporting), and the two achieve logical unity through in-process federated queries (rather than data copying).

GraphScope’s Attempts

At this point, I can’t help but reflect on GraphScope’s attempt to extend graph analysis capabilities based on OLTP databases – the GART project (unfortunately, GART’s latest commits are already six months old).

GART’s original goal was to make OLTP databases the data source for graph analysis:

OLTP databases offer mature data management solutions. In most enterprise applications, cleaned data first lands in OLTP databases (e.g., Ant Group’s OceanBase). Let GraphScope focus on graph analysis while leaving complex data management to mature OLTP databases. This vision aligns perfectly with DuckDB + PostgreSQL = HTAP. However, during GART’s development, GraphScope attempted to integrate graph analysis with transactional processing via real-time graph construction (Binlog parsing), but faced challenges:

Overly Heavy Architecture: Required maintaining an independent graph storage and compute cluster, with overly long synchronization chains (introducing Kafka + GraphEngine).
Limited Scenarios: Strong reliance on external data warehouses made lightweight embedded graph analysis difficult. What if we adopted DuckDB and PostgreSQL’s design philosophy? Here’s a potential approach:
Native Graph Operators: Integrate libgrape-lite (or a lighter single-machine parallel version) via PostgreSQL’s extension interface to enable graph traversal, community detection, and other operations within the transactional database, avoiding cross-system data migration.
Federated Graph Computing: Combine DuckDB’s read_parquet function to directly query graph data (via GraphAr) from cloud storage.

Product-Oriented Extreme Performance

The previous chapters showcased some of DuckDB’s impressive product capabilities, but behind them all lies one defining characteristic: speed – faster than large-scale Spark clusters, faster than PostgreSQL in OLTP scenarios. This performance stems from deliberate system design and architectural choices, such as:

Columnar Storage: In analytical scenarios, computations typically only require a subset of columns. Traditional row storage would result in mostly inefficient scans, whereas columnar storage minimizes I/O by reading only relevant data.
Vectorized (Batch) Processing: Operators process data in batches rather than row-by-row. This reduces function call overhead, improves cache utilization, and leverages modern hardware’s SIMD (Single Instruction Multiple Data) parallelism.
Zero-Copy Operations: DuckDB meticulously avoids memory copies during data transformations. For example, when reading Arrow data, DuckDB directly operates on Arrow memory buffers, and similarly outputs results in Arrow format. This enables seamless parsing of Arrow data within the query engine and direct Arrow-formatted result exports.
…

While these topics are widely discussed, let’s delve deeper into some of the early design choices highlighted in DuckDB’s research papers.

Computational Models

During DuckDB’s development, two dominant computational models existed in the database field — Vectorized Execution and Data-Centric Code Generation (Compiled Execution) — with significant differences in architecture, performance characteristics, and use cases. Both models are backed by numerous well-known systems:

Vectorized Execution: VectorWise, DB2 BLU, Columnar SQL Server, Quickstep, etc.
Compiled Execution: HyPer, Apache Spark, Peloton, etc.

However, comparing these models fairly is challenging. Directly contrasting real-world systems (e.g., HyPer vs. VectorWise) can lead to skewed conclusions due to differences in storage formats, parallelization strategies, etc. The DuckDB team adopted a unified experimental framework for an apples-to-apples comparison:

Methodology

Unified Framework: Implemented both models in the same system — Typer (compiled execution) and Tectorwise (vectorized execution) — ensuring identical algorithms, data structures, and parallel frameworks.
Isolated Variables: Only the execution engine was altered, eliminating interference from storage compression or optimizer variations.
Cross-Scenario Validation: Tested across TPC-H/SSB queries to analyze performance in compute-intensive, memory-bandwidth-sensitive, and other scenarios.

Below is their comparative analysis from this benchmark paper:

Comparison Dimension	Vectorized	Codegen (Compiled)	Why DuckDB Chose Vectorized
Performance Traits
Compute-Intensive Workloads	Materializes intermediates; more instructions	Register-optimized; fewer instructions	Memory access optimization: OLAP workloads (e.g., hash joins/aggregations) benefit from lower memory stall cycles.
Memory-Bound Scenarios	Batch processing hides cache misses; optimized parallel memory access	Complex loops limit prefetching/out-of-order execution
Engineering Practicality
Compilation Overhead	No runtime compilation; primitives pre-compiled	LLVM dynamic compilation adds latency (seconds for complex queries)	Embedded-friendly: No wait for JIT compilation; reduces first-run latency.
Debugging & Profiling	Isolated performance stats per primitive (e.g., selection/hash functions)	Fused operators in generated code obscure performance attribution	Developer-friendly: Easier for contributors to pinpoint optimizations.
Hardware Compatibility
SIMD Optimization	Simple primitives enable manual vectorization (e.g., 8.4x speedup with AVX-512)	Complex loops rely on compiler auto-vectorization (limited to ICC)	Future-proof: Higher potential for SIMD utilization gains.
Multi-Core Parallelism	Balanced batch splitting in Morsel-Driven parallelism	Same framework, but complex code reduces HT efficiency (e.g., low gains on AMD)	Thread scalability: More balanced workload distribution.
Functional Extensibility
OLTP Support	Inefficient for high-frequency single-row ops	Compiled stored procedures excel (ideal for HTAP)	OLAP focus: DuckDB targets analytics, not deep OLTP optimization.
Dynamic Optimization	Supports runtime adaptation (e.g., dynamic aggregation)	Logic hardcoded in generated code; requires recompilation	Flexibility: Adapts to dynamic workloads.
Storage Synergy
Storage Synergy	Native fit for columnar formats (e.g., RLE/Dictionary)	Requires type conversion/decompression (added overhead)	Columnar ecosystem: Reduces intermediate data materialization.

Thus, DuckDB’s choice of vectorized execution was based on a systematic, comprehensive, textbook-grade evaluation — even though Peter Boncz, a pioneer of vectorized execution, was an early core advisor to DuckDB.

High-Efficiency MVCC Implementation

As an embedded database designed for analytical workloads, DuckDB’s concurrency control mechanism fundamentally differs from traditional OLTP databases. To address batch updates, columnar operations, high compression storage, and embedded lightweight requirements in analytical scenarios, DuckDB deeply optimized its MVCC implementation strategy developed by the one and only Thomas Neumann, primarily solving:

Massive version management overhead: Traditional row-level MVCC generates excessive redundant version data during batch updates
Columnar compression vs. update conflicts: Compressed column data cannot be updated in-place, requiring avoidance of frequent decompression/recompression
Resource constraints in embedded scenarios: Ensuring transaction efficiency under memory/storage limitations
Analytical transaction characteristics: Coexistence needs of long-running batch operations and high-concurrency queries

Below summarizes DuckDB’s innovative MVCC designs:

Optimizations for Analytical Workloads

Columnar Batch Granularity
- Row-level versions create massive Undo records during batch updates, consuming memory and incurring high scan costs
- DuckDB uses “per 2048-row × per-column version metadata” to record entire batch modifications at once
Storing Deltas Instead of Old Values
- Compressed pages cannot be modified in-place; Undo Buffer only records “change descriptions” while keeping old versions in original compressed blocks, minimizing write amplification
Write Amplification-Minimized WAL/Checkpoint
- Bulk inserts (e.g., COPY 10GB CSV) first write directly to new data blocks, then log “block references” in WAL; commits atomically replace blocks, while rollbacks mark blocks as recyclable (avoiding “double writes”)
- Default 16MB WAL threshold triggers auto-checkpoint (manual CHECKPOINT supported); fsync enforced pre-write for durability

Lightweight & Embedded Design

Minimal Metadata: 2048-row batch versions + columnar deltas ensure “zero cost when unmodified,” with memory usage scaling linearly only with modifications
Fast Startup: Undo Buffer stored at data page tail; recovery only needs incremental WAL parsing. DuckDB survived 4000 TPCH refresh cycles in LazyFS power-failure/kill-process tests with zero data loss
Compression-Friendly: Version info is separated from data, keeping original column blocks compressed (no MVCC-triggered decompression/rewrites)
Single-Machine Concurrency: Read transactions never block; conflicting batch writes auto-retry without lock contention

This design enables DuckDB to deliver near-zero-cost concurrent transactions and analytical-grade throughput in single-machine memory environments, meeting modern data science and local embedded applications’ demands for “lightweight, compressed, ready-to-use” performance.

Compilation Optimization: Focused Efficiency Where It Matters Most

Traditional database systems rely on “heavyweight” optimizers, while DuckDB implements lightweight yet most-practical optimization rules specifically tailored for embedded scenarios. For comparison, we examined the compiled file sizes of DuckDB versus two well-known optimization frameworks in the industry: Calcite and ORCA:

Clearly, these optimization frameworks are overly bulky for DuckDB’s needs. Although DuckDB’s engineers and researchers fully understand the importance of query optimization for data systems, especially in analytical scenarios, they didn’t blindly pursue optimization perfection in their implementation. Instead, they strategically allocated their limited resources to the most critical areas, aligning with their vision of a “lightweight embedded analytical engine.” They achieved an elegant balance between optimization capability and resource consumption through multi-layered architectural innovations. Here are two particularly intentional design choices:

Rule-First Heuristic Optimization Framework

DuckDB employs a two-stage heuristic optimization architecture: - First Stage: Applies deterministic rewrite rules (e.g., predicate pushdown, expression simplification) based on logical equivalence, ensuring execution plan safety with minimal overhead (O(1) time complexity). - Second Stage: Performs lightweight statistics-based cost optimization, focusing only on critical paths (e.g., join ordering) with limited search scope. Dynamic pruning strategies keep optimization time within milliseconds.

Statistics-Independent Cardinality Estimation Model

DuckDB addresses the challenge of computing/missing statistics in embedded scenarios (e.g., when querying Parquet/CSV files directly where histograms can’t be precomputed) by introducing:

Foreign Key Inference Engine: Automatically detects implied PK-FK relationships in schemas to build dynamic equivalence sets.
Selectivity Propagation Model: Enables multi-table join cardinality estimation via chain calculations like card(A⋈B) = card(A) * card(B) / tdom, achieving over 90% accuracy without stored statistics.

Conclusion

DuckDB didn’t just ride the “big data backlash” —- it weaponized it using a good product. By combining:

Vectorized Execution (2-8x faster than Spark on 100GB datasets)
Zero-Copy Workflows (Arrow/Parquet as first-class citizens)
PostgreSQL Symbiosis (1000x+ OLAP speedups via postgres_scan()) …it redefined what’s possible in single-node analytics, achieving 90% of “big data” outcomes with 10% of the complexity.

DuckDB’s success signals a paradigm shift:

Era	Dogma	DuckDB’s Answer
2000s	“Normalize all data”	“Query raw Parquet”
2010s	“Scale out or die”	“Scale up smarter”
2020s	“ETL everything”	“Embed everywhere”

As LLMs democratize data access, DuckDB’s “small-data engine for a small-data world” approach becomes prescient. The future belongs not to systems that demand data centralization, but to those that dissolve into the workflow – invisible until needed, indispensable when used.

References

GOpt Optimization Process: An In-depth Case Study Based on LDBC SNB Queries

Tue, 25 Mar 2025 00:00:00 +0000

In today’s data analysis landscape, the optimization of complex graph queries has been a persistent challenge for the industry. GOpt is an optimizer framework designed for complex graph queries and was officially presented at SIGMOD Industry 2025. Its core objective is to provide an efficient and universal optimization solution that adapts to various graph query languages and execution engines. In this article, we will delve into the key innovations of GOpt in unified graph query optimization and analyze its optimization process through specific query cases from LDBC SNB.

GOpt achieves breakthroughs in the following areas:

Seamless support for mainstream graph query languages: GOpt is compatible with current leading graph query languages, including Cypher and Gremlin, with future plans to further support the graph query standard GQL.
Unified optimization of complex patterns: Deep optimization of various complex patterns such as Triangle, Path, Square, Clique, etc. Figure Fig.1 illustrates the various complex pattern forms supported by GOpt.

Fig.1 Complex Patterns Supported by GOpt.

Support for multi-pattern features in Cypher queries: GOpt first represents multi-patterns according to Cypher semantics as Inner/Left/Anti Joins, converting optimizations between multiple patterns into optimizations for Joins, thus directly reusing mature Join optimization rules from relational databases. Fig.2 showcases the various multi-pattern queries supported by GOpt and their internal representations within GOpt.

Fig.2 Multi-Patterns Supported by GOpt.

Integration with popular graph query execution engines: In a previous article Revolution: Enhancing Neo4j Efficiency with GOpt, we integrated GOpt into the Neo4j engine. Figure Fig.3 compares the execution effects of Neo4j and the GOpt optimization plans on the Neo4j engine. Within the LDBC Social Network Benchmark (LDBC SNB) standard test suite (where IC and BI represent Interactive-Complex and Business Intelligence queries respectively), GOpt brings an average improvement of 15.8X for Neo4j. We have further integrated GOpt into the GraphScope engine, achieving an average query performance improvement of 243.4X compared to GraphScope’s built-in rule-based optimizer.

Fig.3 Time Cost of LDBC Queries on Neo4j.

In this article, we will further explore points 1, 2, and 3, delving deeply into the optimization process of GOpt. We have chosen standard queries from LDBC SNB as a case study to intuitively demonstrate GOpt’s optimization process. Regarding point 4, GOpt has designed a dedicated Physical Converter layer. However, due to the excessive focus on underlying implementations, we will release subsequent articles on our official account to further decrypt this part, mainly covering: “How to Integrate GOpt with Neo4j?” and “How to Integrate GOpt with DuckDB?”. Stay tuned.

Optimization Process

Fig.4 illustrates the system framework of GOpt, which mainly consists of the following three layers:

Query Parser: Cypher/Gremlin queries first undergo syntax checking via Antlr, followed by conversion of the Antlr AST into an initial GIR structure through the GIRBuilder Tool.
GIR Optimizer: Applies optimizations based on the GIR structure, performing transform operations on the input GIR structure and outputting an optimized GIR structure. Depending on the implementation of transforms, optimizations can be executed based on heuristic or top-down search approaches.
Physical Convertor: Further converts the physical execution plan into code that can be executed by backend engines.

Fig.4 System Overview of GOpt.

GIR Structure

Firstly, let’s introduce what the GIR structure is. GIR (Graph Intermediate Representation) is a graph query language-independent intermediate data structure for graph queries, encompassing definitions of data models and a series of operator combinations for manipulating these data models.

The data model in GIR consists of graph data types and basic data types. Graph data types include Vertex (nodes), Edge (relationships), Path (paths), etc. Basic data types refer to Integer (integer), Float (floating point), String (character), Array/Map/Set (composite types), etc. GIR adopts data formats from relational databases: each data tuple contains multiple items, each with a name and a value, where the value type can be either a graph data type or a basic data type.

Operator combinations in GIR include graph operators and relational operators.

Graph operators define how queries retrieve graph data, including:

GET_VERTEX: Represents fetching node data sources from a graph database or retrieving endpoints from input edge data.
EXPAND_EDGE: Represents fetching edge data sources from a graph database or expanding adjacent edges from input node data.
EXPAND_PATH: Represents expanding paths composed of multiple edges from input node data.
MATCH_PATTERN: Expresses the Match Clause in Cypher or Gremlin as a unified structure. In GIR, there are two ways to represent this: a. A composite structure consisting of the aforementioned graph operators, marked by MATCH_START and MATCH_END indicating the start and end of graph operators. b. Represented as a Graph structure using JGraphT. These two structures are equivalent and can be converted interchangeably, as shown in Fig.5(c), where the Match Clause in the left-side query is represented uniformly in both forms.

Relational operators include Filter, Sort, Limit, Unfold, Project, Group, Join, Union. The support for them in GOpt is consistent with traditional databases, hence we will not elaborate further here. As illustrated in Fig.5(c), both Cypher query Fig.5(a) and Gremlin query Fig.5(b) are uniformly represented as corresponding GIR structures.

Fig.5 GIR Representation of Query Example.

Optimization Strategies

Next, we introduce how GOpt optimizes queries based on the GIR structure. The entire optimization process in the GIR Optimizer can be represented as a Directed Acyclic Graph (DAG). Each node in the DAG represents a Strategy, which embodies the execution of one or more rules. The edges in the DAG represent the sequential order of rule execution. GOpt executes these rules according to the topological sort of the DAG graph. Figure 6 shows a series of Strategies implemented by GOpt and their corresponding DAG relationships.

Figure 6: DAG of Optimization Process in GOpt.

The interface definition for Strategy is as follows:

interface Strategy {
    GIRPlan transform(GIRPlan input);
}

Based on this interface, GOpt provides two types of Strategy implementations: RuleBasedStrategy and PatternStrategy.

RuleBasedStrategy includes a series of heuristic rules, where each rule specifically implements the transform function to perform equivalent transformations on the input GIRPlan and output the transformed GIRPlan. These rules are partly reused from Calcite, including: FieldTrim, SortProjectTrans, AggJoinTrans. Additionally, to handle specific optimizations related to graph data and operations, GOpt has implemented specialized rules for graph data models, including: FilterIntoPattern, JoinToPattern, ComSubPattern, EVFusion, DegFusion, PKIndex, and LateProject.

PatternStrategy primarily optimizes the sequence of graph operators within Patterns, referred to as PatternOrders. The transform method takes a Logical GIRPlan, composed of MATCH_PATTERN and other relational operators, as shown in Figure 7(a); it outputs a Physical GIRPlan representing the PatternOrder, consisting of a series of physical operators, as shown in Figure 7(b). The transform method executes a top-down search algorithm as described in the GOpt paper, obtaining a series of PatternOrders along with their respective costs, and selects the PatternOrder with the lowest cost as the output Physical GIRPlan.

Finally, through the Physical Convertor, the Physical GIRPlan is converted into an Execution Plan supported by various engines. As shown in Figure 7(c), the Physical GIRPlan is converted into an Execution Plan supported by the GraphScope engine, where Expand(v1->v2, v3->v2) is converted to the ExpandIntersect implementation. In the Neo4j engine, this operator is converted to the ExpandInto implementation, as shown in Figure 7(d).

Figure 7: Optimization and Physical Conversion of Query Example.

Case Study

We selected a query case from LDBC SNB that involves complex patterns, multiple patterns, and aggregate computations, which are common optimization requirements. We represent this query using Fig.8 and Cypher as follows:

Fig.8 Query Case Description.

// M1
MATCH (person:PERSON)-[:KNOWS*1..3]-(otherP),
      (otherP)<-[membership:HASMEMBER]-(forum)

// M2
MATCH (otherP)<-[:HASCREATOR]-(post)<-[:CONTAINEROF]-(forum)

// M3
MATCH (otherP)<-[:HASCREATOR]-(:POST)-[:HASTAG]->(tag:Tag)

// Filter
WHERE person.id = $personId
      AND otherP <> $personId
      AND membership.joinDate > $minDate

// Aggregate
RETURN otherP, count(post) AS post_cnt;

GOpt first converts the Cypher query into GIR (Graph Intermediate Representation) through the Query Parser, as shown in Fig.9. M1, M2, and M3 form the initial Join structure, followed by a series of relational operations.

Fig.9 Initial GIR of the Query Case.

Next, the GIR Optimizer applies optimizations to the GIR based on strategies defined in the DAG (Directed Acyclic Graph). In the DAG diagram, we highlight the strategies that take effect for this query, as shown in Fig.10.

Fig.10 DAG of Optimization Process in the Query Case.

Optimization Rules

We further explain the optimization rules applicable to this query:

FilterIntoPattern: An extended implementation based on Calcite’s FilterIntoJoin. It pushes filtering conditions further down to graph operation operators within patterns. For example, given the following Cypher query:

   MATCH (v1)-[e1]->(v2),
       (v2)-[e2]->(v3),
       (v1)-[e3]->(v3)
   // There is a <Join> between the two patterns in GIR
   MATCH (v1)-[e4]->(v4)
   WHERE v3.name = "China"
   RETURN v1.name, count(v2);

After applying this rule, the optimized query is equivalent to rewriting it as:

   MATCH (v1)-[e1]->(v2),
       (v2)-[e2]->(v3 {name: 'China'}),
       (v1)-[e3]->(v3 {name: 'China'})
   // There is a <Join> between the two patterns in GIR
   MATCH (v1)-[e4]->(v4)
   RETURN v1.name, count(v2);

AggJoinTrans: Pushes aggregate operations down to the Join operator. This optimization rule is inspired by relational databases.
JoinToPattern: Removes the Join operator between patterns and merges the left and right sub-patterns into a unified pattern. Whether patterns can be merged depends on whether the semantics remain consistent after merging. Based on the semantic requirements of different query languages, we define three types of semantics within patterns:
- Homomorphism Semantics: Allows repetition of vertices or edges within the pattern, which is adopted by Gremlin.
- Edge-Distinct Semantics: Allows repetition of vertices but not edges within the pattern, which is adopted by Cypher.
- Vertex-Distinct Semantics: Allows repetition of edges but not vertices.
- Vertex-Edge-Distinct Semantics: Neither vertices nor edges can repeat within the pattern.
Since the Join between two patterns inherently represents Homomorphism semantics, this rule only applies under that premise.
PatternStrategy: As described earlier, this strategy applies a Top-Down Search Algorithm to the input pattern structure and selects the optimal PatternOrder based on cost.
ComSubPattern: For the two sub-patterns involved in a Join, extracts their common parts as the original input data. If the common part consists of only a single vertex, the Join operation can be further optimized into an Expand operation.

Optimization Process

The GIR Optimizer executes specific strategies on the GIR in the order specified by Fig.10:

FilterIntoPattern is performed first: The filtering conditions {person.id = $personId, otherP <> $personId, membership.joinDate > $minDate} are pushed down to the graph operators that generate person nodes, otherP nodes, and membership edges.

Fig.11 The Optimization of FilterIntoPattern.

Next, AggJoinTrans is executed: This involves splitting the {otherP, count(post) as post_cnt;} related group operation. Part of it is pushed down into the left branch of the Join, while the other part remains after the Join to sum up the previously computed group results to ensure semantic equivalence.

Fig.12 The Optimization of AggJoinTrans.

JoinToPattern is then applied: The Join operation between M1 and M2 is removed, and M1 and M2 are merged into a unified pattern M4. In this example, we assume the patterns conform to Homomorphism semantics, which is a prerequisite for applying this rule.

Fig.13 The Optimization of JoinToPattern.

PatternStrategy is executed next: The following figure represents the output Optimal Pattern Order, consisting of a series of physical operators.

Fig.14 The Optimization of PatternStrategies in M3 and M4.

Finally, ComSubPattern is applied: This optimizes the reusable common results between patterns. As shown in Fig.14, in the left branch of the Join, M4 produces two columns (p2, _cnt) after performing a Group operation, while the optimized PatternOrder of the right branch M3 starts with Scan p2(PERSON, id <> $id). Given that the common part only contains the single point p2, according to the ComSubPattern rule, the Join structure is further optimized into an Expand operation. Hence, M3 directly continues from the P2 point produced by the Group operation to execute Expand p2->"" (HASCREATOR).

Tab.1 The LDCB₁₀₀ DataSet.

Experimental Results

For the aforementioned case study, we conducted two main experiments: ablation tests and Pattern Orders. We used the LDBC₁₀₀ dataset, as shown in Table Tab.1.

Graph	\|V\|	\|E\|	Size
sf100	283M	1754M	156GB

Tab.1 The LDCB₁₀₀ DataSet.

The system configuration consists of a single node with the following specifications:

8-core Intel Xeon E5-2620 v4 CPUs at 2.1GHz
512GB memory
10Gbps network.

We used the GraphScope v0.29.0 system, powered by the underlying Gaia engine, with 32 threads.

The ablation tests primarily compare the individual optimization effects of the rules: FilterIntoPattern, AggJoinTrans, JoinToPattern, and ComSubPattern. To avoid mutual interference between these rules, we sequentially added them in the order defined in Fig.10 and compared the execution time after each rule was added. The results are shown in Table Tab.2.

Rules	Time Cost (ms)
None	770573
+FilterIntoPattern	234313
+AggJoinTrans	214955
+JoinToPattern	64466
+ComSubPattern	7014

Tab.2 Time Cost of Ablation Tests in the Query Case.

The Pattern Orders experiment compares the optimal order generated by PatternStrategy with two other randomly chosen orders. Figure Fig.16 illustrates the execution sequence of the three Pattern Orders and annotates the actual intermediate data volume produced at each step. Finally, we compare the execution times of the three orders in Table Tab.3.

Fig.16 Pattern Orders of GOpt-plan, Alt-plan1, Alt-plan2.

Rules	Time Cost (ms)
GOpt-plan	7014
Alt-plan1	22211
Alt-plan2	194047

Tab.3 Time Cost of Pattern Orders in the Query Case.

Dissecting Claude Code: A Graph-Powered Deep Dive into an AI Coding Agent's Architecture

The Numbers at a Glance

1. Architecture: Five Layers, One Gravity Well

The REPL: A Component That Imports the Universe

2. Bridge Functions: The Nervous System

3. The Permission Gateway: A Single Chokepoint

The 151:1 Complexity Ratio

4. The Buddy System: A Gacha Game Inside a Dev Tool

How Companion Generation Works

The Hex Obfuscation Trick

Gacha Rarity Distribution

5. Dream Task: The AI That Dreams While You Code

The Four Gates (Cheapest First)

The Four Phases of Dreaming

The Filesystem Retry Trick

6. Swarm: Multi-Agent Orchestration

Backend Detection Priority Chain

Why AsyncLocalStorage?

Task ID Design

7. Cost Tracker: Follow the Money

The Billing Access Gate

What Gets Tracked

8. The Loneliest 1,265

9. The Class Hierarchy: OOP at the Boundaries

10. Patterns Worth Stealing

Deterministic Generation Without Persistence

Gate Ordering by Cost

Single-Letter Task ID Prefixes

Honest Function Names

AsyncLocalStorage for Concurrent Identity

Single Permission Gateway

Conclusion

Following 94 Million Relationships Down the Rabbit Hole

Question 1: Did publication output actually decline?

Question 2: Did the funding dry up?

Question 3: Are journals shutting the door too?

Question 4: What happened to the collaboration network?

Why could we keep asking?

How it works under the hood

Hypergraph Data Model: a schema that grows with your questions

Unbiased Sampling: sub-second responses over 94 million relationships

NeuG: co-located storage and computation

Benchmarks

Try it yourself

We X-Rayed OpenClaw with a Graph Database — Here's the Technical Debt We Found

Why a Graph Perspective?

The Approach: Turn the Codebase into a Graph

Finding 1: High-Risk Functions

Defining Risk Score

Filtering for the Real High-Risk Core

Finding Zombie Functions

Finding 2: The Over-Coupled Module

What Graph Analysis Reveals That Other Tools Don’t

Three Recommendations for OpenClaw

OpenClaw Users: Why Is Your Token Budget Disappearing Out of Nowhere?

Background

The Problem: Session History That Never Stops Growing

What Is Heartbeat Secretly Doing?

Step 1: Locate All Heartbeat-Related Code

Step 2: Find the Core Entry Point via the Call Graph

Step 3: Expand the Call Chain — What Does It Actually Touch?

Why Doesn’t lightContext: true Help?

CodeGraph: Reproduce This Analysis Yourself

Prerequisites

Building the Index

CLI

Python SDK

What You Can Do Now: Four Practical Mitigations

Summary

AI-Powered Open Source Development Management: NeuG's Exploration and Practice

Pain Points in Open Source Development: Lack of Comprehensive Project Management

Spec-Driven: The Best Development Paradigm for NeuG

Use Case: Adding Multi-threaded Transaction Tests to NeuG

Specify Phase: Task Expansion and Analysis

Plan Phase: Persisting Implementation Details

Tasks Phase: Decomposing Tasks, Syncing to GitHub

GitHub Sync Operations

Implement Phase: Preserving Traditional Development

GitHub Integration: A More Automated Issue/PR Experience

Context Collection

Why Doesn’t `lightContext: true` Help?