Skip to content

ADR-005: Cloud-Hosted Memory via Azure Blob Storage

Date: 2026-04-17 Status: Accepted (open questions resolved 2026-04-17; Phases 1, 2, 5, 6a shipped — Phase 6b next) Deciders: the user, EntraClaw Agent Context: Agent memory portability across machines + foundation for a later cloud-hosted poller

Context

Today every piece of agent state lives on a single Mac:

Path What Size
~/.claude/projects/.../memory/ Behavioral memory (markdown + frontmatter: feedback_*, project_*, user_*, reference_*) ~96 KB
~/.entraclaw/data/interactions/YYYY-MM-DD.jsonl Append-only event log of every Teams/email I/O growing, currently ~20 KB
~/.entraclaw/data/summaries/YYYY-MM-DD.{html,json} Rendered daily summaries + sidecar counts ~5 KB/day
~/.entraclaw/data/watched_chats Small state file of polled chat IDs < 1 KB
~/.entraclaw/data/email_cursor.txt ISO 8601 receivedDateTime watermark < 50 B

Consequences of the single-machine model: 1. Switching between Mac Studio and laptop means either manually copying files (fragile, encryption headaches with Keychain) or losing continuity. 2. A planned future cloud-hosted poller (see ADR-006 draft) can't share state with the Claude Code MCP server process if everything is local-filesystem-scoped. 3. Disaster recovery: a lost/reformatted Mac means lost memory.

Decision

Move the four portable data categories to Azure Blob Storage. Keep the Blueprint private key and .env/.mcp.json per-machine (those are derivable or machine-specific). Offer a --keep-memory-local escape hatch in setup.sh for users who don't want cloud sync.

What moves (cloud-backed)

  • behavioral/ — the ~/.claude/projects/.../memory/ tree
  • interactions/ — the daily interaction JSONL
  • summaries/ — the daily summary HTML + sidecar JSON
  • state/watched_chats and state/email_cursor.txt — small operational state

What stays local (per-machine)

  • .env — regenerated by setup.sh
  • .mcp.json — per-machine absolute paths
  • Blueprint cert private key — in OS Keychain; replaced per-machine per ADR-003 + today's warn-and-confirm flow
  • Claude Code session transcripts in ~/.claude/projects/.../*.jsonl — not load-bearing

Design

Storage backend

Azure Blob Storage, one Storage Account per tenant, one container per Agent User.

https://entraclaw<suffix>.blob.core.windows.net/
  entraclaw-memory/   (container — one per Agent User UPN)
    manifest.json
    behavioral/
      MEMORY.md
      feedback_*.md
      project_*.md
      user_*.md
      reference_*.md
    interactions/
      2026-04-17.jsonl
      2026-04-18.jsonl
    summaries/
      2026-04-17.html
      2026-04-17.json
    state/
      watched_chats
      email_cursor.txt

Why Blob over alternatives: - vs Cosmos DB: 10× the cost, unnecessary features for data this size/shape - vs Table Storage: too schema-rigid for markdown/JSONL - vs Git: great for the low-write behavioral layer, terrible for the high-write interaction log (dozens of commits/day, merge conflicts with multiple machines). Considered a split — rejected to keep ops surface small - vs OneDrive / iCloud Drive: not auth-scoped to the Agent Identity; wrong trust boundary

One account per tenant avoids Azure's global-unique-name race while keeping RBAC clean (each Agent User is granted Storage Blob Data Contributor only on its own container).

Auth

The Agent User's three-hop flow produces a delegated token for graph.microsoft.com. For Blob, we do a parallel third hop requesting https://storage.azure.com/user_impersonation via the same Agent User FIC. Same private key, same first two hops; only the resource URI at Hop 3 changes.

Requires: - .default scope for the Storage Account added to the Agent Identity's consent grant during setup.sh - RBAC: Agent User assigned Storage Blob Data Contributor on its container (not the whole account)

Manifest

manifest.json at the container root is the top-level index:

{
  "schema_version": 1,
  "updated_at": "2026-04-17T20:00:00Z",
  "entries": {
    "behavioral/MEMORY.md":        {"size": 1234, "mtime": "..."},
    "behavioral/feedback_*.md":    {"count": 15, "total_size": 24000},
    "interactions/2026-04-17.jsonl": {"size": 8192, "mtime": "..."},
    "summaries/2026-04-17.html":   {"size": 12000, "mtime": "..."}
  }
}

The agent reads the manifest at session start to know what's available without listing the whole container. Maintained automatically on every write.

Local cache + write-through

  • Read-through cache at ~/.entraclaw/cache/blob/ on every machine
  • Read: cache first; on miss or stale (compare mtime to manifest), pull from blob
  • Write: write local cache + blob in parallel
  • High-frequency writes (interactions) batch in-memory for up to 5s or 4KB, then flush
  • On machine start: sync manifest, lazy-pull changed files on first access

Retention + compaction (hippocampus → cortex)

Retention is governed by what still has value in raw form, not by calendar alone. Raw interaction logs stop being useful as verbatim context after about a week — after that, the pattern is what matters, not the trace. Daily summaries and weekly digests persist longer because they compress the value. Behavioral memory (hand-curated rules, facts about people and projects) persists indefinitely because it's already at the distilled layer.

Retention policy:

Layer Retention Rationale
behavioral/ Indefinite (manual prune only) Distilled rules + facts; no auto-decay
summaries/ (daily) 30 days Already compressed; queryable
compacted/YYYY-WW.md (weekly digests) 30 days Second-order compression; captures texture
interactions/YYYY-MM-DD.jsonl (raw) 7 days After a week, raw form is rarely load-bearing

Daily compaction step (runs once per day, before day N-7 is purged):

  1. Read day N-7's raw interactions/YYYY-MM-DD.jsonl.
  2. Extract three kinds of durable content, promoting each to the appropriate layer:
  3. Behavioral learnings ("Brandon prefers Teams over email when initiating") → behavioral/feedback_*.md or project_*.md. Curated; agent writes a draft, confirms with sponsor before promoting.
  4. Patterns, recurring decisions, facts about people/projects → compacted/YYYY-WW.md (this week's digest, append-mode).
  5. Routine status pings, noise, banter → discarded.
  6. Delete the raw interactions/YYYY-MM-DD.jsonl once compaction succeeds.

This is hippocampus → neocortex: the raw episodic trace decays, the consolidated semantic pattern persists. Also mirrors Claude Code's /compact — same information-theoretic move at a different layer.

Token-budget governor

Retention was cross-checked against the agent's context budget. With a 1M-token context window and realistic reserves (current conversation, tool schemas, system prompt), the available budget for hot-loaded memory is ~400K tokens. Worst-case sum of all layers above:

Layer Size estimate Tokens
All behavioral memory ~100 KB ~25 K
7 days raw interactions ~350 KB ~85 K
30 days weekly digests ~200 KB ~50 K
30 days daily summaries ~100 KB ~25 K
Total ~750 KB ~185 K

Comfortable fit. Retention isn't bounded by context; it's bounded by the usefulness of raw traces, which is why 7 days is the right window for interactions/ even though the budget could hold more.

Concurrency

Low-risk in practice — the agent runs on one machine at a time, one process. Guard anyway with ETag-based optimistic concurrency on blob writes. If the ETag changed since our last read, the write fails with HTTP 412; we refetch and retry. This handles the rare case of two of the same user's machines writing simultaneously (e.g. Brandon forgot to stop the Mac Studio server before switching to the laptop). Last-writer-wins semantics after retry, which is correct for single-user-multi-machine.

Migration

First run on a machine where ENTRACLAW_BLOB_ENDPOINT is set and the container is empty:

  1. setup.sh prompts: "This will upload ~{N} KB of local memory + data to Azure Blob. Continue? [y/N]"
  2. Walk local ~/.claude/projects/.../memory/ and ~/.entraclaw/data/, upload to blob with the directory layout above
  3. Verify manifest matches; print summary
  4. Leave local files untouched — blob becomes the source of truth, local is now a cache, not a backup

Rollback: set ENTRACLAW_KEEP_MEMORY_LOCAL=true in .env. Code reverts to pure-local operation. Local files are always authoritative when this flag is set.

Alternatives considered

  1. iCloud Drive symlinks — simplest, zero infra. Rejected because (a) not portable to non-Apple users / cloud-hosted pollers, (b) no auth scoping to the Agent Identity, (c) doesn't align with the "durable services, ephemeral agents" direction.

  2. Git repo for everything — versioning and audit come free. Rejected for the interactions log (high write rate, commit noise, merge conflicts). Split architectures were considered but add ops complexity for minimal benefit.

  3. Leave state local, sync via rsync — one-shot manual migration per switch. Rejected: Brandon's exact complaint that triggered this ADR.

  4. Cosmos DB or Azure Tables — queryability we don't need, costs we do. Rejected.

Implementation phases

Phase What LOC estimate
1 src/entraclaw/storage/blob.py — async blob client (get/put/list/delete/exists + ETag) ~200 + tests
2 src/entraclaw/storage/backend.py — MemoryBackend protocol + Local/Blob implementations; route interaction_log.py, daily_summary.py, memory-file reads through it ~150
3 CachedBlobBackend — local mirror + write-through + ETag retry ~150 + tests
4 Manifest maintenance (atomic read-modify-write on every upload) ~80 + tests
5 setup.sh — provision Storage Account + container + RBAC; --keep-memory-local flag; migration prompt ~120 shell + Python helper
6 ADR + README updates + hard-won-learnings entry docs-only

Ship order: 1 → 2 → 5 → 3 → 4 → 6. Phase 5 comes early so setup.sh works for other devs sooner; the cache/manifest layers refine the running experience.

--keep-memory-local

Escape hatch for: - Privacy-conscious users who don't want Azure sync - Offline / air-gapped environments - Users wanting to evaluate EntraClaw before trusting cloud storage - Dev iteration speed (no network round-trip per write)

Implementation: - setup.sh CLI flag; skips Storage Account provisioning - .env gets ENTRACLAW_KEEP_MEMORY_LOCAL=true - MemoryBackend selection driven by the flag — pure LocalBackend if set, CachedBlobBackend otherwise - Documented in README as a supported mode, not a hack

Default: cloud mode when az login exists + Azure subscription is reachable. Local mode when provisioning fails OR the flag is set explicitly.

Resolved decisions (from sponsor discussion 2026-04-17)

  1. Container per Agent User — confirmed. Container names include the Agent User object ID so multiple CLIs under the same human sponsor don't collide.
  2. Differentiated retention via compaction — confirmed. See "Retention + compaction" above: behavioral indefinite, summaries + digests 30d, raw interactions 7d with a daily compaction step that promotes durable content to longer-retention layers before deletion.
  3. No cost surfacing in the daily summary — confirmed. Monthly projection printed once at setup.sh completion so new users see what they're signing up for; otherwise silent.
  4. ETag-only concurrency for now — confirmed. Adds a TODO below for the cases we're not solving yet.

TODO (future work, explicitly out of scope for this ADR)

  • Document user-visible recovery behavior when two of the user's own machines run simultaneously. ETag handles it correctly (last-writer-wins after retry), but the semantics should be documented so users aren't surprised when their laptop clobbers an unflushed Mac Studio write.
  • Multi-agent coordination on shared memory. Explicitly not supported by this design. If we ever want two distinct Agent Users to share a memory store (e.g. Brandon's EntraClaw and another teammate's agent both writing to a team-wide behavioral layer), that's a future ADR. Would require real locking or CRDT-style merge semantics. Categorized as science-project for now.
  • Compaction quality control. The daily compaction step is the most judgment-heavy part of this design. The first few weeks should log compaction decisions so we can eyeball what got promoted vs discarded and tune the heuristic. Treat the first 30 days of compaction output as reviewable.
  • Manifest consistency under concurrent writes. The manifest is a single file updated on every blob write; ETag protects it, but heavy write bursts could cause retry loops. Monitor; switch to per-prefix manifests if that becomes a real problem.

Consequences

Positive - Weekend-portability use case solved (Mac Studio ↔ laptop) - Foundation for cloud-hosted poller (ADR-006) - Memory survives machine loss - Others adopting EntraClaw get portable memory out-of-the-box

Negative - New dependency: Azure Storage account + RBAC setup in setup.sh - New auth surface: storage.azure.com scope, additional Graph permissions - Read latency on cache misses (mitigated but real) - --keep-memory-local is a code path we now have to maintain

Neutral - Format (markdown + JSONL) unchanged — LLM-optimal, human-readable, no migration pain