Many developers use multiple AI surfaces daily: GitHub Copilot in VS Code, Copilot CLI, Microsoft 365 Copilot, Microsoft Scout, Claude, ChatGPT, or Cursor.

The problem: each surface starts without the context you gave the last one. Preferences, current work, boundaries, and decisions stay trapped in whichever tool you told.

This post proposes a portable personal context source: structured markdown files in a GitHub repo that AI tools can read. No vendor supports this end-to-end today. The storage can be portable; the behavior is not automatic. Each surface still needs its own wiring.

The Problem: Context Islands​

The pattern: each tool has its own user-context format. The result:

• Repeated preferences in every tool ("I prefer concise output," "use TypeScript," "don't auto-push to main")
• Decisions invisible outside the surface where they happened
• Expertise and boundaries known only where you stated them

You Already Know What the Solution Feels Like​

Each tool stores personalization separately. Portable context makes the source shared:

Today:
  M365 Copilot    → knows you like concise output
  VS Code Copilot → doesn't know (asks again)
  Copilot CLI     → doesn't know (asks again)
  Scout           → has its own separate copy
  Claude          → has its own separate copy

With portable personal context:
  Wired surfaces  → read from the same source → load your preferences

Why Not Just Use Those Existing Features?​

Use built-in personalization where it exists; keep portable context as the canonical source you can inspect and version.

Why Not CLAUDE.md, copilot-instructions, or Cursor Rules?​

Those files are instructions TO the AI for one project. Personal context is information ABOUT you across tools. Cursor's current model is Project Rules in .cursor/rules/*.mdc; .cursorrules is legacy.

.github/copilot-instructions.md  → "In this repo, use ESM imports"
personal-context/process/...     → "I always prefer ESM over CommonJS"

Repo-level files govern a codebase. Personal context governs how to work with you.

Why Not Just an Agent or Skill?​

Agents and skills are task-scoped. Personal context is user-scoped.

Personal context should feed agents and skills, not duplicate them. Without it, agents start without your quality bar, boundaries, or past decisions.

Why Not Mem0 or a Cloud Memory Service?​

Mem0 is a cloud API for persistent AI memory. It makes different tradeoffs:

For one developer, dozens of curated facts, decisions, and preferences may not need a hosted dependency.

Personal Context Is Not Memory​

Key distinction: personal context is not memory. They overlap, but they need different storage, governance, and precedence rules.

Context is declared, curated, and authoritative. Memory is accumulated from use.

They are two ends of a promotion pipeline:

observation  →  candidate  →  [ratification gate]  →  context
 (memory)       (proposed)     (a human decision)      (canonical)

Memory is the raw feed. Context is the reviewed output. The ratification gate is the control point. The architecture changes:

• Stores. Memory wants a high-volume append log. Context wants a small, curated, versioned set.
• Precedence. Context outranks memory. A remembered exception does not override a stated boundary.
• Retrieval and governance. Context is load-always instruction; memory is search-when-relevant evidence.

This post is about context, not memory: authored, reviewed, and portable.

The Insight: LLMs Already Speak Markdown​

AI tools read files. LLMs understand markdown. Developer tools commonly authenticate with GitHub.

A private GitHub repo with structured markdown files can be the shared context source.

Any surface that can read the repo can load the same context.

The Architecture: Personal Context as a Repo​

github.com/<yourname>/personal-context  (placeholder private repo)
│
├── context.json              ← Manifest: what's here + retrieval rules
│
├── core/                     ← RARELY CHANGES (your "constitution")
│   ├── expertise.md          # What you know, your domain authority
│   ├── boundaries.md         # What stays human, what AI never does alone
│   ├── role.md               # Job, scope, organization
│   └── communication.md     # How you prefer to interact
│
├── decisions/                ← APPEND-ONLY (your "ledger")
│   ├── _active.md            # Decisions still governing current work
│   ├── 2026-07.md            # This month's new decisions
│   └── ...
│
├── process/                  ← STABLE (your "playbook")
│   ├── content-workflow.md   # How you create content
│   ├── code-workflow.md      # How you write and ship code
│   ├── quality-bar.md        # Definition of done per work type
│   └── tool-preferences.md  # Preferred tools and patterns
│
├── active/                   ← CHANGES OFTEN (your "whiteboard")
│   ├── projects.md           # Current active projects
│   ├── sprint-focus.md       # This sprint's commitments
│   └── parking-lot.md        # Deferred items
│
└── .github/
    └── copilot-instructions.md  # Tells Copilot how to USE this repo

Why Four Layers?​

Separate by durability: how often it changes and who can change it.

What Goes in Each Layer​

Core: Your Constitution​

Rarely changing context: expertise, boundaries, communication preferences.

core/expertise.md — What you know:

## Domain Expertise
- Azure SDK documentation across JavaScript, Python, .NET, Java, Go, Rust
- AI developer tools (MCP servers, AI Toolkit, Copilot extensions)
- Content workflow automation and multi-agent orchestration
- Technical writing for developer audiences

## Not My Expertise (don't assume I know)
- Kubernetes operations / cluster management
- Frontend framework internals (React, Vue, etc.)
- ML model training / fine-tuning

core/boundaries.md — What stays human:

## What AI Should Never Do Autonomously
- Push code to upstream repositories (only to forks)
- Send emails, Teams messages, or any outbound communication
- Close or resolve work items without my confirmation
- Delete files, branches, or repos
- Make irreversible changes without showing me the plan first

## What AI Can Do Without Asking
- Read files, search code, explore repos
- Draft content for my review
- Run tests, linting, builds
- Create branches on my fork
- Propose edits (but not commit without confirmation)

Decisions: Your Ledger​

Decisions can be proposed from any surface so settled questions stay settled after review.

decisions/_active.md — Still-relevant decisions:

### [2026-07-06] Branch naming convention
- **Context:** Inconsistent branch names across repos
- **Decision:** Always use `{type}/{work-item-id}-{brief-slug}`
- **Types:** feat, fix, docs, refactor, test

### [2026-06-15] Prefer tables over prose for comparisons
- **Context:** AI kept writing long paragraphs comparing options
- **Decision:** When comparing 3+ options, always use a table
- **Supersedes:** Nothing (new preference)

### [2026-05-28] No hand-written namespace files
- **Context:** Generated files were higher quality than hand-written
- **Decision:** All namespace articles must come from the generation pipeline
- **Implications:** Slower to ship, but deterministically correct

Process: Your Playbook​

Work preferences. Update as workflow changes.

process/quality-bar.md:

## When Is a Pull Request Done?
- [ ] Work item linked with "Fixes AB#{id}"
- [ ] Meaningful title and description (not just commit messages)
- [ ] No unrelated changes (surgical edits only)
- [ ] CI passes
- [ ] Review comments addressed, not dismissed
- [ ] Staged preview links included for doc changes

## When Is an Article Done?
- [ ] Technically accurate (verified against product behavior)
- [ ] Code samples run without modification
- [ ] All links resolve (no 404s)
- [ ] Metadata correct (ms.topic, ms.date, ms.service)
- [ ] Reviewed by at least 1 peer

Active: Your Whiteboard​

Current work state, writable by any surface.

active/sprint-focus.md:

## Sprint 14 (2026-07-01 → 2026-07-12)

### Committed
1. Ship MCP auth namespace docs (AB#4521)
2. Review 3 community PRs on azure-dev-docs
3. Update AI Toolkit quickstart for v0.9

### Stretch
- Prototype portable context layer (this project!)

How Surfaces Consume It​

Selective Retrieval: Only Load What's Relevant​

Do not load the whole repo. Use context.json to choose task-relevant files:

1. Read context.json (< 1KB, always cached)
2. ALWAYS load: core/boundaries.md + core/communication.md (~400 words)
3. Classify the current task → match to load_by_task
4. Load those 2-3 files (~500 words)
5. Load decisions/_active.md (~300 words)

Total: ~1,200 words ≈ 1,600 tokens

The Priority Stack​

Resolve contradictions deterministically:

core/boundaries.md          ← ALWAYS wins. Non-negotiable.
decisions/_active.md        ← Settled questions. Don't re-ask.
process/*.md                ← How to do things. Follow unless overridden in-session.
active/*.md                 ← Informational state. Not authoritative.

Threat Model: Retrieval Is the Boundary​

The primary risk is prompt injection causing a surface to retrieve or reveal context it should not have. Trust tiers must be enforced before retrieval, by deciding what files or slices enter the prompt. Output scanning is not a security boundary; use it only as defense in depth. Keep two boundary files if needed: shareable operating rules that most tools can load, and private sensitive constraints that only trusted surfaces can retrieve.

Writing Back: Closing the Loop​

After human confirmation, any surface can write decisions back:

# After making a decision in any surface:
cd ~/personal-context
echo "
### [$(date +%Y-%m-%d)] {decision title}
- **Context:** {why this came up}
- **Decision:** {what was decided}
- **Implications:** {what this means going forward}
" >> decisions/_active.md

git add decisions/_active.md
git commit -m "decision: {brief title}"
git push

That simple append is safe only for a single writer with a fresh clone. Multi-surface writes need write intents with IDs and timestamps, pull-before-push, and PR-based reconciliation for stale updates or conflicts. The core layer stays human-only; non-active layers should go through review instead of direct overwrite.

Connecting Each Surface (Proposed Integrations)​

Examples only. Some work manually; others need vendor support. The mechanism: read files and inject context.

GitHub Copilot in VS Code​

In your user-level settings.json:

{
  "github.copilot.chat.codeGeneration.instructions": [
    { "file": "~/personal-context/core/boundaries.md" },
    { "file": "~/personal-context/core/communication.md" },
    { "file": "~/personal-context/decisions/_active.md" }
  ]
}

Or reference the repo in any project's custom instructions:

<!-- .github/copilot-instructions.md in any repo -->
For my personal preferences and decisions, reference:
https://github.com/<yourname>/personal-context

Copilot CLI​

Use the current standalone copilot CLI. Put durable CLI instructions in $HOME/.copilot/copilot-instructions.md, then point those instructions at the cloned context repo:

mkdir -p ~/.copilot
cat > ~/.copilot/copilot-instructions.md <<'EOF'
For personal preferences and decisions, read:
- ~/personal-context/core/communication.md
- ~/personal-context/core/boundaries.md
- ~/personal-context/decisions/_active.md

Treat the repo as reference context. Do not rewrite core files without human approval.
EOF

For richer integration, expose the same repo through an MCP server and register it with copilot mcp.

Microsoft Scout​

Scout exposes settings for memory, personality presets, workspace, and permissions. Use those surfaces to mirror the same repo-backed preferences manually or through a sync process. A generated profile file can work as an implementation pattern, but the path below is illustrative, not a documented Scout contract:

# Sync script: pull personal-context → render an illustrative Scout profile
$role = Get-Content ~/personal-context/core/role.md -Raw
$comms = Get-Content ~/personal-context/core/communication.md -Raw
$boundaries = Get-Content ~/personal-context/core/boundaries.md -Raw

@"
# Personal Profile
$role

## Communication
$comms

## Boundaries
$boundaries
"@ | Set-Content ./scout-profile-example.md

Microsoft 365 Copilot​

Sync the repo to a OneDrive folder:

OneDrive/personal-context/ → synced from GitHub repo

Any MCP-Enabled Tool (Claude, Cursor, ChatGPT)​

Expose the repo as an MCP resource server, or clone it locally and point the tool config to the files. MCP is the integration protocol for tools, resources, and prompts; use it to expose context as resources or tools where supported.

Getting Started: Example 30-Minute Setup​

This is a rough first-pass estimate, not a guarantee. The ongoing cost is maintenance: review proposed changes, resolve conflicts, and prune stale active context.

1. Create the repo (5 minutes)​

gh repo create personal-context --private
cd personal-context
mkdir -p core decisions process active .github

2. Write your identity (10 minutes)​

Write what you would tell a new team member on day one.

3. Capture your first decisions (10 minutes)​

Write five repeated preferences or decisions in decisions/_active.md.

4. Connect one surface (5 minutes)​

Wire up VS Code settings, Scout profile, or a CLI alias. Verify it loads context.

5. Evolve naturally​

When an AI asks something it should know, write it down, commit, and push.

The Payoff​

Likely payoff: fewer repeated preferences, fewer re-decisions, and faster starts. Keep time-saved claims only if measured.

Beyond the Repo: When a Service Makes Sense​

The repo is the floor: markdown, git, no server, no vendor, no API key. Use a service only when a flat repo cannot enforce access. Git gives readers the whole file; a hosted service can return only authorized slices.

What a hosted version would buy you​

• Server-side redaction by trust tier. Enforce public / work / private tiers at the server. A flat repo cannot do that; clone access gets everything.
• Identity-based audit and access control. Log who read context, when, and from which surface.
• Central precedence. Resolve boundaries, decisions, process, and active state once instead of per surface.

The key design call: contract vs. transport​

Do not make MCP the canonical contract. MCP is the integration protocol, not the canonical data model. Keep Git as the source of truth. If you build a service, make it a derived read facade over the repo, with a REST/OpenAPI contract and MCP exposed as a thin facade where clients support it.

Keep the contract you can't afford to rewrite in Git and REST; expose MCP as a facade you can afford to replace.

Version the REST facade carefully. Treat MCP adapters as replaceable.

What the service actually is: a context broker​

The service has four jobs:

1. Merge — combine the layers (core, decisions, process, active) into one view.
2. Priority — apply the precedence stack so conflicts resolve deterministically.
3. Redaction — return only the caller's trust tier.
4. Defense-in-depth scanning — flag output that appears to reveal a tier the caller should not see.

The third job is the security boundary for prompt-injection exfiltration. With server-side redaction before retrieval, private context never enters the prompt for an untrusted caller. The fourth job can catch mistakes, but it cannot make unsafe retrieval safe.

Even a service still hits the standards wall​

The limitation: even with a hosted service, consumption stays uneven:

The hosted version still needs a shared standard. For one developer, the repo is usually enough. A service earns its complexity only with multiple trust tiers, multiple consumers, or a real injection threat model.

What's Next: The Standard That Doesn't Exist Yet​

This post is a proposal, not a product announcement. Today, none of this works automatically. Each tool reads its own context files, in its own format, from its own location. Vendors are adding memory, custom instructions, project files, and agent profiles, but not a shared context standard.

What's missing is a shared standard for where personal context lives and how to read/write it. The Model Context Protocol standardized tool integration; user identity needs the same kind of agreement. No shared standard exists today.

The GitHub repo approach is a bet: structured markdown plus a retrieval manifest could work if tool builders agreed to read it.

The ask to tool builders: Add an $AI_CONTEXT_PATH or equivalent. Let users point to markdown context. Portable context works when surfaces agree to read the repo.

This project started as hands-on image generation, not an abstract model experiment. I was working with SDXL (Stable Diffusion XL), an open text-to-image generation model, and the path felt natural: first run it locally on my own machine, then package the same work in a local container with Docker, then deploy that container to Azure Container Apps.

The local and local container stages were mostly smooth. They gave me enough confidence that moving from laptop to container to cloud would be more plumbing than discovery. The surprises showed up when self-hosting moved to Azure, where model files, process readiness, storage, and deployment behavior all became part of the system.

What the Model Produces​

Before the surprises, here is what the self-hosted SDXL pipeline actually generates once it is loaded and running. These is direct output from the same code, unretouched.

This is the payoff I was working toward. The rest of the post is about everything that stood between the container starting and this image coming out.

The False Assumption​

Calling a vendor API makes image generation look like one operation: send a request, get an image back. Self-hosting a generative model turns that single operation into a system I have to own.

I wanted control over inference settings, no lock-in to a hosted image API, and a cost model I could reason about. At the top level, the choice sounded simple: stop calling the vendor service and run the model myself.

That choice moved hidden responsibilities into my application boundary. I inherited the model runtime, storage lifecycle, readiness state, and deployment behavior. I also inherited the difference between persisted model assets and a process that has loaded the model and can generate an image.

The container was only the packaging format. Self-hosting meant owning everything the model needed after the container started.

The Architecture I Expected​

The system shape looked straightforward before the edge cases showed up. Local code would become a web API, the web API would run in a container, and the deployed container would use external storage for the model cache.

The first version looked like this:

The concrete version ran my Flask-wrapped SDXL (Stable Diffusion XL, an open image-generation model) code in Docker on Azure Container Apps (ACA, the managed container hosting service used here), with an Azure Files share (a network-attached mount) for cached model weights. An azd (Azure Developer CLI) postdeploy hook—an automation step after deploy—pulled the model.

The real deployment runs on CPU in Azure Container Apps: 4 vCPU, 16Gi memory, device=cpu, and 136Gi ephemeral storage, the container's local scratch disk, wiped on restart. That 4 vCPU / 16Gi shape requires an Azure Container Apps Dedicated (D4) workload profile; the Consumption plan caps at 4 vCPU / 8Gi. The mounted Azure Files share holds the model cache, because the SDXL assets are too large to treat as incidental container filesystem state.

That architecture was directionally right, but it left out most of the operational work.

What Was Actually Happening​

The first clean picture hid two separate truths: the container could be up without running Flask, and model files could exist without the process being ready.

Surprise #1: Persistent Storage Does Not Mean Warm Application State​

The first lifecycle mistake was treating stored model assets as if they were the same thing as a ready application. Persistent storage can keep files between revisions, ACA's immutable deployment versions. It cannot keep a new container process warm.

After the first cold download succeeded, I expected the next revision to be warm. The model files were on the Azure Files share, the share was mounted, and the path existed.

Then the app reported readiness state held in the process's memory, through the /model/status state field:

{
  "state": "not_started"
}

That looked wrong until I separated two states I had been mentally combining:

• model assets persisted on disk
• model loaded and ready in this process

Those are not the same lifecycle. The Azure Files share can be warm while the container process is cold. A new revision starts a new process. That process can see the cached files, but it still has to initialize the SDXL pipeline in memory.

The specific not_started state did not mean "the share is empty." It meant "this process has not begun loading the model." The useful ready state was ready, also reported by /model/status.

That distinction changed how I read status. A cold path downloads model assets and then loads the model. A warm path skips the download but still loads the model from the mounted cache. In my deployment, warm load from the cached share was about 48 seconds. Cold download took minutes.

Those are different costs, and they happen for different reasons. The deployment gate needed to care about readiness, not just file presence. Checking whether a directory exists is not enough. Checking whether the model cache is populated is not enough. The application has to report that this process is ready to serve generation requests.

Persistent storage keeps assets. It does not keep application memory warm. For model-serving systems, readiness is process state.

I built a small web console over these endpoints so I could see that distinction directly. Each endpoint the deployment depends on has its own section, and /model/status reports the process state in plain language: here it shows READY - Model weights cached on disk. /generate will load from cache, which is exactly the difference between "the files are on the share" and "this process can answer a request."

Surprise #2: Model Files Are Not Just Files​

The next storage mistake was treating model acquisition as a small deployment detail. For a generative model, the weights are a deployable asset with their own lifecycle.

A model that is about 7GB in FP16, or roughly 14GB loaded as FP32 on CPU the way this deployment runs it, is not a small deployment detail. It is its own deployment phase.

The model weights do not appear during azd up. Infrastructure provisioning creates the place where the model can live, but it does not populate that place with model assets.

I made model acquisition part of postdeploy. The deployment automation step that runs after deploy, the azd postdeploy hook, calls the app's model-download endpoint, POST /model/pull:

curl --fail -X POST "$APP_URL/model/pull"

Then it blocks until the app reports through the model-status endpoint, /model/status, that the model is ready:

curl --fail "$APP_URL/model/status"

A useful response includes the process readiness state, target device, and model path:

{
  "state": "ready",
  "device": "cpu",
  "model_path": "/models/stable-diffusion-xl-base-1.0"
}

That was the right shape, but the storage layer had its own constraints. The mounted network-attached file storage is an Azure Files share, which uses Server Message Block (SMB, the network file-sharing protocol Azure Files uses). SMB does not support POSIX flock, the file-locking call a local Linux filesystem supports. The first version of the download logic assumed file locking would behave like local disk, and that assumption broke on the mounted share.

That kind of bug feels like an operating system problem until you remember that self-hosting makes the filesystem part of the application architecture. I had to rework the download logic so the app did not depend on unsupported locking behavior on the mounted share.

Once that was fixed, the cold download was faster than I expected: about 2 minutes 23 seconds over the Azure backbone. The newer Hugging Face transfer path helped here; hf_xet, Hugging Face's newer fast download transport, replaced the deprecated hf_transfer, and the transfer itself was not the bottleneck I feared.

The useful takeaway was not simply that downloads can be fast. The model is a deployable asset with its own lifecycle. With a vendor API, the weights are someone else's problem. With self-hosting, model acquisition needs ordering, retries, logs, status, and a failure mode that stops the release instead of hiding the problem until the first image request.

Surprise #3: "CPU Offload" Doesn't Work on a CPU​

The runtime mistake was trusting a helper name before checking the hardware contract behind it. I expected memory to be an issue, and it was. The memory-saving helper I reached for had a name that sounded perfect for CPU hosting but failed because the container was actually running on CPU.

In diffusers, Hugging Face's Python library for running diffusion image models, the helper is enable_model_cpu_offload():

pipe.enable_model_cpu_offload()

The name sounds like exactly what a CPU deployment wants. I read it as: use CPU memory carefully, offload model pieces as needed, survive inside the container limits.

That is not what it means. On a pure-CPU container, it raises the kind of error that makes the naming clear:

requires accelerator, but not found

enable_model_cpu_offload() means "offload to CPU from an accelerator." It is for a system that has an accelerator and wants to move parts of the model back to CPU memory. It is not a CPU execution mode.

The fix was explicit CPU-safe initialization:

pipe = StableDiffusionXLPipeline.from_pretrained(
    model_path,
    torch_dtype=torch.float32,
    use_safetensors=True,
)

pipe.to("cpu")
pipe.enable_attention_slicing()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

The idiomatic diffusers pipeline-level equivalents are pipe.enable_vae_slicing() and pipe.enable_vae_tiling(); both forms are equivalent.

The memory-saving calls here are literal: attention slicing computes attention in smaller chunks, and VAE (the variational autoencoder stage that decodes latents into the final image) slicing and tiling decode the image in pieces.

Model libraries encode hardware assumptions. Sometimes those assumptions are obvious. Sometimes they are hidden inside method names that sound like they were written for your exact scenario.

For this deployment, "CPU offload" means "offload to CPU from somewhere else." It does not mean "run on CPU." The app is not just my Flask routes; it is also the model runtime, tensor dtype, memory behavior, and hardware profile lining up correctly.

Surprise #4: The Container Was Running, But Not My App​

The startup mistake was using a running container as proof that my application was running. A container can be healthy enough to accept traffic while the wrong process is listening.

The container app was up, the revision existed, the endpoint responded, and GET / returned 200. Every real route still returned a 404 Not Found from Python's static file server, with this line in the HTML body:

Message: File not found.

At first, that looked like my Flask routing was broken. Maybe the app was not binding correctly. Maybe the container port was wrong. Maybe the health route was missing. Maybe the image was stale.

The error page pointed to the real problem. Message: File not found. is not Flask's default response; it is Python's SimpleHTTPRequestHandler, the built-in static file server returning its HTML error page. My container was running, but my Flask app was not.

On a fresh environment, azd provisions the Azure Container App before the real application image exists. To make the infrastructure deployment succeed, it uses a temporary placeholder web server, python3 -m http.server 8000:

python3 -m http.server 8000

That is reasonable during provisioning. The surprise came later, when azd deploy swapped in my real Flask image and preserved the placeholder command. The image changed, but the runtime command did not.

So my real container image started successfully and then ran Python's static file server instead of my Flask app. That is why / returned 200, and why /health, /model/status, and /model/pull returned 404 Not Found responses whose HTML body said Message: File not found. Those routes only exist in Flask, and Flask was never running.

I stopped treating "container is up" as proof that the application is running. I added a self-heal step in the postdeploy hook that resets the command explicitly:

az containerapp update \
  --name "$CONTAINER_APP_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --command "python3" "app.py"

Then the hook waits for the actual application route before it does any model work:

curl --fail "$APP_URL/health"

Only after /health responds from Flask does the deployment continue. Calling /model/pull before proving Flask is running is just sending a request to whatever process happens to be listening.

I now treat the command, the image, and the health endpoint as three separate facts. The deployment is not ready until all three are true.

The self-heal works, but it treats a symptom. The root cause is that the container command was set in the Bicep template, and azd deploy swaps only containers[0].image, so the placeholder command survives and overrides the image's own start command. The cleaner pattern is to put CMD ["python3", "app.py"] in the Dockerfile, remove command and args from the Bicep entirely so the image command is used, and gate readiness with an ACA startup probe on /health instead of a manual wait loop. I kept the self-heal hook because it is what is working in this deployment, but if I were starting clean I would remove the Bicep command override and let the image plus a startup probe do this job.

Surprise #5: Tooling Silence Is Also a Failure Mode​

The verification mistake was trusting quiet tooling. Some failures throw obvious errors. Others look like nothing happened.

One problem was visibility. The azd postdeploy hook was running, but when its output was piped or non-interactive, stdout was invisible. Nothing in the terminal made it obvious what the hook was doing, so I verified through the platform logs:

az containerapp logs show \
  --name "$CONTAINER_APP_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --follow

Those logs became the source of truth.

Another problem was configuration shape. I had invalid keys in azure.yaml during one iteration. A top-level dockerfile: or port: looks plausible if you are moving fast, but azd did not fail the way I wanted. It ignored the invalid shape and fell back to default behavior.

The Dockerfile must be configured through the supported docker: block in azure.yaml:

services:
  image-generation:
    project: .
    language: docker
    host: containerapp
    docker:
      path: ./Dockerfile.cpu
      context: .

That small indentation decision changed what image was built.

The Dockerfile also had to default to the Flask server as its entrypoint. If the platform command was absent or wrong, the image still needed to know how to run the app:

CMD ["python3", "app.py"]

Without that default, ACA could end up in ContainerBackOff or ActivationFailed, ACA states for a container that cannot start or stay up, depending on which part of startup failed.

There was also an infrastructure bug: a circular dependency in the Bicep, Azure's infrastructure-as-code language, for the container app failed template validation until I broke the cycle. That was not an SDXL issue. The model deployment made the infrastructure graph more complicated, and the graph had to be correct before the app could even try to start.

Automation needs observable verification. In this setup, a successful command did not prove the deployment was correct, a running container did not prove Flask was running, a mounted share did not prove the model was ready, and a quiet hook did not prove the hook was idle.

What the Final Architecture Became​

The final shape is the concrete version of the earlier diagram. Each piece now has an explicit responsibility, and deployment gates on the application being ready, not just the infrastructure existing.

The container image defaults to Flask:

CMD ["python3", "app.py"]

The runtime behavior is explicitly CPU:

device=cpu
vCPU=4
memory=16Gi
ephemeral storage=136Gi

The model pipeline uses CPU-safe initialization:

pipe.to("cpu")
pipe.enable_attention_slicing()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

Here too, the pipeline-level pipe.enable_vae_slicing() and pipe.enable_vae_tiling() calls are the idiomatic diffusers form.

The Azure Developer CLI configuration points at the CPU Dockerfile through the supported shape:

services:
  image-generation:
    language: docker
    host: containerapp
    docker:
      path: ./Dockerfile.cpu
      context: .

The postdeploy hook does four jobs, in order:

1. Reset the container command to the Flask app.
2. Wait for /health so I know Flask is actually running.
3. POST /model/pull so model acquisition is part of deployment.
4. Poll /model/status until state is ready, with a configurable timeout and fail-fast behavior.

In shell form, the core idea is simple:

az containerapp update \
  --name "$CONTAINER_APP_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --command "python3" "app.py"

curl --fail "$APP_URL/health"
curl --fail -X POST "$APP_URL/model/pull"

until curl --fail "$APP_URL/model/status" | grep '"state":"ready"'; do
  sleep 10
done

The real script has more defensive handling, because production scripts should fail clearly. But that is the architecture.

The app owns readiness. The hook gates deployment on readiness. Logs validate reality. I did not end up with just a container that runs SDXL; I ended up with a deployment lifecycle for a self-hosted generative model.

The Decision Framework I Actually Trust Now​

I still like the decision to self-host for this project. The tradeoff is just clearer now.

Self-hosting buys control over inference settings, portability, model loading strategy, and deployment lifecycle. It also moves hidden responsibilities into your application boundary: runtime assumptions, model storage, download orchestration, readiness, deployment verification, logs, sizing, and the difference between "files exist" and "the model can answer this request."

A vendor API charges for convenience, but the convenience is real. It is not just inference. It is the operational surface area you do not have to build.

For a prototype, that surface area may not be worth it. For a workflow where settings, portability, and control matter, it can be.

The question I trust now is simpler: do I want to own everything this model needs to be reliable?

Closing​

Self-hosting SDXL showed me how much the vendor API had been handling. Once I owned the model, I owned the runtime, storage, lifecycle, readiness, and observability around it.

Self-hosting a generative model is not just replacing an API call with a container. It means the model is part of the system, with the same deployment and reliability responsibilities as the rest of the application.

And once all of that is in place, the model just does its job:

For the engineers building with AI every day — and the leaders setting the guardrails around them.

Take a few minutes to think about where we are. We're past the demo phase. The exciting question used to be "can a model even do this?" — and now it's the much less glamorous "how do we run this every day, at a price we can actually justify, without handing an autonomous process the keys to everything?"

That question really has three threads tangled together, and it helps to pull them apart up front:

• The money — don't pay twice, and use the right tool for the job.
• The craft — move from one-shot prompts to repeatable, testable processes.
• The guardrails — least privilege, real observability, and accountability for things that aren't people.

There's one idea underneath all three: the graduation handoff. Almost everything here describes a single moment — a skill or agent growing up from "you're watching it work, turn by turn" into "it runs in production on its own." Before that moment, you are the scaffolding that earns the trust. After it, your standard production stack — telemetry, observability, authorization — becomes the permanent structure that keeps it. Keep that handoff in the back of your mind. It's the spine of everything below.

That watching phase—where you're present for each turn—is the subject of what comes next. It's not forever, but it's the cost of earning trust.

Part 1 — The money: stop paying twice​

Pay for tokens once​

Here's the thing about tokens: they're wonderful for exploring and terrible as a permanent runtime. Once you really understand a workflow, push it toward determinism — scripts, apps, tests, CI — so you pay the model once to figure it out, not on every single run.

I want to be straight with you, though: determinism isn't a free lunch, it's a trade. Scripts rot. APIs change, schemas drift, dependencies break. You're swapping a per-run token cost for a smaller, recurring maintenance cost — not erasing the cost.

So when do you actually harden something? It comes down to stakes. If a workflow is quick and clearly defined, move it to determinism early. If the consequences are bigger, give it a middle ground: let it mature with AI first, and as individual parts settle, move those parts into hardened, deterministic form — within the real-world limits of security, maintenance, and interoperability. The two failure modes to avoid: hardening a moving target, and paying frontier prices for something that's already settled.

Use the right tool for the job — on a cost gradient​

"Use AI for everything" isn't a strategy. The real principle is the boring, durable one you already know: right tool for the job. And the tools sit on a gradient — a classic deterministic tool, then a small or local model, then a frontier model. Pick the cheapest rung that actually does the work.

Choosing the right one saves money, but more importantly, it saves cognitive load. You're not wrestling with overkill; you're matching capability to need.

Spelling and grammar is a nice illustration, precisely because it usually lands on the deterministic rung. Mature tools handle it cheaply and predictably, so you'd never reach for a frontier model — same instinct as "pay once." The interesting rung is the one above deterministic but below frontier, and that's where small or local models genuinely earn their keep: classification, routing, redaction, embeddings — work that needs more flexibility than a fixed tool can give but doesn't need state-of-the-art reasoning.

A concrete example I like: redacting PII from your logs before they're stored. A regex can't reliably catch a name or address it's never seen. A frontier model is overkill on every log line. A small local model is right in the sweet spot — flexible enough to generalize, cheap enough to run on every write, and local so the sensitive data never leaves your boundary.

Here's how to think about it systematically:

Turn that gradient into a system: model routing​

Don't leave "good enough vs. state of the art" as a gut feeling. Systematize it with model routing, or cascades: try the cheap model first, and escalate to the frontier one only when confidence is low. Same instinct as above — but now it's a measurable, tiered spend you can actually reason about instead of a vibe.

Don't pay for compute you don't need — and remember local buys more than savings​

If you don't need cloud or remote access, a small or local model lets you skip the ongoing API bill. But please don't fall for "local is free" — it's a myth. You're really just swapping one kind of cost for two others: you trade a pay-as-you-go bill for a big upfront purchase plus a new set of ongoing costs — hardware, electricity, wear-and-tear, and the engineer-hours to run and patch the thing. For low or spiky volume, cloud is often both cheaper and more secure than a box you babysit.

So find the real break-even: volume, latency, and data-residency or compliance constraints — not cost alone. And notice the two things local buys you beyond money. First, privacy and residency — the data never leaves your boundary, which is honestly the real reason to go local more often than cost is. Second, a chance to work in an emerging space. Local model management is still young — tools like Ollama, LM Studio, and Docker's Model Runner have made a real start, but enterprise basics are still thin: per-user access control, audit logging, versioning, and cost tracking. If you like building tools, there's real room here.

Treat AI spend like any other tech spend​

After all this talk of cost, let me be clear about the goal: it isn't to fret over every dollar. It's the opposite. AI spend deserves the same deliberate budgeting you give every other part of your stack — it's real, it's recurring, and it should be a line item you own on purpose, tracked per team and per workflow.

Part 2 — The craft: from prompt to process​

AI builds the prototype; you harden what matters​

One of the most durable patterns here is an old one wearing new clothes. AI builds the prototype fast; once it works, you decide what becomes a repeatable, testable, secure process.

What's the artifact of that transition? Today it's a skill, an agent, or some other markdown file — but the format is incidental. What matters is the information that persists: a durable, reviewable source of truth that outlives whatever wrapper happens to hold it.

The work is real and tangible—you're shaping raw materials into something refined and usable:

And here's a bonus that's easy to miss: the hardening boundary is also the line between your two test regimes. What you've hardened gets a deterministic unit test. What stays probabilistic gets an eval — golden sets, LLM-as-judge, acceptance bands. Deciding what to harden is deciding how each piece gets tested. The test surface didn't shrink when AI showed up; it grew to cover both. Good news: the frameworks for this already exist, so you don't have to invent them.

Move from one-shot prompts to repeatable processes — and give them an owner​

Wrap your deterministic scripts in AI skills that lean toward repeatable processes anyone can pick up. But reuse without ownership just turns into shadow IT — handy, but unowned and ungoverned. So give every reusable thing a home.

A frame I borrowed from a governance thinker in the company helps here: think federal, state, and local, where you, the individual contributor, are local. Federal assets are organization-wide. State builds on federal. Local builds on both. The pipeline updates assets as they change upstream — a local copy can be refreshed from its non-local source instead of drifting out of date. The point is that every reusable unit has an owning tier and an update path. That's the difference between a library and a junk drawer.

Chain and gate skills into workflows — but only what you can watch​

You can absolutely chain skills into full workflows — just respect the math, because reliability compounds downward. Five steps at 95% each lands you around 77% end to end. So chain only what you can observe and gate, and keep your chains short.

What's a "gate"? Any defined checkpoint — a human in the loop or an AI, as long as it's explicit. The simplest version: look at the output or log of the last skill; if it has everything the next skill needs and shows no failures, move on. And a gentle reminder: more agents doesn't automatically produce better results.

A team of specialists beats one do-it-all generalist​

There's a lot of research showing that teams of specialists outperform teams of generalists, and the same holds for AI agents. If you need a team, a set of focused specialist agents will beat one general-purpose agent copied over and over to fill every seat.

The usual objection to agent teams is that things get lost in the handoff — when one agent passes work to the next, the second agent doesn't know what the first one already figured out. That objection is fading fast. Most AI platforms now give agents shared memory (Squad does this for me today), so the specialists all read from and write to the same memory. Nothing has to be re-explained and lost at each step; the context is simply there for whoever needs it next.

So what's actually left to weigh? Mostly cost. A team of agents costs more than a single agent, because each one does its own thinking and runs up its own bill. That's the real trade-off — not lost context — and it's usually worth paying when each specialist does its part better than a generalist would.

The only time to reach for a single agent is when a team would be overkill: a small, simple job where spinning up specialists adds cost and coordination for no real gain. That's a simplicity call, not a limit on what specialists can do.

Capture tribal knowledge — turn repeatable work into skills​

Here's a simple test: if you had to hand your work to someone else before going on vacation, and you'd explain a process step by step, that process is probably a skill. The moment you find yourself writing "first do this, then do that, then check for this" — you're describing something a skill can hold. Capture it.

Not everything passes the test. Codify the parts that are repetitive and stable; leave the genuinely judgment-heavy or one-off work to a person. Repetition is the tell — when you catch yourself doing the same thing the same way again and again, that's the signal.

Part 3 — The guardrails: things that aren't people​

Least privilege for agents — and the third thing that's easy to miss​

You already put real effort into deciding what access your people should have. Do the same for agents — don't let them roam your systems with your full authorization. And notice there are three distinct ideas here, not two:

• Identity — who the agent is. It'll likely look a lot like an app or service principal (Entra Agent ID and similar platforms point that way).
• Authorization — what it's allowed to do. Scope it as carefully as you'd scope a person.
• Accountability — who answers when it causes harm.

With an employee, all three exist: a badge, access grants, and a liable person with intent, a contract, and consequences. With an agent, even identity is messier than it looks. Either the agent borrows a real person's identity — so every action it takes shows up as that human doing it — or it runs as a service principal that owns resources outright, with no person attached at all. Neither option cleanly separates the three ideas, and accountability has no home in either. When a correctly-scoped agent still deletes prod, leaks a secret, or does something destructive a human would've paused on, who's liable? The agent has no intent. The IC who launched it didn't author the step. The team that built the skill didn't run it. The vendor's model chose the action.

This is what separation of concerns looks like in practice—distance and clarity between who's watching and who's doing:

The principle is simple—scope each agent to the single, smallest set of permissions it actually needs. Nothing more. Think of it as giving someone only what they need to do the job, not the keys to everything. There's one more wrinkle worth knowing about: an agent can be tricked into misusing access it legitimately has. A well-documented pattern (often called prompt injection) is when text the agent reads — a web page, a file, an email — contains hidden instructions, and the agent treats them as if you'd asked for them. The access was scoped correctly; the agent just used it on the wrong instructions. Weigh it when you decide how much an agent should be allowed to do on its own.

Log the output your gates depend on​

"Log everything" needs a sharper edge, because prompt-and-response logs are a huge new sensitive-data surface. So let me be specific about what I actually mean: not the prompts and responses, but the core output of each step — the data flowing through your scripts.

That's the logging that makes chained gates effective. A gate can only ask "did the last step produce what the next one needs, with no failures?" if that output is captured. It's also what tells you when something went wrong and how much it affected. Without those logs, you can't even see how far the damage spread. And that's really the point: before anything runs on its own in production, you map out how far a failure could spread and build up trust in the process over a good stretch of running it with a person watching.

AI won't stay a black box — until it earns the right to be one​

You're going to have to understand what's happening at levels you used to happily ignore. The usual car analogy — turn the key, drop it in gear, hit the gas, never think about the engine — actually cuts the other way right now. Cars earned that abstraction by working reliably for a century. AI hasn't earned it yet.

You need to see inside the box until it consistently succeeds. There's a whole range from "barely works" to "works every single time," and watching turn by turn is the discipline for the immature end of it. The watching doesn't disappear as the system matures — it changes form. When a skill or agent graduates from "watch every turn" to "production-automated," your bespoke attention gets handed off to the standard production stack: telemetry, observability, authorization. Manual vigilance is the scaffolding you use to earn trust; mature observability is the permanent structure that keeps it. That's the graduation handoff again — the very same moment that hardening, scoped authorization, and step-output logging each describe from their own corner.

Where this leaves us​

Read straight through, these aren't a dozen scattered tips — they're one lifecycle. A capability starts life as an expensive, non-deterministic, hand-watched experiment, and if it proves out, it graduates into something cheap, deterministic, governed, and observable. The money tells you which rung to run it on. The craft tells you how to harden and test it. The guardrails tell you what it can touch and who answers when it goes wrong. And the handoff — your scaffolding giving way to standard ops — is where all three meet.

That journey is real work—intentional, deliberate, with a clear view of where you're building:

And this is how it unfolds end to end—from experiment to automation, from manual scaffolding to permanent structure:

So that's the real work in front of teams and organizations right now. It isn't adopting AI. It's graduating it. If you're already wrestling with any of this, I'd genuinely love to hear what's working for you and what isn't — that's how all of us find the real value faster.