Notes from the work

One email when a tutorial or migration write-up goes live. Nothing else, and an unsubscribe link in every email.

LeanZero

Two people in Romania doing Atlassian migrations, Forge apps and practical AI work for teams that would rather talk to the person doing the job. Most of what we learn ends up on this site.

Services

  • Atlassian migrations
  • Atlassian FastShift
  • Atlassian maintenance
  • Forge app development
  • AI development consultation

Topics

  • Jira
  • Jira Service Management
  • Confluence
  • Bitbucket
  • Atlassian Forge
  • Cloud Migration
  • Local AI
  • AI Coding
  • Claude
  • Certifications
  • Atlassian Team
  • All topics

Company

  • Blog
  • Tutorials
  • Contact

Community

  • Join Discord
  • Support this site

© 2026 LeanZero. All rights reserved.

Privacy policyTerms of service24/7 SupportTrust centerSecurityDPALegal notice
  1. Home
  2. Topics
  3. Ai Coding
Topic hub

AI Coding

Coding agents in practice, and how to tell whether they were right.

Read the articlesAll topics

AI coding agents fail in specific, repeatable ways, and almost none of them look like the demos. A test command piped through head exits 0 when nothing ran. A 27B model re-reads a file it was handed moments ago, ninety-five percent of the time. A swarm reaches agreement with itself and ships the bug anyway.

Most of this was written while building LeanZero's own agent running against local models, so the examples come from a harness that had to actually work. Expect measured before-and-after numbers and the levers that moved them. Expect a lot on verification too: making an agent act is the easy half, and knowing whether it was right is the half that takes the work.

Read in order

Agent control protocols

4 parts

What actually constrains an AI coding agent: the tool you run it in, the protocol underneath, and what the wire really carries.

  1. 1Part 1 — ArticleThe Obedience Trade-off: Why I Swapped Claude Code for Zed + GLM-4.710 min
  2. 2Part 2 — ArticleAI Agent Control Protocols: The Missing Link in LLM Engineering5 min
  3. 3Part 3 — ArticleClaude Code in Zed vs the terminal: what the ACP wire actually carries16 min
  4. 4Part 4 — ArticleWhat 145 MCP Tools Cost Before the Model Reads Your Question14 min

Agents that run unattended

3 parts

What it takes to leave an AI agent running without a human watching, and the ways it quietly stops working.

  1. 1Part 1 — ArticleIntroducing the MCP Document Processor: A Practical Tool for Document Processing5 min
  2. 2Part 2 — TutorialAgent Skills, working: a guide with 6 real Atlassian examples12 min
  3. 3Part 3 — TutorialYour Unattended AI Agent Looks Healthy and Does Nothing: 8 Silent Failures on macOS launchd (2026)12 min

Articles 38

Qwen4 Max benchmark: below Qwen3.8 Max Prime on both our boards, at a tenth of the cost
ArticleAI Coding

Qwen4 Max benchmark: below Qwen3.8 Max Prime on both our boards, at a tenth of the cost

Qwen4 Max scored 0.7365 on our Gauntlet coding benchmark and 0.3473 and 0.4087 on Forge 2.0, below Qwen3.8 Max Prime on both, for $9.30 against $88.72. On Gauntlet it earned slightly more than Max Prime and lost the difference to a score ceiling. We also ran Baidu's ERNIE Light DT 1.0.

Nov 18, 202614 min read
Ricursive Ripple-65B Preview benchmark: 0.028 on Gauntlet, 0.0814 on Forge, plus LFM3-1.7B
ArticleLocal AIAI Coding

Ricursive Ripple-65B Preview benchmark: 0.028 on Gauntlet, 0.0814 on Forge, plus LFM3-1.7B

We ran Ricursive's Ripple-65B Preview and Liquid AI's LFM3-1.7B through our agentic coding benchmarks. Ripple scored 0.028 on the Gauntlet: one wiring line broke its API, and its tests went around the server. Its two Forge 2.0 builds scored 0.0858 and 0.0770, a little above the 0.0617 of the unchanged starter app. LFM3-1.7B stopped itself with 30 calls left and changed no file on Forge 2.0. On a Mac, llama.cpp runs both and mlx-lm runs neither yet.

Nov 11, 202616 min read
Nanbeige4.3-33B benchmark attempt on a Mac Studio, plus Particle-1 vs DeepSeek V4.1 Flash
ArticleLocal AIAI Coding

Nanbeige4.3-33B benchmark attempt on a Mac Studio, plus Particle-1 vs DeepSeek V4.1 Flash

We tried to benchmark Nanbeige4.3-33B on a Mac Studio with MLX and it never started: our engine dropped its tool calls, and the 96 GB Studio served at most 75,925 tokens of the 322,851 our Gauntlet needs. We can give you measured speed and memory, a fit table by Mac size, and Particle-1 measured against two DeepSeek V4.1 Flash snapshots.

Nov 6, 202624 min read
Qwen3.8-Flash-Next on a Mac Studio vs OpenRouter: what it took to reach 0.6932
ArticleLocal AIAI Coding

Qwen3.8-Flash-Next on a Mac Studio vs OpenRouter: what it took to reach 0.6932

Our own 4-bit DWQ build of the official Qwen3.8-Flash-Next weights ran on one M5 Max Mac Studio in our MLX engine and scored 0.6932 on our Gauntlet sb-7.2 coding benchmark. Qwen3.8 Flash on OpenRouter has 0.6846 on the same board, and I explain why that comparison isn't settled. This is what we built to get there, with the numbers, and what went wrong on the way.

Nov 4, 202620 min read
GPT-6.1 Terra benchmarks on agentic coding: 13th of 48, and what beat it
ArticleAI CodingAtlassian Forge

GPT-6.1 Terra benchmarks on agentic coding: 13th of 48, and what beat it

GPT-6.1 Terra scored 0.6953 on our Gauntlet, 13th of 48, and 0.4384 and 0.4174 on two Forge 2.0 builds, for under $1.10 a run. It passes almost as many checks as GPT-6.1 Sol. It loses on what a demo never checks: a background reconcile that never finished, a flick it never tested, and a six-hour run it admitted it hadn't verified.

Oct 28, 202616 min read
Muse Spark Open benchmark: where the open-weight models we ran this month broke as coding agents
ArticleAI CodingAtlassian Forge

Muse Spark Open benchmark: where the open-weight models we ran this month broke as coding agents

Meta's Muse Spark Open scored 0.091 and 0.0877 out of 1 on our Forge 2.0 coding benchmark and never got a Gauntlet verdict. The other open-weight models we ran locally, Google's Gemma 4 among them, did no better, and neither did our own Qwen3.8-27B fine-tunes. The best open result on our Macs came from the plain Qwen3.8-27B base. We read every build to find where each one broke, and a fair share of the breakage was ours.

Oct 22, 202620 min read
Are AI benchmarks reliable? We had 32 models build the same app twice
ArticleAI CodingAtlassian Forge

Are AI benchmarks reliable? We had 32 models build the same app twice

Gemini 4 Argon built our Forge 2.0 test app twice and scored 0.79, then 0.29. Claude Sonnet 5.5 scored 0.59 and 0.55. I wanted to know which models are the unpredictable ones. Two builds can't tell you, and part of one swing was our own harness.

Oct 22, 202620 min read
Mistral Large 4 coding benchmark: the money bug was ours, and DeepSeek V4.1 Flash still won on Forge
ArticleAI CodingAtlassian Forge

Mistral Large 4 coding benchmark: the money bug was ours, and DeepSeek V4.1 Flash still won on Forge

Mistral Large 4 scored 0.5462 on our Gauntlet because a check of ours compared its newest payments with our oldest ones. Every amount it showed was right. On Forge 2.0 it was slow and expensive, mostly for reasons of its own, and DeepSeek V4.1 Flash 1017 built the same app better in about a quarter of the time, for about an eighteenth of the bill.

Oct 21, 202618 min read
Best model for agentic coding in October 2026: who won our benchmarks, and who surprised
ArticleAI CodingAtlassian Forge

Best model for agentic coding in October 2026: who won our benchmarks, and who surprised

We ran 44 models through two agentic coding benchmarks, then read their builds line by line. Paying per token I'd use GPT-6.1 Sol, and MiMo V2.6 Flash came 7th on our Jira benchmark at 31 cents a build. The winners mostly did dull things right, and Claude Haiku 5.5 stopped with the app half written.

Oct 20, 202615 min read
DeepSeek V4.1 Flash 1017 benchmark: same task, two builds, 0.24 and 0.66
ArticleAI CodingAtlassian Forge

DeepSeek V4.1 Flash 1017 benchmark: same task, two builds, 0.24 and 0.66

One Sunday, three Macs, two agentic coding benchmarks. A one-day-old DeepSeek point release scored above its predecessor on both, and its own two Forge builds landed 0.42 apart. NVIDIA's Nemotron 3.5 Super VL lost its context halfway through a build. A local model declared the same name twice and reported the app ready. And our board had been understating what the runs cost.

Oct 19, 202616 min read
Four Qwen3.8-27B fine-tunes and merges on a coding benchmark: none beat the base
ArticleAI CodingAtlassian Forge

Four Qwen3.8-27B fine-tunes and merges on a coding benchmark: none beat the base

We ran four popular Qwen3.8-27B fine-tunes and merges through Forge 2.0, our long agentic coding benchmark, on one 64 GB MacBook next to the stock base, one build each in our own 8-bit conversions. In our setup one model's write calls came back malformed, one wrote careful code and then got stuck hunting for a method that doesn't exist, one finished fast with placeholders, and the merge got the hardest parts right and scored 0.1105. The base's median is 0.1852.

Oct 19, 202617 min read
Jev Router topped our coding benchmark. 117 of its 118 calls went to GPT-6.1 Sol
ArticleAI CodingAtlassian Forge

Jev Router topped our coding benchmark. 117 of its 118 calls went to GPT-6.1 Sol

TypeSafe's Jev Router took the top median on our Forge 2.0 agentic coding benchmark, 0.746 from two builds. OpenRouter's own generation records show it sent 117 of its 118 calls to GPT-6.1 Sol, which sits third on the same board. What a model router is, what Jev Router did here, and why two long builds per model can't tell a router from the model it picked.

Oct 17, 202614 min read
Best local image generation model on a Mac? ERNIE-Image Turbo won our blind test
ArticleAI Coding

Best local image generation model on a Mac? ERNIE-Image Turbo won our blind test

I ran six open image models through six prompts in mflux on a Mac Studio M3 Ultra and had two blind AI judges score the 48 images. ERNIE-Image Turbo scored 4.28 of 5 at about 80 seconds an image, ahead of its own 13-minute base model. FLUX.2 klein 4B is 3.6 times faster, scored 3.07 and misspelled the shop banner on both seeds. I also read each model's license for commercial use.

Oct 16, 202619 min read
Claude Code Hits a Jira Workflow Validator: CogniRunner Blocks the Change Request
TutorialCogniRunnerJira

Claude Code Hits a Jira Workflow Validator: CogniRunner Blocks the Change Request

I gave Claude Code a service-desk agent's API token and asked it to move a change request to Ready for CAB. Two plain-English CogniRunner validators sent back HTTP 400 with both reasons. Claude fixed the backout plan, left the failed rehearsal report alone and handed it back. I ran it three times and got the same outcome.

Oct 15, 202617 min read
Jira local LLM integration: pin one HR project to LM Studio with CogniRunner
TutorialCogniRunnerJira

Jira local LLM integration: pin one HR project to LM Studio with CogniRunner

One Jira Cloud project sends its AI checks to a Qwen model in LM Studio on our Mac Studio, and the site stays on its cloud provider. I ran it 8 times on 14 October 2026: 8 refusals and 8 passes, about three seconds per click. Then I saved a wrong token and the request went straight through.

Oct 14, 202615 min read
LLM regression testing for a Jira workflow validator: CogniRunner before you switch models
TutorialCogniRunnerJira

LLM regression testing for a Jira workflow validator: CogniRunner before you switch models

A Jira filter of 8 labelled requests, an 80-line Node script and CogniRunner's rule test over REST. I ran the set on the model the site uses and on the one I wanted to move to. The site's model matched 6, 7 and 7 of 8, the candidate 8 of 8, and one sentence in the rule brought both to 8 of 8 before I switched anything.

Oct 14, 202617 min read
DeepSeek V4.1 Flash coding benchmark: 0.33 on a Forge app upgrade, Opus 5.5 got 0.67
ArticleAI CodingAtlassian Forge

DeepSeek V4.1 Flash coding benchmark: 0.33 on a Forge app upgrade, Opus 5.5 got 0.67

I ran DeepSeek V4.1 Flash twice through Forge 2.0, our Atlassian Forge app-upgrade benchmark. It scored 0.2893 and 0.3790, a median of 0.3342 against 0.6745 for Claude Opus 5.5 and 0.5283 for MiMo V2.6 Flash. Then I read the code behind its biggest penalty, and I don't think our scorer got that one right.

Oct 13, 202614 min read
Claude Haiku 5.5 vs GPT-6 Luna on two coding benchmarks: same price, 3.5x the agent bill
ArticleAI CodingAtlassian Forge

Claude Haiku 5.5 vs GPT-6 Luna on two coding benchmarks: same price, 3.5x the agent bill

Claude Haiku 5.5 and GPT-6 Luna both list at $0.10 in and $0.50 out per million tokens. In our agent runs Haiku billed 3.5 times as much, mostly because 64 of its 111 calls crossed the 100k-token line where its price goes up five times, and part of that was a bug in our own agent. It also built the better apps: 0.9661 to 0.395 on an Atlassian Forge app, 0.661 to 0.4797 on a payments app. The checks that decided both, the call-by-call bill, and how to keep Haiku under the step.

Oct 12, 202622 min read
Jira automation rules API and cloud-to-cloud migration: two new agent skills
ArticleAI CodingJira

Jira automation rules API and cloud-to-cloud migration: two new agent skills

Two new skills in LeanZero's open-source agent skills pack. automation-engineer drives the Jira automation rules API under /automation/public/ to read, create and migrate rules between Cloud sites. atlassian-cloud-to-cloud-migration-skill copies Jira projects, JSM desks and Confluence spaces into a live site over REST. How to install them, what each one has proven, and what it hasn't.

Oct 3, 20267 min read
CogniRunner Coder: a Jira coding agent that commits to GitHub from the issue
TutorialCogniRunnerJira

CogniRunner Coder: a Jira coding agent that commits to GitHub from the issue

A Jira coding agent that works inside the issue: it reads your GitHub repository, plans, stages the edits, and on its default setting commits or opens a pull request only after you confirm a card, with the commit card naming every file. Walked end to end on our production build, with the pull request it opened and how to check it, and what 5.2.0 changed.

Oct 2, 202613 min read
lsof: command not found in a packaged Mac app, and the port reclaim that never ran
ArticleAI CodingLocal AI

lsof: command not found in a packaged Mac app, and the port reclaim that never ran

Our port reclaim logged a warning every single time it ran in the shipped app, and nobody noticed for weeks — because the warning branch was the only branch that could execute there. macOS keeps lsof in /usr/sbin, and a packaged app's PATH does not have it.

Sep 5, 20268 min read
I did not think certifications were worth much. Then one of them showed me what I had been missing
ArticleCertificationsAI Coding

I did not think certifications were worth much. Then one of them showed me what I had been missing

I passed the Claude Certified Architect and ACP-120 within days of each other. One of them changed how I build agentic workflows, and not for the reason I expected. Here is the honest version, and what I am building next.

Sep 3, 20268 min read
The best 4-bit format in MLX is the one its own converter sets up to lose
ArticleAI Coding

The best 4-bit format in MLX is the one its own converter sets up to lose

NVFP4 has a second scaling level. mlx-lm never sets it, so NVFP4 converts to the worst 4-bit result on the table — worse than a format using fewer bits. Set it and NVFP4 wins. I nearly published this as a defect in NVIDIA's spec; it was my own unit error.

Sep 1, 20269 min read
Recover every Jira worklog when the API hands you only the oldest twenty
TutorialJiraAI Coding

Recover every Jira worklog when the API hands you only the oldest twenty

getJiraIssue reports worklog.total correctly and then returns twenty entries. Sum those and your monthly report is quietly short — one person measured 21%. Five steps to detect the truncation, recover the full set, and know exactly which part of your number is proven.

Aug 29, 202612 min read
What 145 MCP Tools Cost Before the Model Reads Your Question
ArticleAI CodingLocal AI

What 145 MCP Tools Cost Before the Model Reads Your Question

MCP went stateless on 2026-07-28. I probed ten real servers: none implement it. Then I measured the thing that actually costs you — 145 tool definitions, 40,784 tokens, and the 30% that is one server repeating itself.

Aug 17, 202614 min read
96 GB is 77.76 GiB: the real memory ceiling on an M3 Ultra
ArticleAI CodingLocal AI

96 GB is 77.76 GiB: the real memory ceiling on an M3 Ultra

Metal will not give you the RAM on the box, and the number it does give is not the 75% everyone repeats. I measured the three ceilings on a 96 GB Mac Studio, then measured what modern hybrid-attention models actually spend against them — including a Gemma 4 cache that quietly holds three times its own sliding window.

Aug 13, 202613 min read
llama.cpp vs MLX on Qwen3.6-27B: MTP is 1.04x here, not 1.85x
ArticleAI CodingLocal AI

llama.cpp vs MLX on Qwen3.6-27B: MTP is 1.04x here, not 1.85x

Multi-token prediction is merged in llama.cpp and still an open PR in mlx-lm. I measured both on an M3 Ultra with the same model. Every default MTP setting was slower than no MTP at all, and the runtime that deletes the MTP head outright was still the fastest thing I ran.

Aug 10, 202615 min read
Ling 3.0 Flash won't load on a Mac Studio. Qwen3-Coder-Next does, at 73 tok/s.
ArticleAI CodingLocal AI

Ling 3.0 Flash won't load on a Mac Studio. Qwen3-Coder-Next does, at 73 tok/s.

Ant Group's 124B/5.1B-active hybrid-linear MoE hit Hugging Face on 2 August. The memory arithmetic fits a 96 GB Mac with room to spare, and mlx-lm still refuses it. I counted exactly which tensors block it — 385 of 62,237 — then measured what that sparsity actually buys on the models that do run.

Aug 5, 202618 min read
Claude Code in Zed vs the terminal: what the ACP wire actually carries
ArticleAI Coding

Claude Code in Zed vs the terminal: what the ACP wire actually carries

I wrote my own Agent Client Protocol client, ran the same bug-fix fourteen times, and logged every frame. The agent never once touched my client's filesystem — and inside the working directory, the editor's permission dialog never sees a read at all.

Aug 2, 202616 min read
MXFP8 vs Q8: 10x the weight error, 1% the perplexity
ArticleAI CodingLocal AI

MXFP8 vs Q8: 10x the weight error, 1% the perplexity

I quantised real Qwen3-Coder weights both ways on an M3 Ultra. MXFP8 reconstructs them about 10x worse than 8-bit affine at identical size — and then costs only 1% perplexity end to end. Both numbers are true, and the gap between them is the interesting part.

Aug 1, 202614 min read
Your Unattended AI Agent Looks Healthy and Does Nothing: 8 Silent Failures on macOS launchd (2026)
TutorialAI Coding

Your Unattended AI Agent Looks Healthy and Does Nothing: 8 Silent Failures on macOS launchd (2026)

An agent scheduled with launchd wrote 288 healthy log lines a day and produced nothing. Here are the eight failure modes behind that, each of which looks identical from the outside, and the checks that make them loud.

Jul 27, 202612 min read
MLX vs GGUF on Apple Silicon: Benchmarking the Same Local Model Two Ways
ArticleAI CodingLocal AI

MLX vs GGUF on Apple Silicon: Benchmarking the Same Local Model Two Ways

We built a harness that makes local coding agents produce real software, grades it by running it, and ran the same model as GGUF and MLX. Here's the harness, its modes and archetypes — and which build to run.

Jul 2, 202612 min read
What's New in CogniRunner: Local AI, Zero-Key Models & Actions That Write Back
ArticleAtlassian ForgeAI Coding

What's New in CogniRunner: Local AI, Zero-Key Models & Actions That Write Back

CogniRunner's latest update brings local AI inference, a zero-key Atlassian-hosted model, and declarative AI actions that write back to Jira — all configured in plain English.

Jun 18, 20266 min read
Agent Skills, working: a guide with 6 real Atlassian examples
TutorialAI Coding

Agent Skills, working: a guide with 6 real Atlassian examples

Agent Skills are how you give a coding agent reliable knowledge of a framework it's never read. This guide walks the mechanics — progressive disclosure, SKILL.md, auto-trigger via description — and demonstrates each step with one of six open-source Atlassian skills you can install in two minutes.

May 19, 202612 min read
Qwen3-Coder-Next + MXFP8: The 128GB Local LLM That Runs Predictably
ArticleAI CodingLocal AI

Qwen3-Coder-Next + MXFP8: The 128GB Local LLM That Runs Predictably

A deep dive into Qwen3-Coder-Next, the 80B MoE model that brings powerful local development within reach. Learn why this 2026 release matters for developers who value control, predictability, and consistent results.

Feb 4, 202610 min read
AI Agent Control Protocols: The Missing Link in LLM Engineering
ArticleAI Coding

AI Agent Control Protocols: The Missing Link in LLM Engineering

A systematic approach to controlling how AI agents behave when working on software projects. Addresses the fundamental problem that AI models are unpredictable by default.

Jan 28, 20265 min read
The Obedience Trade-off: Why I Swapped Claude Code for Zed + GLM-4.7
ArticleAI Coding

The Obedience Trade-off: Why I Swapped Claude Code for Zed + GLM-4.7

Autonomous AI agents are impressive, but their "helpfulness" often leads to unauthorized changes. After Claude Code silently removed rate limiters and rewrote my email templates, I found a more reliable alternative in the Zed Editor paired with GLM-4.7. Description: A technical deep-dive into the risks of "Vibe Coding" and AI overreach. This post analyzes a real-world incident where an autonomous agent broke production features while fixing bugs, and explores why the strict instruction-following capabilities of Zhipu's GLM-4.7 model inside the Zed editor offer a safer, more deterministic workflow for serious engineering.

Jan 24, 202610 min read
Introducing the MCP Document Processor: A Practical Tool for Document Processing
ArticleAI Coding

Introducing the MCP Document Processor: A Practical Tool for Document Processing

Discover how the MCP Document Processor helps AI agents work with document formats more effectively. This Model Context Protocol server provides tools for reading PDFs with OCR and creating DOCX and Excel files. Tested with Qwen3-8b-VL for vision processing and GLM-4.7-REAP for cloud models, it handles document structure, table extraction, and professional document generation with practical styling options.

Jan 19, 20265 min read