CogniRunner: Jira OpenAI, Claude or Forge LLMs behind one workflow validator
Mihai Perdum
Author
14 min readOctober 1, 2026
Key takeaways
END STATE: a Jira workflow transition guarded by a CogniRunner AI validator, running on the provider you chose, tested with one REST call that returns HTTP 400 for bad input and HTTP 204 for good input.
CogniRunner 5.0.0, deployed to production on 30 September 2026, offers eleven providers, including the zero-key Atlassian Forge LLM. Google Gemini, Ollama, any OpenAI-compatible server and Goose Swarm are new since 4.1.0, and CogniRunner Cloud AI was removed. 5.0.0 is a major version, so a site still on 4.1.0 gets them once its admin approves the update.
On your own key you pay the provider and pick the model. On Forge LLM there is no key and no egress, the tokens are billed to us as the vendor, and so the model is set by edition: Haiku on Standard, Haiku plus Claude Sonnet 5 and Opus 5 on Coder.
The site-wide key is stored in the app's ordinary Forge KVS storage and never sent back to the browser; it is not in Forge secret storage. Only the per-project override key uses secret storage.
Choosing Forge LLM does not make the app eligible for Runs on Atlassian. The egress is declared in the manifest whatever provider you pick.
Connecting Jira to OpenAI for a workflow validator sounds like one decision. In CogniRunner it is a choice between eleven providers in 5.0.0, up from eight in 4.1.0, and the one that matters most is not which model is smartest. It is who holds the key. If you paste your own OpenAI API key, you pay OpenAI, you pick the model, and the issue text you validate leaves Atlassian for OpenAI's servers. If you pick the zero-key Atlassian Forge LLM instead, nothing leaves the platform, we pay for the tokens, and the edition you are on decides which Claude model you get.
This is the admin's view of that choice, done as a procedure. By the end you will have a Jira transition guarded by an AI validator on the provider you picked, and a one-line REST check that proves it is judging. Here is what that check printed on our test site on 30 September 2026, against an issue whose whole summary is asdfghjkl:
text
1{"errorMessages":["AI Validation failed: The summary is gibberish and does not describe a software task."],"errors":{}}
2HTTP 400
Two versions matter here, and I want to keep them apart. The production build is 5.0.0, deployed on 30 September 2026. The one before it was 4.1.0, from 15 September. What I read and ran is our code at commit d44e683, from 29 September, on wolfaenpak, our own test Jira Cloud site, and 5.0.0 was built from a later commit on the same line, so the procedure and the provider mechanics below are what 5.0.0 carries. Because 5.0.0 is a major version, a site that installed 4.1.0 stays on it until an admin approves the update in Manage apps. If that is you, the box below is what you will not have until you do. Where I could not run something, I say so. If you have not decided between a validator and a condition yet, our tutorial on jira:workflowCondition vs workflowValidator covers that first.
Tip
Not in 4.1.0 (in 5.0.0) — Gemini, Ollama, the OpenAI-compatible server and Goose Swarm providers; the private-address rule for self-hosted URLs and the end of the Tailscale-only rule (4.1.0 accepts LM Studio only on a *.ts.net Funnel URL); the 10-minute verdict cache (on 4.1.0 every transition calls the model); the per-project key override; the API_KEY_SAVED audit event; the Opus-to-Sonnet sub-cap on Coder; and the </think> rule that blocks instead of failing open. 4.1.0 also still lists CogniRunner Cloud AI as a provider.
Note
Prerequisites
A Jira Cloud site where you are a Jira admin. The key and provider settings are admin-only: the resolver that saves a key refuses anyone else.
CogniRunner installed from the Atlassian Marketplace. It is free for up to 10 users; larger tiers are on the listing.
An API key for the provider you want (for OpenAI it must start with sk-), or nothing at all if you are going to use Forge LLM.
curl and a Jira API token, for the check at the end.
A throwaway project with two issues: one with a sensible summary and one with junk in it.
The providers behind one Jira OpenAI validator
The provider list lives in one place in the app, a map called PROVIDERS in src/index.js. In 4.1.0 it has eight entries: OpenAI, Azure OpenAI, OpenRouter, Anthropic, LM Studio, AWS Bedrock, Forge LLM, and CogniRunner Cloud AI, a managed option routed through OpenRouter. Between 20 and 21 September we added Google Gemini, Ollama, a generic OpenAI-compatible server and Goose Swarm, and removed CogniRunner Cloud AI, so 5.0.0 has eleven. All four shipped in 5.0.0. The table is 5.0.0; the rows marked after 4.1.0 are the ones a site that has not approved the update will not see.
Provider
Key
Where the request goes
OpenAI
yes, starts with sk-
https://api.openai.com/v1
Azure OpenAI
yes
your own Azure endpoint (you supply the base URL)
OpenRouter
yes
openrouter.ai
Anthropic
yes
https://api.anthropic.com
Google Gemini (after 4.1.0)
yes
generativelanguage.googleapis.com
AWS Bedrock
yes, a Bedrock API key
https://bedrock-runtime.<region>.amazonaws.com
LM Studio
depends on your server
your own https host
OpenAI-compatible server (after 4.1.0)
depends on your server
your own https host (vLLM, llama.cpp, LocalAI, Jan and similar)
Ollama (after 4.1.0)
yes for Ollama Cloud
ollama.com, or your own host
Goose Swarm (after 4.1.0)
yes, a token
your own host
Atlassian (Forge LLM)
none
stays inside the Atlassian platform
12 rows × 3 columnsHeader row enabled
Two of those rows carry a caveat that is written in the code itself, and you should know them before you pick. Azure OpenAI has a comment on it saying it is mostly untested end to end and that its runtime behaviour should be treated as unverified. And Bedrock is authenticated with a Bedrock API key sent as a plain bearer token, with no AWS SigV4 signing, which means an IAM access key pair is not what it wants. The code also prefers regional inference profile ids such as eu. and us. for Bedrock models, because bare model ids return 403 for many models.
If you pick a provider and never choose a model, these are the defaults the code falls back to (src/shared/model-resolution.js): gpt-5.4-mini for OpenAI and Azure, openai/gpt-5.4-mini on OpenRouter, Claude Haiku 4.5 for Anthropic and for Forge LLM, eu.anthropic.claude-sonnet-4-6 on Bedrock, and gemini-3.8-flash on Gemini. LM Studio has no default, because only you know what is loaded on your server.
Step by step: set up the provider and the validator
1
Open the CogniRunner Settings page
CogniRunner adds a Jira admin page titled "CogniRunner Settings". Everything in steps 2 to 5 happens there.
2
Pick the provider
choose one from the provider picker. For Azure, LM Studio, an OpenAI-compatible server or a self-hosted Ollama, enter the base URL too. For Bedrock, pick one of the 13 regions the picker lists.
3
Paste the API key
skip this for Forge LLM, which has no key. For anything else, paste the key and save. Keys shorter than 8 characters are rejected, and an OpenAI key that does not start with sk- is rejected with "OpenAI API keys must start with sk-".
4
Choose the model
or leave it on the provider's default listed above. On Forge LLM the list is limited by your edition.
5
Press Test connection
this sends a live 1-token call to the active provider and shows a verdict chip. You should see Connected before you go any further.
6
Add the CogniRunner Field Validator to a transition
edit the workflow, open the transition you want to guard, and add the validator with a plain-language prompt, for example that the summary must describe a real software task.
7
Verify it with two transitions
move the junk issue and the good issue through that transition with the REST call below and confirm you get a 400 and a 204.
The rest of this article walks through what each of those steps does behind the button, because that is where the provider choice actually shows up.
Add your OpenAI API key to the validator
When you save a key, the admin page calls a resolver named saveOpenAIKey (it keeps that name for every provider, a leftover from when OpenAI was the only one). I read what it does in src/index.js around line 7368:
It checks you are an admin and refuses otherwise.
It rejects a key shorter than 8 characters with "Invalid API key format".
If the provider is Forge LLM, it refuses to store anything: "Atlassian Forge LLM does not use an API key — inference runs on the Atlassian platform."
For OpenAI, it insists on the sk- prefix.
It writes the key to the app's storage under a slot named COGNIRUNNER_KEY_<provider>, so each provider keeps its own key and switching back and forth does not make you paste them again.
It writes an audit event, API_KEY_SAVED, with a flag saying whether an earlier key was replaced.
Here is the part I want to be plain about, because it is the question a security reviewer will ask first. The key is stored with an ordinary Forge KVS set, the same app storage the rest of CogniRunner's settings live in. The site-wide key is not stored with Forge's secret storage calls. The per-project key override described below is different: it does use Forge secret storage (storage.setSecret). What the code does guarantee is the other direction: the resolver that reports key status says in its own comment that it never returns the actual key to the frontend, so once saved, the browser never sees it again. I have not checked what Atlassian documents about at-rest encryption of KVS, so I am not going to claim anything about it here.
There is also a per-project override, and its key is the one kept in secret storage. The code has resolvers to set and clear an AI key for a single project, so one team can run on its own provider account while the rest of the site uses the default. I did not exercise that path for this article.
How to know the key works
The Test connection button is the check for this step. It maps the provider's answer to one of these verdicts, straight from OpenAIConfig.jsx:
text
1Connected The active provider answered a live test call.
2Auth failed The API key was rejected, check the key below. (401 / 403)
3Model / endpoint not found The base URL or model may be wrong for this provider. (404)
4Rate-limited The provider is throttling, temporary; validators fail OPEN meanwhile. (429)
5Provider error The provider returned a server error, usually temporary. (5xx)
6Unreachable Couldn't reach the host, check the base URL, egress, and that the service is up.
Anything else shows as "Error (HTTP n)" with the provider's own message. Only a successful call produces Connected, so you should see that word and nothing else before you move on. Read the Rate-limited line twice. A validator whose provider is throttled or erroring lets the transition through. That is a design choice (a flaky AI vendor should not freeze your Jira), but it means an AI validator is a quality gate, not a security control. 5.0.0 has one narrow exception: when a reasoning model's reply ends with a bare </think> tag and what remains holds no single verdict, the validator blocks. A reply cut off at the output limit, or an empty one, still lets the transition through.
A self-hosted server has one more rule. In 4.1.0, LM Studio already had to be https, not localhost, and on a *.ts.net Tailscale Funnel host. 5.0.0 drops the ts.net requirement and extends the rule to all four self-hosted providers: for LM Studio, an OpenAI-compatible server, Goose Swarm and a self-hosted Ollama (an Ollama Cloud URL is left alone), the provider save refuses anything that is not https and anything pointing at localhost or a private address: 127.x, 10.x, 172.16-31.x, 192.168.x, 169.254.x, 0.0.0.0, and IPv6 loopback, private and link-local. Forge runs in Atlassian's cloud, so a URL that only resolves on your laptop could never work anyway. You need a public https address for the machine, and on 4.1.0 that has to be a Tailscale Funnel URL; on 5.0.0 any public https host works.
Forge LLMs: the zero-key default and how the edition decides the model
Forge LLM is the provider with nothing to paste. The comment above its entry in the code says it in one line: Claude models served inside the Atlassian platform through @forge/llm, "No API key, no egress, no BYOK", with the token costs "billed to the app vendor's Forge bill". The vendor is us. It is also text-only for now, so a validator that needs to read an image attachment needs a different provider.
Because we pay, the model is a pricing boundary, and the rule for it lives in one file, src/shared/edition.js:
On the Standard edition, Forge LLM runs Claude Haiku 4.5 only.
On the Coder edition, it can run Haiku, Claude Sonnet 5 or Claude Opus 5.
If the app cannot read a licence, or the licence is inactive, it treats you as Standard.
What happens at the edge of that rule is more interesting than the rule itself. A Standard admin can browse the model list and will see Sonnet 5 and Opus 5 there, marked locked. Trying to save one returns an upgrade prompt, not an error. And at run time, a model the edition does not allow is quietly replaced by Haiku. The code calls this the billing backstop, and its commit message is specific: if anything goes wrong while resolving the edition or the allowance, it "degrades the model, it never breaks the transition". There is also a monthly allowance. On Coder, once its Opus share is used up, Opus requests run on Sonnet 5 instead. When the allowance is hard-capped, rule inference drops to Haiku whatever the edition. I did not find the size of that allowance for this article.
This used to be simpler. Until 12 September a function called isForgeLlmModelAllowed did nothing but test whether the model id contained "haiku". It was deleted when the Coder edition shipped, and if you read our older CogniRunner material, or even our own product page, you will still see the Haiku-only description. The code is the current answer.
One more difference matters if you use CogniRunner's agent features rather than plain validators. On your own key, agents are enabled whatever the model. On Forge LLM, a Standard edition is refused with a need-Coder reason, and Haiku on Forge LLM is refused for agent work too. The code's own comment notes that on a bring-your-own key, Haiku drives agents perfectly well; the refusal is about what we pay for, not about what Haiku can do.
What bring your own key changes: egress, model, cost, Runs on Atlassian
Put the two paths side by side and the admin's trade-off is four questions.
Where the data goes. With Forge LLM, the issue text is judged inside the Atlassian platform. With any other provider, the fields the validator reads are sent to that provider over the internet. Our product page has the right way to say it: if you configure no cloud provider, only Forge LLM or your own server, no issue content reaches a third-party AI vendor. It is provider-dependent, not absolute.
Who picks the model. On your own key, you do, from anything your provider offers. On Forge LLM, your edition does.
Who pays. On your own key, your provider bills you per token, at your provider's prices, on your own account. On Forge LLM we carry the token bill, which is exactly why the model is clamped.
Runs on Atlassian. This is the one people get wrong, and I would have got it wrong too without running the check. You might expect that choosing Forge LLM turns CogniRunner into a Runs on Atlassian app. It does not, because eligibility is about what the app's manifest declares, not about which provider you picked in its settings. CogniRunner's manifest declares egress to every provider host in the table above, and in 5.0.0 the backend egress for self-hosted servers is a wildcard *, because a self-hosted server lives on a host only you know. 4.1.0 lists explicit hosts instead, but it declares egress all the same. The manifest comment says it outright: "The app is already outside the Runs-on-Atlassian program." Here is the check, run against our development environment:
bash
1forge eligibility --non-interactive -e development
text
1The version of your app [29.12.0] that's deployed to [development] is not eligible for the Runs on Atlassian program.
2- App is using remote services
3- App is egressing data
4- App is using a webtrigger module that can egress data
So if your organisation requires the Runs on Atlassian badge, the provider setting will not get you there. If what you need is the property the badge stands for, that issue content never leaves Atlassian, Forge LLM gives you that for the validator calls, and you should confirm it against your own policy rather than against the badge. I ran this on development, which carries the same code as 5.0.0, not on the production environment customers install.
Verify the validator with one REST call
The admin page proves the provider answers. It does not prove the validator is attached to the right transition or that the prompt does what you meant. For that, move a real issue through the guarded transition. I did it with curl so the result is a status code, not a dialog I have to describe. Find your transition id first with a GET on /rest/api/3/issue/<key>/transitions; on our test project the guarded transition is 9001.
The junk issue, COGTEST-37, whose summary is asdfghjkl:
bash
1curl-s-w'\nHTTP %{http_code}\n'-u"$JIRA_ADMIN_EMAIL:$JIRA_API_TOKEN"-H'Content-Type: application/json'-X POST "$JIRA_BASE_URL/rest/api/3/issue/COGTEST-37/transitions"-d'{"transition":{"id":"9001"}}'
text
1{"errorMessages":["AI Validation failed: The summary is gibberish and does not describe a software task."],"errors":{}}
2HTTP 400
The good issue, COGTEST-31, "Implement password reset via a one-time email link", through the same transition:
bash
1curl-s-w'\nHTTP %{http_code}\n'-u"$JIRA_ADMIN_EMAIL:$JIRA_API_TOKEN"-H'Content-Type: application/json'-X POST "$JIRA_BASE_URL/rest/api/3/issue/COGTEST-31/transitions"-d'{"transition":{"id":"9001"}}'
text
1HTTP 204
You should see exactly that pair: a 400 with the model's reason after "AI Validation failed:", and a 204 with an empty body. A 204 on both means the validator is not attached, your prompt is too loose, or the provider failed and the validator failed open; the logs below will tell you which. A 400 on both means the prompt is too strict, or the validator is failing closed on a reply it could not parse.
Confirm which provider actually answered
Our test site was not on OpenAI when I ran this, and I only found out from the logs. If you have the app's Forge logs (you will, if it is your own deployment; a Marketplace customer will not), this shows the model that judged each call:
bash
1forge logs -e development --since 5m |grep-iE"openrouter|modelUsed|COGTEST|verdict"
The model id is an OpenRouter one, and a few minutes earlier the same site had logged openrouter openai/gpt-6-luna on the first run. So the provider that answered was OpenRouter, not OpenAI directly and not Forge LLM. That is the whole point of checking: the settings page tells you what someone meant to configure, and the log tells you which model produced the verdict, fresh or from the cache.
The other line worth reading is cachedVerdict: true. My curl calls came a couple of minutes after an earlier run on the same two issues, and CogniRunner answered both from its verdict cache instead of asking the model again. The cache lasts 10 minutes. Its key is a hash over the provider, the model, the prompt, the field value and the documents and memory the rule reads, so changing the provider or the model can never serve you a verdict the old one gave. The "(cached verdict)" marker only goes into the execution log and debug trace; the person moving the issue sees the plain reason, which is why the curl output above does not mention it. In practice: changing the provider, the model or the prompt always gets a fresh call, while re-running the identical transition on an unchanged issue within 10 minutes does not.
The cheapest validator does not call a model at all
Before you pick a provider, check whether the rule needs AI at all. CogniRunner ships deterministic, zero-AI checks for the common ones: field required, a regex match, allowed values, text length, a relative date, issue type, all sub-tasks resolved, linked issue resolved, a user in a field, and git checks such as a merged or approved pull request and a passing build. I counted 32 case labels in src/premade-rules.js; some of those are fall-throughs, so I would not call it 32 distinct rules. They cost nothing per transition, they never fail open because a vendor is down, and no text leaves Jira. The AI validator earns its place on the rules a regex cannot express, like "does this summary describe a real task". If a rule needs to see a specification before it judges, attaching documents to a CogniRunner validator covers that.
What I could not test
I did not switch provider or paste a key live for this article. The provider switch is admin-UI only, with no REST path, and I did not drive the page with a browser. So the only provider I saw judging real issues is OpenRouter. The OpenAI-direct, Anthropic, Gemini, Bedrock and Forge LLM paths are described from the code, not from a run. Azure OpenAI is marked as mostly untested by its own authors. I did not find the Forge LLM monthly allowance or the Coder edition's price, and I ran the eligibility check on our development environment only.
CogniRunner is a commercial app now, no longer open source, so the file and line references here are for our own record of where each fact came from rather than something you can clone.
Recap
Key takeaways
You now have a transition guarded by a CogniRunner AI validator on the provider you chose, and a curl pair that proves it: 400 with a reason for junk, 204 for a real task.
Eight providers sit behind the same validator in 4.1.0, eleven in 5.0.0. Your own key means you pay, you pick the model, and the issue text goes to that provider.
Forge LLM means no key and no egress for the call, the vendor pays, and the edition picks the model: Haiku on Standard, Haiku plus Sonnet 5 and Opus 5 on Coder. A disallowed model becomes Haiku, never an error.
The site-wide key sits in ordinary Forge KVS app storage and never goes back to the browser; only the per-project override key uses Forge secret storage.
Picking Forge LLM does not earn the Runs on Atlassian badge; the manifest's declared egress decides that.
Check the Forge logs, not the settings page, if you need to know which model actually judged a transition.
Not covered here: the per-project key override, agent rules, and the providers I could not run live.
JCMA "We couldn't export Custom Field Config Scheme": three causes, three fixes
