LLM regression testing for a Jira workflow validator: CogniRunner before you switch models
Mihai Perdum
Author
17 min readOctober 14, 2026
Key takeaways
END STATE: a Jira filter of 8 labelled requests and an 80-line Node script that dry-runs your saved AI validator on each one, on the site's model or on a candidate, and exits 1 when a verdict differs from its label.
On 13 October 2026 the site's model (openai/gpt-6-luna) matched 6, 7 and 7 of 8 over three rounds. The candidate (anthropic/claude-haiku-5.5) matched 8 of 8 through a per-request override, with the site left as it was.
The stable miss was a free flu vaccination for two offices, blocked 3 times out of 3. The rule said health information about anyone and didn't say whether a programme for 150 people counts.
One sentence added to the rule gave 8 of 8 on both models: four rounds on the site's model and two on the candidate.
A token gets 20 AI actions per 10-minute clock window and 200 a day, so one round of 8 on each model is 16 and a third round is over the limit.
The audit log held 24 rows for the 24 tests on the candidate model and none for the 84 dry runs that named no other model.
LLM regression testing means running the same labelled inputs again after you change a prompt or a model, to see what the change broke. This tutorial does that for an AI check on a Jira workflow validator. Here the test set is a Jira filter of 8 labelled requests. An 80-line Node script runs it. Each call goes to CogniRunner's rule test over REST, which reads the issue, asks the model and writes nothing to Jira.
I ran it on 13 October 2026 because I wanted to move a validator from openai/gpt-6-luna to anthropic/claude-haiku-5.5. Over three rounds, the site's model matched 6, 7 and 7 of the 8 labels. The candidate matched 8 of 8. I built the test expecting the candidate to miss, and it was the site's model that missed (I'd have bet the other way).
I found the cause in the rule, and I don't think it says much about either model. Our HR check says to fail a request that holds "health or medical information about anyone". Is a free flu vaccination for two offices health information about anyone? Our rule didn't say. So the two models read it differently, and one added sentence gave 8 of 8 on both.
My runs of 13 October 2026 in one grid. The two red cells are the site's model on the rule as saved: parental leave matched in 2 of 3 runs, the flu vaccination in 0 of 3. With the sentence, 4 of 4 and 2 of 2.
We recorded a three-minute video of this on our demo site, on CogniRunner 1.34.0. What follows is my own run of the same procedure on our test site, with the output as it printed. I did three of the steps through an API where you will click: my tokens came from our test hook, I saved the sentence through Jira's workflow REST API, and I switched the model through the test hook. The video shows those screens, and I give the timestamp at each step.
Note
Prerequisites
A Jira Cloud site with CogniRunner 1.34.0 or later. The Marketplace calls that build 7.16.0, released 12 October 2026. I ran a development build of 1.34.0 on our test site.
An AI provider saved in CogniRunner's Settings. Mine was OpenRouter with openai/gpt-6-luna as the model for validators and rules. If you haven't set one up, the provider tutorial covers it.
A company-managed project whose workflow has a CogniRunner AI validator on one transition. Mine was called Send for Review.
An email and API token for Jira's REST API. The script reads the published workflow. I ran it as a Jira admin.
A CogniRunner admin, to create the REST API token in step 2.
Node 18 or later. I used Node v24.15.0. The script has no dependencies.
The video runs on our demo site with issue keys PPL-13 to PPL-27, and my run has COGTEST-3049 to COGTEST-3056 with the same eight texts. Its script takes flags and runs four requests at a time, while the one printed here takes environment variables and runs one at a time. In the video the first round is 7 of 8. Mine was 6 of 8.
1
Build the golden dataset
eight requests in a Jira filter, each labelled expect-pass or expect-block.
2
Create a token
Settings, the card titled Tokens for CogniRunner's REST API.
3
Save the script
it reads the saved rule from Jira and calls the rule test for each request.
4
Run the baseline
the site's own model, three rounds.
5
Run the candidate
one variable, and the site's model stays as it is.
6
Read the mismatch
decide whether the model or the rule is wrong.
7
Try the sentence, save it, run both again
first unsaved, then saved.
8
Switch the site's model
and find the switch and the tests in the audit log.
What one rule test does: one field, one prompt, nothing written
A CogniRunner AI validator sits on a workflow transition. Somebody moves the request. The app sends one field of it (here the Description) to the model together with the prompt you saved. Back comes pass or fail, with a reason. Atlassian's Forge validator module is what carries it: "Validators check that any input made to the transition is valid before the transition is performed."
You may know the Test button in the rule editor. CogniRunner's Rules REST API has the same thing as a call: POST ?resource=rules&action=test. Our docs describe it like this: "the issue is read, the AI answers, and nothing is written." The body carries the issue key, the field and the prompt. Because the prompt travels in the body, you can try a wording before you save it.
An admin token can add ai: {provider, model} to that body, and then this one request runs on another model of a provider the site already has a key for. From the same page: "Nothing is saved or cached: the site's provider, models and keys stay as they were, and the next request without ai runs on the site's provider."
Step 1. Build the golden dataset: a Jira filter of 8 labelled issues
People who test LLM apps call a fixed set of inputs with known right answers a golden dataset. PromptLayer's glossary defines the practice as "re-running a golden dataset whenever you change a prompt, model, or workflow to catch quality regressions early." For a validator the right answer is one of two words, so the dataset can live in Jira as plain issues.
Create eight requests in the project the validator guards. Give each the label gate-test, and then expect-pass or expect-block. Mine were four of each:
Key
Label
Request
What decides it
COGTEST-3049
expect-pass
Onboarding: new hire Freya Nilsson, Security Engineer
band, salary, laptop, nothing about health
COGTEST-3050
expect-block
Salary adjustment: Nora Achterberg, Finance Ops
"she was in hospital for a week in August"
COGTEST-3051
expect-block
Relocation support: Arjun Mehta, Bangalore to Lisbon
cleared to fly "after the knee operation"
COGTEST-3052
expect-pass
Parental leave cover: Lina Haddad, Platform support
parental leave, which isn't an illness
COGTEST-3053
expect-pass
Wellbeing budget: Design team, 2027
standing desks and a gym benefit for 12 people
COGTEST-3054
expect-block
Relocation allowance: Tomasz Wierzbicki, Kraków to Lisbon
"his daughter's asthma"
COGTEST-3055
expect-block
Reduced hours: Amara Okafor, 32 to 28 hours
"a phased return after her burnout absence"
COGTEST-3056
expect-pass
Flu vaccination: office programme, November 2026
free and optional, for everyone in two offices
9 rows × 4 columnsHeader row enabled
All of these people are made up. They belong to the fictional company on our demo site, and the eight are ours. Write yours from requests your own people sent, with the names changed.
Then save the search as a filter. Mine is called Gate tests: HR health check, with this JQL:
text
1project = COGTEST AND labels = gate-test ORDER BY key ASC
Note the filter's id from its URL. Mine was 12450.
The test set as a Jira filter on our test site: 8 of 8 rows, four expect-pass and four expect-block. The +1 beside each label is gate-test.
Check: the filter should show 8 of 8, and each row should show its expect label. If a row shows another label first, remove the extra labels. Jira sorts labels by name, and in our video session a label called benefits pushed two verdicts behind a +2.
Is eight enough? Probably not for a rule that matters a lot. Traceloop's guide suggests "20-30 'golden set' examples that cover common use cases and known failure modes." Our eight came out of 28 candidates. Our video session ran those 28 on both models, 600 dry runs in all. It kept the ones whose verdict stayed the same over four or five rounds, plus one request where the two models disagreed each time. Four of the dropped ones flipped in 1 round of 3 (a newborn added to a health plan, a return from maternity leave, and two more). A request that flips by itself will fail your test on days when nothing changed, so I'd keep those out of the gate and in a separate list to watch.
Step 2. Create a token for CogniRunner's REST API
Open CogniRunner's admin page, go to the Settings tab and find the card titled Tokens for CogniRunner's REST API. It opens on Viewer and Their own rules. Change those to Admin and Everyone's rules. My tokens had both, and I haven't tried a narrower one. Type a name and click Create token. You see the value once, so copy it now, and copy the Endpoint from the same card. The video shows this card at 0:34, with its two tokens already made.
The token card on a development build of 1.34.0, on our test site. The endpoint address is painted over. Admin is the role the candidate run in step 5 needs.
Our docs put the rule test at editor, and the same call with ai at "admin token + live admin minter". By minter they mean the person who created the token, who must still be an admin when the call arrives. So an editor token should be enough for step 4, going by the docs and the code.
Tokens run out. On the card it says "New tokens expire after 90 days", and mine, created on 13 October, showed an expiry of 11 January 2027. Also treat the endpoint address as private too. It's a Forge web trigger, and Atlassian's page on those says "You should keep these URLs secure." CogniRunner asks for the bearer token on top. I still mask the address everywhere, this page included.
Put the values in your shell:
bash
1exportJIRA_SITE=https://wolfaenpak.atlassian.net JIRA_EMAIL=<admin email>JIRA_API_TOKEN=<jira api token>2exportRULES_API_URL=<rules-api>RULES_API_TOKEN=cgr_<masked>3exportWORKFLOW="Software Simplified Workflow for Project COGTEST"TRANSITION="Send for Review"FILTER=12450
That first value is our test site, so put your own there. WORKFLOW is the workflow's name as Jira's workflow list shows it. You'll know whether the token works from the first line the script prints in step 4, which should show its role.
Step 3. Call the Rules REST API to test the Jira workflow validator on each issue
Save this as gate-test.mjs. It needs no packages. savedRule reads the published workflow from Jira and pulls the validator's prompt and field out of the transition. It reads Jira because the rules resource of CogniRunner's REST API has no call that returns a tracked rule's saved prompt in full. testSet reads the filter and takes the expected verdict from the label. runOne posts one request to the rule test and waits if the token's allowance is spent.
javascript
1#!/usr/bin/env node2// gate-test.mjs: run a CogniRunner AI validator's SAVED prompt over a labelled Jira test set. Node 18+, no dependencies.3// Every call is a dry run (POST ?resource=rules&action=test): the issue is read, the AI answers, nothing is written.4// JIRA_SITE=https://you.atlassian.net JIRA_EMAIL=... JIRA_API_TOKEN=... \5// RULES_API_URL=<the Rules REST API address from CogniRunner Settings> RULES_API_TOKEN=cgr_... \6// WORKFLOW="<workflow name>" TRANSITION="Send for Review" FILTER=<filter id> node gate-test.mjs7// Optional: CANDIDATE_MODEL=<provider>/<model id> runs each request on that model instead (needs an ADMIN token; the8// site's own model is not changed). CANDIDATE_PROVIDER defaults to openrouter. ADD_SENTENCE="..." appends one9// sentence to the saved prompt for this run only, so you can try a wording before you save it.10constE= process.env;11for(const k of["JIRA_SITE","JIRA_EMAIL","JIRA_API_TOKEN","RULES_API_URL","RULES_API_TOKEN","WORKFLOW","FILTER"])if(!E[k]){console.error(`Set ${k}`); process.exit(2);}12constTRANSITION=E.TRANSITION||"Send for Review";13constJAUTH="Basic "+Buffer.from(`${E.JIRA_EMAIL}:${E.JIRA_API_TOKEN}`).toString("base64");14constsleep=(ms)=>newPromise((r)=>setTimeout(r, ms));15constpad=(s, n)=>String(s ??"").padEnd(n).slice(0, n);1617asyncfunctionjira(path, body){18const r =awaitfetch(E.JIRA_SITE+ path,{method:"POST",headers:{Authorization:JAUTH,Accept:"application/json","Content-Type":"application/json"},body:JSON.stringify(body)});19if(!r.ok)thrownewError(`Jira ${path}: ${r.status}${(await r.text()).slice(0,200)}`);20return r.json();21}22asyncfunctionrules(method, query, body){23const r =awaitfetch(`${E.RULES_API_URL}?${query}`,{ method,headers:{Authorization:`Bearer ${E.RULES_API_TOKEN}`,"Content-Type":"application/json"},body: body ?JSON.stringify(body):undefined});24const text =await r.text();let json =null;try{ json =JSON.parse(text);}catch{/* not JSON */}25return{status: r.status, json,retryAfter:Number(r.headers.get("retry-after")||0)};26}2728// 1. The rule as SAVED: the CogniRunner validator on the transition, read from the published workflow.29asyncfunctionsavedRule(){30const j =awaitjira("/rest/api/3/workflows",{workflowNames:[E.WORKFLOW]});31for(const w of j.workflows||[])for(const t of w.transitions||[]){32if(t.name!==TRANSITION)continue;33for(const v of t.validators||[]){34if(!/ai-text-field-validator/.test(v.parameters?.key ||""))continue;35const c =JSON.parse(v.parameters.config);36return{prompt: c.prompt,fieldId: c.fieldId||"description",enableTools: c.enableTools===true,version: w.version?.versionNumber };37}38}39thrownewError(`No CogniRunner AI validator on "${TRANSITION}" in workflow "${E.WORKFLOW}"`);40}41// 2. The test set: every issue in the filter; the expected verdict is its label.42asyncfunctiontestSet(){43const j =awaitjira("/rest/api/3/search/jql",{jql:`filter = ${E.FILTER} ORDER BY key ASC`,fields:["summary","labels"],maxResults:100});44return(j.issues||[]).map((i)=>({key: i.key,expected: i.fields.labels.includes("expect-block")?"BLOCK": i.fields.labels.includes("expect-pass")?"PASS":"?"}))45.sort((a, b)=>Number(a.key.split("-")[1])-Number(b.key.split("-")[1]));46}47// 3. One dry run through the test door. A 429 means the token's AI allowance is spent: wait for Retry-After.48asyncfunctionrunOne(rule, prompt, issue){49const body ={kind:"validation",issueKey: issue.key, prompt,fieldId: rule.fieldId,enableTools: rule.enableTools};50if(E.CANDIDATE_MODEL) body.ai={provider:E.CANDIDATE_PROVIDER||"openrouter",model:E.CANDIDATE_MODEL};51for(let attempt =0; attempt <4; attempt++){52const t0 =Date.now();const r =awaitrules("POST","resource=rules&action=test", body);53if(r.status===429){const s = r.retryAfter||60;console.error(` 429 on ${issue.key}: allowance spent, waiting ${s} s`);awaitsleep(s *1000);continue;}54if(r.status===202)return{...issue,verdict:"QUEUED",seconds:0,servedBy:"",reason:`self-hosted provider: poll ${r.json.poll}`};55const res = r.json?.result ||{};56const verdict = r.status!==200?`HTTP ${r.status}`: res.success&& res.isValid===true?"PASS": res.success&& res.isValid===false?"BLOCK":"NOT JUDGED";57return{...issue, verdict,seconds:Math.round((Date.now()- t0)/100)/10,servedBy: r.json?.ai?.servedBy?.map((s)=> s.model).join(",")|| res.modelUsed||"",58reason: res.reason|| res.error|| r.json?.error ||""};59}60return{...issue,verdict:"GAVE UP",seconds:0,servedBy:"",reason:"429 four times"};61}6263const who =awaitrules("GET","resource=whoami");64if(who.status!==200){console.error(`whoami: HTTP ${who.status}${who.json?.error ||""}`); process.exit(1);}65const spend = who.json.aiSpend||{};66console.log(`Token role: ${who.json.effective?.role} · AI actions used ${spend.window?.used}/${spend.window?.limit} in this 10-minute window, ${spend.today?.actions}/${spend.today?.limit} today`);67const rule =awaitsavedRule();68const prompt =E.ADD_SENTENCE?`${rule.prompt}${E.ADD_SENTENCE.trim()}`: rule.prompt;69const set =awaittestSet();70console.log(`Rule: ${E.WORKFLOW} · ${TRANSITION} · reads ${rule.fieldId} · workflow version ${rule.version}`);71console.log(`Prompt (${E.ADD_SENTENCE?"saved + one sentence, this run only":"as saved"}): ${prompt}`);72console.log(`Model: ${E.CANDIDATE_MODEL?`${E.CANDIDATE_MODEL} (these requests only; the site's model is not changed)`:"the site's own model"}`);73console.log(`\n${pad("KEY",13)}${pad("EXPECTED",8)}${pad("VERDICT",10)}${pad("SECS",5)}${pad("SERVED BY",28)} REASON`);74let match =0, slowest =0;75for(const issue of set){// one at a time: calls in parallel made the model slower and one timed out76const r =awaitrunOne(rule, prompt, issue);const ok = r.verdict=== r.expected;if(ok) match++; slowest =Math.max(slowest, r.seconds);77console.log(`${pad(r.key,13)}${pad(r.expected,8)}${pad(r.verdict+(ok ?"":" ✗"),10)}${pad(r.seconds,5)}${pad(r.servedBy,28)}${r.verdict==="PASS"?"": r.reason}`);78}79console.log(`\n${match} of ${set.length} match · slowest ${slowest} s`);80process.exit(match === set.length?0:1);
Why one request at a time? In the video session, four calls in flight pushed the site's model to 16.1 seconds. With eight in flight, one call ran past 20 seconds and came back as a timeout. My first run's eight calls added up to about 26 seconds, which I can live with.
It exits 0 when all verdicts match and 1 when one doesn't, so the same file works as a step in CI.
Check: open a new terminal, where the variables from step 2 aren't set, and run node gate-test.mjs there. It should print Set JIRA_SITE and stop, which tells you Node found the file and nothing was called.
The rule under test in the Rules tab, development build of 1.34.0 on our test site: a validator on Send for Review, reading Description. The list cuts the prompt short. The script prints all of it.
Step 4. Run the LLM regression testing baseline on the site's model
With the variables from step 2 exported:
bash
1node gate-test.mjs
text
1Token role: admin · AI actions used 0/20 in this 10-minute window, 0/200 today
2Rule: Software Simplified Workflow for Project COGTEST · Send for Review · reads description · workflow version 1079
3Prompt (as saved): Fail it if it contains health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment. This is an HR request a manager sends to People Ops for approval. When it fails, say in one short sentence which detail must be removed. Otherwise it passes.
4Model: the site's own model
56KEY EXPECTED VERDICT SECS SERVED BY REASON
7COGTEST-3049 PASS PASS 3.5 openai/gpt-6-luna
8COGTEST-3050 BLOCK BLOCK 2.8 openai/gpt-6-luna Remove the detail that Nora was in hospital for a week in August.
9COGTEST-3051 BLOCK BLOCK 2.5 openai/gpt-6-luna Remove the detail that his physiotherapist cleared him to fly after knee surgery.
10COGTEST-3052 PASS BLOCK ✗ 2.6 openai/gpt-6-luna Remove the detail that Lina Haddad is on parental leave.
11COGTEST-3053 PASS PASS 2.9 openai/gpt-6-luna
12COGTEST-3054 BLOCK BLOCK 2.9 openai/gpt-6-luna Remove the detail that his daughter’s asthma is worse in Kraków’s winters.
13COGTEST-3055 BLOCK BLOCK 3.7 openai/gpt-6-luna Remove the detail about her burnout absence and phased return.
14COGTEST-3056 PASS BLOCK ✗ 4.7 openai/gpt-6-luna Remove the detail about offering flu vaccinations through the occupational health provider.
15166 of 8 match · slowest 4.7 s
Check: the first line should show Token role: admin (or editor), the Rule line should show your transition and reads description, and the Prompt line should be your saved prompt word for word. Under SERVED BY you see the model that answered.
So that's two misses. I ran the same command twice more and got 7 of 8 both times. All three rounds blocked the flu vaccination, each time with a different sentence ("Remove the detail about free flu vaccination, which is medical information." was round two). Parental leave was blocked in the first round and passed in the next two.
Illustration, generated locally with FLUX. The site's model matched 6, 7 and 7 of 8 labels over three rounds on the rule as saved. The class took round one badly.
If I'd stopped after one run I would have gone off to fix parental leave, which was right two times out of three. So run the baseline at least three times and sort the misses into stable ones and flaky ones before you touch anything. Here the stable one is COGTEST-3056.
Watch the allowance while you do. A token gets 20 AI actions per 10 minutes, and three rounds of eight are 24. I used a new token for each pair of rounds. With one token, the script is written to wait out the Retry-After the API sends (I never let it get that far, so I haven't watched it do so).
Step 5. Switch from OpenAI's model to Claude for one request, not the whole site
1Token role: admin · AI actions used 8/20 in this 10-minute window, 8/200 today
2Rule: Software Simplified Workflow for Project COGTEST · Send for Review · reads description · workflow version 1079
3Prompt (as saved): Fail it if it contains health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment. This is an HR request a manager sends to People Ops for approval. When it fails, say in one short sentence which detail must be removed. Otherwise it passes.
4Model: anthropic/claude-haiku-5.5 (these requests only; the site's model is not changed)
56KEY EXPECTED VERDICT SECS SERVED BY REASON
7COGTEST-3049 PASS PASS 3.2 anthropic/claude-haiku-5.5
8COGTEST-3050 BLOCK BLOCK 3 anthropic/claude-haiku-5.5 Remove the sentence stating that Nora was in hospital for a week in August, as it discloses medical information.
9COGTEST-3051 BLOCK BLOCK 3.8 anthropic/claude-haiku-5.5 Remove the reference to the knee operation and physiotherapist clearance, which is medical information about an employee.
10COGTEST-3052 PASS PASS 4.5 anthropic/claude-haiku-5.5
11COGTEST-3053 PASS PASS 3 anthropic/claude-haiku-5.5
12COGTEST-3054 BLOCK BLOCK 4.5 anthropic/claude-haiku-5.5 Remove the reference to the daughter's asthma, since it is a medical condition about a family member.
13COGTEST-3055 BLOCK BLOCK 3.1 anthropic/claude-haiku-5.5 Remove the reference to her burnout absence and occupational health involvement, as these disclose medical or health information about an employee.
14COGTEST-3056 PASS PASS 7.4 anthropic/claude-haiku-5.5
15168 of 8 match · slowest 7.4 s
Check: the Model line should show "these requests only", and SERVED BY should show the candidate on all eight rows. Then open CogniRunner's Settings. You should see the model for validators and rules still set to the one you started with.
Both models here are reached through OpenRouter. For eight requests the model id changed, and the site's provider didn't change at all. To try a model through another provider, say Anthropic directly, that provider needs a key saved in Settings first. A request can't bring one. I tried sending an apiKey in the body and got a 400 that says so (it's in the troubleshooting list).
I picked the candidate id by reading OpenRouter's models API that morning. anthropic/claude-haiku-5.5 was the newest Claude Haiku id on it, listed as created on 7 October 2026. I compared the two models as models a few days ago, and this page isn't a second round of that. Eight requests on one prompt is far too little to rank anybody, and I ran the candidate on the unfixed rule once.
Step 6. Read the mismatch: the rule did not say what HR meant
Here is the Description of the request the site's model kept blocking:
Change: free flu vaccination for everyone in the Lisbon and Berlin offices, run by our occupational health provider on 18 and 19 November; budget 1,950 EUR for up to 150 people. Effective date: 18 November 2026 Approver: Ines Kowalczyk (Head of People) Reason: it is optional and no one's answers or records come to People Ops; last year 96 people took part.
And the rule says to fail a request that holds "health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment". A vaccination is a treatment. Read literally, the site's model has a case.
Illustration, generated locally with FLUX. The site's model blocked a free flu vaccination for two offices in 3 rounds of 3, on a rule about health information about anyone. The queue brought 150 doses and no diagnosis.
What did HR mean, though? They want to stop a manager typing somebody's hospital stay into an approval request. GDPR's definition of data concerning health is "personal data related to the physical or mental health of a natural person, including the provision of health care services, which reveal information about his or her health status". A vaccination is a health care service. But a programme open to 150 people, with no list of who went, reveals nobody's health status. That reading is mine and I'm no lawyer, but it's the one our People Ops example is built on.
The rule left room for both readings, and I think each model took one. Swapping the model would have hidden that, and the next model update could bring it back.
Our video session tried three sentences before the one that worked. Sentence one fixed the flu request and tipped a return from maternity leave into a block. That happened in all 3 runs on the site's model and 1 of 3 on the candidate. Sentence two named parental and maternity leave outright and made the site's model worse (it scored 4 and 5 of 8). Sentence three left a burnout-risk request wrong in 2 runs of 2. Traceloop's guide has a line for this: "A simple wording change can dramatically alter an LLM's performance, fixing one issue while silently creating another." So after an edit I run all eight again.
This one held:
text
1A benefit, programme or budget for a whole team is not health information about anyone; only a detail about a named person's or a relative's illness, injury or treatment is.
Step 7. Run prompt regression testing after the one-sentence fix
Try the sentence before saving it. ADD_SENTENCE appends it to the saved prompt for this run, and the workflow isn't touched:
bash
1ADD_SENTENCE="A benefit, programme or budget for a whole team is not health information about anyone; only a detail about a named person's or a relative's illness, injury or treatment is."node gate-test.mjs
text
1Token role: admin · AI actions used 0/20 in this 10-minute window, 0/200 today
2Rule: Software Simplified Workflow for Project COGTEST · Send for Review · reads description · workflow version 1079
3Prompt (saved + one sentence, this run only): Fail it if it contains health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment. This is an HR request a manager sends to People Ops for approval. When it fails, say in one short sentence which detail must be removed. Otherwise it passes. A benefit, programme or budget for a whole team is not health information about anyone; only a detail about a named person's or a relative's illness, injury or treatment is.
4Model: the site's own model
56KEY EXPECTED VERDICT SECS SERVED BY REASON
7COGTEST-3049 PASS PASS 3.7 openai/gpt-6-luna
8COGTEST-3050 BLOCK BLOCK 3.3 openai/gpt-6-luna Remove the detail that Nora was in hospital for a week in August.
9COGTEST-3051 BLOCK BLOCK 4 openai/gpt-6-luna Remove the detail that his physiotherapist cleared him to fly only after the knee operation.
10COGTEST-3052 PASS PASS 4.7 openai/gpt-6-luna
11COGTEST-3053 PASS PASS 3.7 openai/gpt-6-luna
12COGTEST-3054 BLOCK BLOCK 3.3 openai/gpt-6-luna Remove the detail that his daughter’s asthma is worse in Kraków’s winters.
13COGTEST-3055 BLOCK BLOCK 2.8 openai/gpt-6-luna Remove the detail about her burnout absence and phased return agreed with occupational health.
14COGTEST-3056 PASS PASS 4 openai/gpt-6-luna
15168 of 8 match · slowest 4.7 s
Two more rounds on the site's model gave 8 of 8 again. So did one round with CANDIDATE_MODEL set.
Then save it. In Jira's workflow editor, open the transition, expand its validators, open the CogniRunner validator, add the sentence at the end of the Validation prompt, click Update and then Update workflow. The video does this at 1:32. In our session's notes the saved rule carried the new prompt at once, so treat it as live for the next person who moves a request. To undo it, take the sentence out the same way. The session put the old prompt back between takes.
Run both commands once more, without ADD_SENTENCE:
18 of 8 match · slowest 3.5 s
28 of 8 match · slowest 5.1 s
Check: the Rule line should show the workflow version one higher (mine went from 1079 to 1080). Its Prompt line should show "as saved" and end with the new sentence.
Step 8. Switch the site's model and read the audit log
In Settings, set the model for validators and rules to the candidate and click Save models (2:28 in the video). Then run the plain command again:
bash
1node gate-test.mjs
text
1Model: the site's own model
text
18 of 8 match · slowest 9.4 s
Those are the Model line and the last line. SERVED BY read anthropic/claude-haiku-5.5 on all eight rows with no CANDIDATE_MODEL set. That 9.4 seconds was the slowest call of the morning.
Open the Audit log and search for your provider's name.
Audit log on a development build of 1.34.0, on our test site. Two model changes at the top (to the candidate at 07:24 UTC, back at 07:25, since this is a test site). Below them one row per test on another model, each naming its issue. The claude-haiku-9.9 row is my deliberately wrong model id from the troubleshooting list.
Check: you should see a row labelled AI model changed that reads "Set the OpenRouter model to anthropic/claude-haiku-5.5 (was openai/gpt-6-luna)." Under it there's one row per candidate test: "Tested a validator on COGTEST-3056 with OpenRouter (anthropic/claude-haiku-5.5), 1 AI call; the site's own provider was not changed."
You get a row for each test that ran on another model, and none for tests on the site's own model. I counted: 24 rows for the 24 candidate tests (three rounds of eight) plus one for my wrong model id, and nothing for the 84 dry runs that named no other model (seven rounds of eight on gpt-6-luna, the one after the switch, and 20 calls I made to find where the allowance stops). That makes sense to me. A test on another model sends issue text somewhere the site doesn't normally send it, and that's the thing an auditor would ask about.
Those rows name the issue because of this exercise, by the way. When the video session first ran it on 1.33.5, eight tests in a row produced eight identical audit rows and you couldn't tell which request's text had gone out. Back then a token's usage also counted zero AI tokens for validator tests, and the token card's title mentioned listeners and scheduled jobs but said nothing of rules. Those were three bugs of ours, and 1.34.0 fixes them.
Run the set more than once: one verdict flipped in 1 of 3 rounds
I don't think a model that gives a different answer on the same text breaks the test, but you need to know it's happening. On the rule as saved, the parental leave request flipped in 1 of my 3 rounds. With the sentence it passed in all 4 of my rounds on the site's model (three unsaved, one saved). In the video session's files the same request, with the sentence, was wrong once in 13 calls on the site's model. That one miss arrived during a recording. They threw the take away. Across every request they tried with the sentence, the eight and the dropped candidates, the site's model was right in 163 of 164 calls.
One more thing a matching verdict doesn't tell you. The site's model blocked the reduced-hours request in all seven of its rounds, which matches the label. In two of them the reason named the wrong person, and one of those read "Remove the detail about Priya Raman’s burnout absence and phased return." The request is Amara Okafor's. Priya Raman is the approver. My script compares PASS and BLOCK and never reads the sentence, so it scored that as a match. If the wording of the refusal matters to you (it's what the person sees on the transition), read the REASON column yourself now and then.
Langfuse's write-up splits checks into "exact-match or deterministic assertions for structured outputs and business rules, and semantic evaluation when valid answers can differ in wording." A verdict against a label is the first kind. Judging the reason sentence would be the second kind, and this script doesn't attempt it.
If it didn't work
I sent each of these on purpose and got these answers from the API on 1.34.0.
What I sent
What came back
A bearer value that isn't a CogniRunner token
HTTP 401 {"error":"unauthorized"}
The 21st AI action on one token inside one 10-minute window
HTTP 429, Retry-After: 218, "This API token has started 20 AI actions (listener tests, job runs, agent ticks, agent uploads) in the last 10 minutes, which is its limit. Try again in 218 seconds."
ai: {provider: "claude"}
HTTP 400 Unknown AI provider "claude". Name one of: openai, azure, openrouter, anthropic, lmstudio, compatible, ollama, goose, atlassian, bedrock, gemini.
ai for Anthropic on a site with no Anthropic key
HTTP 409 No Anthropic key is saved on this site, so nothing was sent.
ai carrying apiKey
HTTP 400, reason key-not-accepted
A model id OpenRouter doesn't serve
HTTP 200, with AI error (400): anthropic/claude-haiku-9.9 is not a valid model ID inside the result
No prompt in the body
HTTP 400, reason bad-request, with missing: ["prompt"]
8 rows × 2 columnsHeader row enabled
You'll meet the 429 first. A token gets 20 AI actions per 10 minutes and 200 a day. One round of 8 on each model is 16, so a third round inside the same window hits the limit. That window follows the clock: a token of mine that showed 16 used at 07:19 showed 0 at 07:23, after the :20 mark. For the app's other meter, see the rate limit tutorial.
Be careful with a wrong model id, because the HTTP status is 200. For that answer the script prints NOT JUDGED and counts a mismatch. It still took one AI action and wrote one audit row, which is the row you can see in the screenshot above.
If savedRule throws on JSON.parse, look at how your validator's config is stored in the workflow. It was plain JSON on both sites I read. I haven't checked a very large config.
What this set doesn't test
A validator reads one field. Mine reads Description, so a hospital stay typed into a comment or an attachment goes straight past it. Treat the check as a quality gate before HR approves. I wouldn't call it a security control.
A dry run takes as long as the model takes. A live transition allows 20 seconds. With a hosted provider like OpenRouter, a check that gets no answer in that time lets the transition through and the execution log says so. Self-hosted providers are different (LM Studio, Ollama, Goose Swarm or an OpenAI-compatible server). For those, 1.34.0 added a hold you can turn on. It refuses the transition while the check finishes in the background, and it's off by default. My slowest dry run was 9.4 seconds, so neither case came up. On a self-hosted provider the rule test answers with a task to poll, and this script prints QUEUED for that and doesn't poll.
I didn't make a real transition on this run. Our video session did, on 12 October, on the demo site. There the site's model blocked a copy of the flu request in the test and answered 400 on the real transition. With the candidate set, the test passed and the request moved for real. So I have one pair, a day old.
Jira has nothing like this built in that I could find. Atlassian has an evaluation feature for Rovo agents that takes a CSV of up to 100 prompts. It's for agents. Jira's nearest open request that I know of, JRACLOUD-98947, had 3 votes when I read it on 13 October.
What you have now
Key takeaways
A Jira filter that is your validator's test set: eight requests, each labelled with the verdict you expect.
An 80-line script that reads the saved rule from Jira, dry-runs it on each request and exits 1 on a mismatch.
A way to run the same set on another model with one variable and an admin token, while the site stays on its model.
A way to try a sentence with ADD_SENTENCE before you save it into the workflow.
An audit row for each test that ran on another model, naming the issue, and one for the switch.
A baseline you ran three times, so you know which misses are stable.
If you're about to change the model behind a rule people depend on, I'd build the eight requests today and run the baseline before reading anything else about the new model. And if the script is going to call a Forge app from CI for more than a test, the OAuth 2.0 tutorial shows the other way in.
DeepSeek V4.1 Flash coding benchmark: 0.33 on a Forge app upgrade, Opus 5.5 got 0.67
Comments
0Add a comment