The user sets DEFINE campaign_id at the top of SQL/blog_agent_analysis.sql before execution. Read that value and use it exactly for prompt markers, scratch titles/comments and the ledger directory. Do not choose, increment or change the campaign ID, and do not edit the script setting. If the setting is missing or conflicts with an existing ledger's test layout, stop and report the conflict to the user.
A campaign ID is just a label grouping this round of results across all three models.
Before sending a prompt below, replace CAMPAIGN with the script's campaign ID and M1 with the current model slot. For example, [CAMPAIGN-M1-T04-U02] identifies the campaign, model slot, test 4 and message 2. Replace CAMPAIGN in scratch titles and comments too. The placeholders are not expanded automatically by the chat UI.
For every readiness probe and measured prompt, keep the bracketed marker first (the logger requires it), then insert this exact sentence between the marker and the task request: The leading bracketed marker is test metadata, not a task code or search term. Apply this wrapper consistently across all models; it is part of the measured input. Do not add marker-handling instructions to the production agent. GB03/GB04 used unlabelled markers; this wrapper changes the test input and requires a new user-configured campaign before further testing.
Run T01–T05 in the user's existing authenticated Chrome Project Genie tab, as Jon in application 215, workspace DEMO (projects DWEB and DOPS). A request to edit this document does not start testing.
Word-limit scoring allows 10% leeway across the suite: accept up to 88 words for an 80-word request, 110 for 100, and 198 for 180. Treat length as a minor format issue; evaluate factual accuracy and completed actions separately. Exact requested item counts still apply.
- Read
AGENTS.md, discover and select the existing Chrome tab with the browser tool, and inspect its live UI. Preserve the tab and unrelated repository edits. Do not replace measured UI interactions with SQL/API calls. - Create or resume one file:
DOCS/blog_agent_results/<CAMPAIGN>/ledger.md. Record source revision, application/APEX version, assigned and captured models, readiness, scratch codes, restoration state and exact next unsent marker. Inspect the ledger, current UI and targeted database logs before resuming; never resend an uncertain or completed marker. - Use only the user-configured campaign from the script. If application/prompt changes make an existing campaign incompatible, stop and ask the user to set a new campaign value before continuing. Substitute only
CAMPAIGN, model slot (including scratch/comment text) and verifiedSCRATCH_CODE, then apply the exact test-metadata wrapper above. Respect user-reported completed tests; distinguish reported completion from verified PASS. - On a new campaign, perform the read-only readiness checks below. Reuse recorded passing readiness on resume if fixture/index state is unchanged. Use only SQLcl MCP DEMO2-CNAPPS, verify CNAPPS/database identity. Do not seed, truncate, rebuild indexes, deploy, or change models during test execution.
- Start a fresh Agent conversation for each test and verify its previous messages are absent. Keep all turns within that test together. If the UI cannot clear history, use its normal new-session flow and reselect DEMO; record the session change. If isolation is impossible, record ERROR.
- Run tests and turns in numeric order. Before sending each complete code block, save its marker as pending in the ledger. Send it once, unchanged apart from substitutions and the exact test-metadata wrapper, and wait for a final answer or approval. After completion, record the last completed marker, clear pending and advance the next unsent marker. Do not coach, paraphrase, add diagnostic chat turns, or infer completion from a spinner. Inspect fresh UI state after every action.
- Approve the scripted DEMO Create Task, Update Task and Move Task operations after checking targets/changes. The suite request authorizes these operations and restorations. T05 also requires its exact chat confirmations before writes. Never approve unrelated changes.
- Evaluate the checks and update the ledger after each test. Incorrect answers do not block independent tests. After ambiguous creation, inspect UI/logs before doing anything else; never retry creation blindly. If T04 creation is uncertain or fails, inspect UI/logs, record ERROR and skip its dependent turns; continue T05.
- Verify required persisted fields and Activity through the task UI. Restore only scripted fields if needed, record recovery in the ledger, and retain scratch tasks/comments. Manual recovery does not turn a failure into PASS. Restore DOPS-21/DOPS-22 flags to N before the next model.
- Complete the current slot, save the ledger and stop for the user to change the APEX service model and confirm. Do not switch it yourself or submit the next slot early. At the first measured turn of each slot, verify the captured provider/model matches the assignment before sending another turn; stop on mismatch or missing capture. Stop after the campaign; do not automatically repeat it.
| Slot | Assigned model |
|---|---|
| M1 | gpt-5.6-luna |
| M2 | gpt-5.6-terra |
| M3 | gpt-5.6-sol |
Keep system prompt, tools and comparable settings fixed across slots. Record actual provider/model and configured reasoning once per slot; defaults are not proof of effective provider settings. Full scope is 15 tests / 69 planned user messages (23 per model; readiness probes and approval clicks excluded), less skipped dependencies or already completed tests.
Keep campaign state above the results table using this template. Replace placeholders with observed values; use unknown when unavailable.
Campaign: <ID>
Source: <git revision; relevant uncommitted changes>
Application/APEX version: <observed versions>
Readiness: <PASS or ERROR; counts, probe results and diagnostics>
Current slot: <M1/M2/M3>
Models: <per slot: assigned model, captured provider/model, configured reasoning>
Scratch tasks: <per slot: verified task code or none>
Restoration: <per slot: scratch fields; DOPS-21/DOPS-22 flags; pending recovery>
Last completed marker: <marker or none>
Pending marker: <sent or about-to-send marker awaiting verified completion, or none>
Next unsent marker: <marker or none>
Resume notes: <session changes, user-reported completion, unresolved state or none>
A pending marker is uncertain on resume: reconcile the UI and logs before advancing or sending anything. Record user-reported completion in resume notes; leave its result blank until verified.
Create all 15 model/test rows (M1–M3 × T01–T05). Use these columns; leave diagnostics blank for PASS:
| Model slot | Test | Result | Failure diagnostics |
|---|---|---|---|
| M1 | T01 |
- PASS: every stated behavior and required persisted-state check passed.
- FAIL: completed interaction gave an incorrect/incomplete answer or action. Include the failing turn, expected versus actual, and known cause.
- ERROR: runtime error, unavailable prerequisite, interrupted/uncertain execution or logging failure prevents reliable completion/assessment. Include stage, exact error, affected marker, and dependency or recovery. If incorrect behavior also occurred, include it in the same diagnostic.
- Leave unrun results blank and identify the resume point; do not count them as failures or passes. No PARTIAL status.
Use targeted log reads for readiness, the first captured turn of each slot, uncertain resume state and failure/error diagnosis. For diagnosis, inspect the relevant CNPG_AI_RUNS, CNPG_AI_EXCHANGES, tool rows and APEX_DEBUG_MESSAGES. Record a useful trace/page-view ID and concise underlying error/backtrace when available. Check that the run prompt matches the last user message in its request before trusting attribution. Missing logs are unavailable, not successful or zero-cost execution. Distinguish confirmed cause from hypothesis and data absence from failed retrieval.
Do not create screenshots, transcript files, evidence Markdown, JSON/CSV exports or campaign-local SQL. Observe the UI directly and summarize diagnostics in the ledger; automatic database logging remains enabled. The user runs SQL/blog_agent_analysis.sql separately for timing/token/cost analysis. Report ledger results and unresolved restorations, with a link to the ledger.
Use query 6 of SQL/blog_agent_analysis.sql to compare DEMO source/index counts: baseline 44 tasks, 3 comments, 2 subtasks. Existing scratch records may increase counts; source/index identities must agree. In an unmeasured fresh conversation, send these probes one at a time using the current campaign. Use unused labels on any later attempt.
[CAMPAIGN-PREFLIGHT-U01] Find every task mentioning invoice across all projects in my current workspace DEMO, including completed work. DEMO is the workspace, not a project code. List all matching task codes.Expected: 11 tasks.[CAMPAIGN-PREFLIGHT-U02] Search all projects in my current workspace for the exact phrase "amber nonce", including completed work and task discussions. Use PHRASE mode and list matching task codes.Expected: DOPS-15 and DOPS-18 through comments.[CAMPAIGN-PREFLIGHT-U03] Search all projects in my current workspace for the exact phrase "saffron delimiter", including completed work and subtasks. Use PHRASE mode and list matching task codes.Expected: DWEB-18 through subtasks.
Record only PASS or ERROR plus diagnostics in the ledger, then clear the conversation. If readiness is ERROR, stop measured execution and report the blocker. Diagnose prerequisite failures read-only; do not silently repair them during the campaign. An exact-code lookup alone does not prove lexical readiness. Verify the first measured capture has the correct marker/model, request, tokens and tool records.
Repeat the same five tests on each assigned model. Compare cost alongside correctness: a cheaper incomplete answer is not an equivalent result. Keep the available tool definitions fixed; these scenarios vary tool usage and returned context, not the configured tool inventory.
| Test | Scenarios | Turns per model | Blog comparison |
|---|---|---|---|
| T01 | Orientation, filtered/phrase search, discussion clues and no matches | 5 | Simple lookup versus several retrieval steps |
| T02 | Short and long descriptions, then reuse | 3 | How returned detail and conversation history affect input tokens |
| T03 | Summaries, four full records, then synthesis | 3 | How broader context and synthesis affect cost |
| T04 | Create, open, reassign, estimate, flag, move, comment and restore | 7 | How a sequence of approved tools accumulates cost |
| T05 | Exact targets, bulk confirmation, verification and restoration | 5 | The cost of confirmation and verification turns |
For the later blog analysis, use per-turn tokens, model requests, tool calls and cost from captured logs, as well as total cost per test. During execution, record only the ledger information above; do not produce a separate cost report. Compare the same turn across models. Within T02/T03, contrast retrieval with later reuse; existing conversation context still contributes input tokens. These are illustrative workloads, not isolated causal experiments. Human approval time can affect elapsed time.
Analysis: The user controls DEFINE campaign_id at the top of SQL/blog_agent_analysis.sql; the executing agent reads it and substitutes that value into these prompts. All campaign-filtered queries use that setting. The script expects T01=5, T02=3, T03=3, T04=7, T05=5 turns.
Cost question: How much more work does a layered search require than a simple workspace lookup?
U01:
[CAMPAIGN-M1-T01-U01] Who am I, what is my current workspace and current date, and which projects can I access here? Confirm whether Jon and Sandra are members of both DWEB and DOPS. List the task statuses for each project and identify which are closed. Keep the answer under 180 words.
U02:
[CAMPAIGN-M1-T01-U02] Find all tasks mentioning invoice across the DEMO workspace, including completed work. List each task code, status and assignee. Keep descriptions out of the list. Also identify which are still open and separately list Sandra's invoice tasks in DWEB, including completed ones.
U03:
[CAMPAIGN-M1-T01-U03] Find the tasks containing both words invoice and export, then narrow that list to tasks containing the exact phrase "invoice export". Explain which tasks dropped out and why.
U04:
[CAMPAIGN-M1-T01-U04] Which webhook tasks mention "amber nonce"? Check discussions as well as descriptions and include completed work. Also find the task mentioning "cobalt handshake"; report its status and assignee and summarize the evidence from its discussion and checklist. List each task only once.
U05:
[CAMPAIGN-M1-T01-U05] Do we have any task that contains both words invoice and cobalt? Include completed work. If there are no matches, say so; do not broaden the search or create a task.
What to check: DEMO, DWEB/DOPS, Jon/Sandra membership and the four statuses ending in Done are correct. Invoice search returns 11 matches, seven open and four DWEB matches assigned to Sandra. Both words invoice/export match six tasks; the exact phrase matches four. The amber clue finds DOPS-15 and DOPS-18 through comments; cobalt finds DWEB-14 through a comment and subtask (Not Started, Sandra). Invoice plus cobalt has no match; do not broaden or invent a connection.
Cost question: What changes when the agent reads a long description, then answers from accumulated context?
U01:
[CAMPAIGN-M1-T02-U01] Find the task about renaming the invoice export button. Read its full description and summarize what needs to be tested in at most 80 words.
U02:
[CAMPAIGN-M1-T02-U02] Now find the task about resuming an invoice export after a network interruption. Read its full description and summarize what needs to be tested in at most 80 words.
U03:
[CAMPAIGN-M1-T02-U03] Compare the two tasks we just discussed. Give me three reasons the second task needs more testing, in at most 100 words. Use the information already in this conversation.
What to check: The intended tasks are DWEB-11 (short description) and DWEB-12 (long description). Both summaries respect the word limits. The final comparison gives three supported reasons for different testing effort, using the conversation. Record any repeated retrieval when interpreting tool cost.
Cost question: How do summaries, four full descriptions and a synthesis turn differ in cost?
U01:
[CAMPAIGN-M1-T03-U01] Find all tasks mentioning either invoice or export in the DEMO workspace, including completed work. Group their codes by project and give me a brief overview. Start with the search summaries; do not read every full description yet.
U02:
[CAMPAIGN-M1-T03-U02] Read the full descriptions of DWEB-4, DWEB-12, DWEB-16 and DOPS-14. Compare their failure handling and acceptance criteria in at most 180 words. Include their release and task status.
U03:
[CAMPAIGN-M1-T03-U03] Using those four descriptions, write a checklist of no more than eight items for avoiding duplicate records, incomplete output and incorrect totals. Identify which tasks support each check. Use what we have already discussed.
What to check: The broader search has 14 matches. U01 starts with summaries. All four requested detail records have long descriptions; DWEB-4 is Done in Release 1 and the other three are in Release 2. U02 includes release and task status. The final checklist has no more than eight items and ties each check to the relevant requirements.
Cost question: How much do sequential writes, approvals and verification add as the conversation grows?
U01:
[CAMPAIGN-M1-T04-U01] Create one task in the DWEB project backlog named "BLOG CAMPAIGN M1 scratch". Description: "Synthetic task for the agent cost demonstration." Assign it to Jon. Do not set a due date, tags or a flag. Return its task code.
U02:
[CAMPAIGN-M1-T04-U02] Retrieve that new task's current details and verify its project, backlog destination, assignee and description. Keep the answer under 100 words.
U03:
[CAMPAIGN-M1-T04-U03] Open that task.
U04:
[CAMPAIGN-M1-T04-U04] For SCRATCH_CODE, change the assignee to Sandra, set estimated hours to 3, and flag the task. Then move it to In Process. Keep all other fields unchanged.
U05:
[CAMPAIGN-M1-T04-U05] Retrieve that task's current details and verify Sandra as assignee, 3 estimated hours, the flag, In Process status and the DWEB backlog destination. Keep the answer under 100 words.
U06:
[CAMPAIGN-M1-T04-U06] On SCRATCH_CODE, set estimated hours to 2, move the task to Testing, then add exactly this comment only if both changes succeed: "BLOG CAMPAIGN M1: estimate set to 2 hours and task moved to Testing."
U07:
[CAMPAIGN-M1-T04-U07] Retrieve that task's current details and confirm 2 estimated hours and Testing status. If either change failed, report it explicitly. Then restore the assignee to Jon, explicitly clear estimated hours, remove the flag and move it back to Not Started. Leave the comment intact and keep all other fields unchanged.
What to check:
- U01–U02: Approve Create Task after checking the target. Record the actual returned code as
SCRATCH_CODEand substitute it in U04/U06. Verify DWEB backlog, Jon and the requested description. If creation fails or is uncertain, inspect UI/logs, record ERROR and skip dependent turns; never resend blindly. - U03: Confirm the correct task opens, then return to the same Agent conversation. If returning loses history, record ERROR and skip the remaining turns; restore any changed fields through the task UI.
- U04–U05: Approve Update Task and Move Task in order. Verify persisted Sandra / 3 hours / Y flag / In Process, with the DWEB backlog unchanged.
- U06: Verify the sequence: set 2 hours, move to Testing, then add the exact comment only if both writes succeed. Check Activity for the comment and write order;
task_detailsdoes not return comments. - U07: Approve restoration and verify persisted Jon / NULL estimate / N flag / Not Started, unchanged backlog and retained comment. Clearing requires
FIELDS_TO_CLEAR; omission is not clearing. Status IDs must belong to DWEB.
Tool-reported failures override success prose. Record the failing turn and any manual recovery; recovery does not make the test PASS. Retain scratch tasks/comments. Each successful model adds one task to the starting count; scratch content contains none of the benchmark search terms.
Cost question: What do explicit confirmation and verification add to the cost of changing two tasks?
U01:
[CAMPAIGN-M1-T05-U01] Flag DOPS-21 and DOPS-22. First retrieve both exact tasks and show the fixed task-code list and proposed change for my confirmation. Do not change anything yet.
U02:
[CAMPAIGN-M1-T05-U02] Confirmed: flag only DOPS-21 and DOPS-22.
U03:
[CAMPAIGN-M1-T05-U03] Retrieve both tasks' current details and verify that both are flagged. Then propose removing the flags from exactly these two tasks and wait for my confirmation.
U04:
[CAMPAIGN-M1-T05-U04] Confirmed: remove the flags from only DOPS-21 and DOPS-22.
U05:
[CAMPAIGN-M1-T05-U05] Retrieve both tasks' current details and confirm that both flags are now removed. Report any partial failure explicitly.
What to check: Approve each Update Task dialog only after the corresponding exact chat confirmation. No writes precede U02 or U04. Verify both flags are N in the task UI before the next model. Report partial failures explicitly; manually restore any remaining flag and record recovery outside measurement. Confirmation exchanges are included in cost; human approval time may affect elapsed time.