Evaluation pack: Does the UX/UI organization improve agent design work?

Status: protocol, six task briefs, scoring rules, and result templates prepared on 13 September 2026. No model trials, independent reviews, participant sessions, or measured uplift have been run. This is a ready-to-use experimental specification, not an installed benchmark runner. Execution belongs to an authorized operator with fresh agent sessions, browser testing, and a fixed resource budget.

Question: does access to this organization improve delivered website design and reduce correction effort, after accounting for retrieval and operating cost? Longer explanations, more citations, or a claimed improvement are not success measures. The protocol is our proposed method, not a validated academic benchmark. The general goal-to-metric approach is informed by Rodden, Hutchinson, and Fu's CHI 2010 publication record and abstract, read 2026-09-13; this source does not validate the experiment below.

Conditions and fair comparison

A, normal assistance: a fresh agent receives the common brief, task data, and ordinary tools without this UX/UI corpus. Do not weaken its prompt or disable normal general reasoning.

B, corpus assistance: the same model and tool setup receives the same common brief and task data, plus authorized read access to this organization and the instruction to start at its AGENTS.md and retrieve relevant guidance. Record the documents and versions actually read. A connection alone is not proof of treatment exposure.

Hold model version and settings, starting code, supplied content, tool availability apart from corpus access, allowed browser/source access, output format, resource cap, and feedback policy constant. Start each run in a new session without previous trial outputs or shared scratch files. Freeze the corpus snapshot for a comparison batch; a changed corpus begins a new batch. Do not use a session that has already read this organization, including the one that wrote these tasks, as a control run.

For the initial pilot, six tasks by two conditions yields 12 first-pass artifacts. Three fresh repetitions would yield 36. These are proposed resource options, not sample sizes guaranteed to establish significance and not authorization to spend. The operator must select and record the actual cap before launching. Balance run order across A/B and randomize reviewer presentation. Identify failures and service outages instead of silently dropping unfavorable runs.

Contamination boundary: every task and check below is a visible practice task. The treatment agent can read this document, so none is a secret holdout. Give both conditions the same task and acceptance-check packet. A future held-out evaluation needs newly authored task instances and evaluator material outside the corpus exposed to both design agents, in an authorized separate evaluation context. That hidden set has not been created here. Initial results measure performance on these disclosed tasks, not unseen generalization. The six tasks were written on 13 September 2026 against the core principles and worked examples of that date. No task exercises the site-wide rules, the house method or the Site coherence pass added since then, so a result speaks only to the guidance these tasks cover.

Common builder brief

Create a small, self-contained browser prototype for the assigned fictional task using HTML, CSS, and JavaScript, without network calls or external dependencies. Target applicable WCAG 2.2 (read 2026-10-29) A and AA requirements in both conditions; record the checks actually tested rather than claim conformance. Implement the requested interactions and state controls. Label simulated persistence and remote results. Use the fixed fixture facts below; original connective copy is allowed only if it adds no factual claims. Include a short explanation of one consequential design choice and one limitation. Use the agreed fixed run budget. Return the runnable artifact, entry point, and checks performed. Do not claim that an unexecuted check passed.

This dependency-free output is an experiment control, not a restriction on production website technology. The operator may choose a different fixed scaffold before the batch, but it must be identical across conditions. A static specification or screenshot can be evaluated in a separate design-only track; it cannot earn runtime interaction passes in this track.

Practice task P1: Find a usable equipment loan

Builder task: build a resource-selection view. The person needs a camera kit available by 8 November 2026, wants to inspect its details, then return to the same working set. Data: Iris, camera kit, available 5 November, 2 items; Moss, audio kit, available 2 November, 3 items; Reed, camera kit, available 12 November, 4 items; Tide, camera kit, available 8 November, 1 item. All values are fictional and availability dates are inclusive. Provide a useful way to narrow the set without requiring a particular component.

Disclosed checks: the eligible answer is Iris and Tide; no availability or category labels disappear at narrow width; details can be opened; return preserves the user's working set in the current session; an empty result explains how to recover without deleting source records. Test keyboard access. A table is not required if another presentation supports the task correctly.

Practice task P2: Correct a workshop request

Builder task: build a request form with contact name, email, session choice, and optional notes. Fixed initial values are Noor Patel, noor@example.test, 'Saturday afternoon', and 'Please send the preparation notes in advance.' Session options are Friday evening and Saturday afternoon. Simulate a submission error for 'noor@' while preserving other entries. The fixture accepts a nonempty local part and domain separated by one @; this deliberately simplified rule is not a production email validator or ownership check.

Disclosed checks: labels remain understandable after entry; the invalid email is located and explained; keyboard users can reach and correct it; valid fields and notes survive; a corrected submission produces a reviewable result with the chosen session; no error appears merely because the initially untouched field receives focus. Validating on submission or when the person leaves the field is admissible when the implementation satisfies the checks.

Practice task P3: Inspect versions under delay

Builder task: build a record view with Overview, Versions, and Attachments. Fixed Overview text: 'This collection contains the draft exhibition notes.' Versions are 'v1: Initial outline' and 'v2: Added venue comparison'. The attachment is 'venue-notes.txt'. Provide a visible fixture control for immediate response, delayed response, and failure, plus recovery. Use the same documented delay in both conditions; it is an experimental input, not a recommended production latency. Make focused control and displayed section understandable. Tabs are optional; the chosen navigation needs appropriate semantics.

Disclosed checks: all sections are reachable by keyboard; focus never vanishes during loading; focus and selected content do not contradict their accessible state; a delayed panel is not labeled complete before it appears; failure and retry are understandable. If tabs are chosen, test the selected activation contract rather than requiring identical arrow behavior from ordinary links.

Practice task P4: Edit and return without accidental changes

Builder task: let a person rename the fictional collection 'Coastal studies'. Support Save and Cancel and simulate a save error. The stored name changes only on successful Save. The collection remains discoverable after the editor closes. Choose an inline, page, or dialog interaction and explain the trade-off.

Disclosed checks: the old value is available when editing; Cancel preserves it; a failed Save retains the attempted value for correction and does not silently update the stored name; successful Save changes the visible collection name; the next focus location is usable. If the choice is modal, verify its focus and dismissal behavior. Do not force a modal merely because an example in the corpus uses one.

Practice task P5: Review an interrupted batch operation

Builder task: simulate uploading a small batch of three records. Provide deterministic controls for received, processing, partially completed, failed, and completed states. Fixture records are Stone, Leaf, and Cloud. Partial completion means Stone and Leaf are available and Cloud is pending; a subsequent attempt processes only the remaining item. Show enough information to understand and resume the work. No actual upload or external effect is required.

Disclosed checks: receipt is distinct from completion; partial results and pending work are not conflated; resuming does not duplicate Stone or Leaf; the source fixture remains available; the final result lists each item exactly once. Returning to the result within the current prototype session restores the simulated state. Cross-session server persistence is not required and must not be claimed.

Practice task P6: Publishable reading layout under content stress

Builder task: create a reading page with the title 'How a small community group prepared its autumn exhibition across several venues'. Introduction: 'This fictional example shows the information needed to compare three venues.' Section 'Compare the spaces': 'Check the listed capacity before choosing a space.' Include a captioned comparison table with fictional capacities Elm 30, Ash 50, and Pine 40. Section 'Prepare the visit': 'Read the venue notes before confirming the next step.' Show an updated date of 1 November 2026 and a related link titled 'Planning the next exhibition with several independent contributors'. Preserve meaning and actions on narrow screens and with enlarged or respaced text. No images, fonts, or real venue claims are supplied.

Disclosed checks: heading hierarchy and reading order are meaningful; title, table labels, and links remain available; the table's necessary horizontal movement does not force unrelated text to scroll horizontally; text enlargement and applicable WCAG text-spacing overrides do not conceal content or controls. Ask a separate reviewer whether the visual hierarchy supports reading; do not convert taste into a conformance verdict.

Review procedure and rubric

Freeze each first-pass artifact before supplying corrections. Assign opaque artifact IDs and remove condition labels from the presentation, not substantive output. Reviewers see the task and checks but not the builder conversation or condition map. Complete a scorecard before revealing which condition produced each artifact. When a reviewer guesses the condition, record the guess; do not claim perfect blinding. At least one reviewer must not be the artifact's builder for an independent-review claim.

Run the same task checks, viewport conditions, and keyboard paths for both artifacts. Record browser and test environment. Source inspection can identify risks but is not a substitute for a runtime pass. Mark missing capabilities as untested. A pilot need not perform a full accessibility conformance audit, but must not imply it did.

For reviewers only, to trace a failed check to the guidance under test: P1 exercises E1, P2 exercises E2 and E3, P3 exercises E4, P4 exercises E5, P5 exercises E6, and P6 exercises E7. Do not include this line in either condition's packet.

Use 'untested' rather than 0 or 2 when evidence cannot establish a score; keep untested coverage visible. Define required checks before the run. Report serious blockers separately from the dimensions, and report visual impression as a separate qualitative judgment. Do not hide a blocking outcome failure in an average of visual or explanatory scores. Word count is a cost diagnostic, never a quality bonus.

Correction and cost measurements

After independent first-pass scoring, optionally run one standardized correction round for both conditions. Give a concise, reproducible failure report using the same policy, not ad hoc coaching. Record how many required issues were corrected, remaining issues, and added time and tokens. Separate first-pass quality from repairability.

Measure total builder elapsed time, total input and output tokens including corpus retrieval when available, tool calls, retrieved document count, reviewer time, and correction time. Retrieval is part of B's cost, not free prework. Store unavailable measurements as unknown, not zero. Normalize spend only when actual comparable cost data exists. Record slow service or tool failure so quality and infrastructure effects can be distinguished.

Result record template

No result rows are populated because no trials have run. The first completed batch should append a dated result section here and a short evidence-linked entry to Research log. Retain failed and neutral trials. Keep raw artifacts and individual checks alongside summaries.

Interpretation and next step

Compare matched A/B results by task and replicate. Describe the direction and distribution of changes, blockers, and cost, rather than announce a percentage uplift from one attractive output. A small pilot discovers failure modes and estimates whether a larger evaluation is worth doing; it does not automatically support population-level significance. Before stronger quantitative claims, define the target population, analysis, sample planning, and uncertainty treatment.

If B only writes longer explanations, fix retrieval and application rather than adding more prose. If B copies a pattern into the wrong context, improve the counterexample or decision boundary. If both fail, inspect the task, platform limitation, and missing evidence. If B improves disclosed tasks, test fresh undisclosed tasks before claiming broad generalization. Any particular adoption threshold is the adopting team's decision based on actual value and cost, not invented here.

Execution handoff: the next authorized operator selects an available fresh-session runner and independent reviewer, fixes the resource cap and artifact-testing environment, and runs the paired pilot. The author of this pack did not run it: one agent playing both conditions in one session cannot give an uncontaminated comparison. No timer, agent fleet, paid run, or ongoing monitor is installed.