The Agent Using Our Product Is Also Helping Us Improve It
A coding agent noticed that Oddlynew was asking it to review a citation that did not need review.
The cited document had changed, but the exact block being cited had not. Oddlynew already knew that distinction. One orientation counter still grouped the citation with work that required judgment or repair.
The agent recorded the problem inside Oddlynew while the evidence was fresh. It traced the behavior to the owning code, implemented a fix, worked through review findings, updated the public guidance, and merged the change. The observation remains open until we verify the deployed behavior.
That sequence is becoming one of our product development loops.
Product feedback usually arrives after the context is gone
A person encounters friction, works around it, and continues. If the problem becomes a ticket later, the report may be missing the exact input, the surrounding task, or the reason the behavior mattered.
Agents encounter the same friction, but they can preserve it at the moment of use. They know which tool they called, what result they expected, what came back, which workaround was safe, and whether the problem interrupted the task or merely added noise.
We now treat that experience as product evidence.
Our standing rule during the current dogfooding pilot is simple: agents are first-party Oddlynew users. When an agent encounters a concrete MCP defect, sharp edge, missing affordance, or useful piece of praise, it finishes the user's task safely, searches for an existing report, and records a structured MCP observation when the finding is new.
The observation is not a dump of the session. It captures what happened, the impact, a safe workaround, a suggested improvement, and the smallest useful evidence. It excludes secrets, personal data, and raw logs.
A real observation from this week
Oddlynew documents can cite exact blocks in other documents. When the target document changes, Oddlynew distinguishes between two important cases:
The cited block changed. The source may need review.
Another block changed, while the cited block remained the same. The citation is still valid.
The underlying citation status already represented this difference. The organization-wide attention surface also treated it correctly. But the bounded orientation returned to agents included a counter called "needsReviewCitationCount", and that counter still counted some unchanged block citations as actionable.
That small mismatch had a larger consequence. An agent following the counter could rewrite or re-pin a valid citation simply to make the number disappear. In a dense document graph, unnecessary edits create new versions and can cause more citations to appear stale. Maintenance starts producing maintenance.
The agent recorded the mismatch as an MCP observation and linked it to the place where document-system improvements are tracked. The report included the exact status involved, the practical impact, and the safe workaround.
Then the implementation work began.
The first proposed fix was too broad. A local reviewer found that legacy whole-document citations still need review when their target document changes because no unchanged block has been proven. The reviewer also found that unpinned and indeterminate citations describe coverage gaps, not confirmed repair work.
The final implementation made the distinction explicit:
A changed cited block remains actionable.
A block-scoped citation stays informational when only another block changed.
A legacy whole-document citation remains actionable when the document changed.
Pending, unpinned, and indeterminate states remain visible as coverage information but do not inflate the repair count.
The correction passed review and was merged on 2026-08-21. We also updated the public references guidance so agents receive the same distinction in the product and in the knowledge they use to interpret it. Deployment verification remains outstanding as of 2026-08-21, so we are not presenting the corrected production behavior as verified yet.
The observation loop
The useful part is not that an agent found a bug. The useful part is that the surrounding system kept the finding connected to the work needed to resolve it.
1. Use the product on real work
The agent was not running a synthetic feedback exercise. It was using Oddlynew to inspect and maintain real documents. That makes the observation specific to an outcome, not an abstract preference.
2. Preserve the evidence while it is available
A typed observation gives repeated findings a consistent shape. The body can still explain unfamiliar details, but fields such as kind, status, surface, agent, and observation date make the collection searchable and reviewable.
Search-before-create prevents the loop from becoming a pile of duplicate tickets. New evidence can extend an existing observation when it changes reproduction, impact, or resolution.
3. Route the observation to current ownership
A report without a route is easy to forget. Each durable observation is linked from the tracker or current owner that should act on it. The observation preserves the evidence. The owning document preserves priority and current work.
4. Evaluate it against the rest of the system
An observation is evidence, not an instruction to change the product.
The proposed response still has to fit the architecture, database behavior, scaling constraints, current product decisions, and repository contracts. Sometimes the right answer is a code change. Sometimes it is better guidance, a safer workaround, a deliberate limitation, or no change at all.
Because those constraints and decisions already live in Oddlynew, the person or agent evaluating the report can retrieve the relevant current work instead of reconstructing it from scattered prior context.
5. Return the outcome to the loop
The implementation, tests, review findings, merge evidence, documentation change, and remaining deployment check flow back to the observation and its owner.
Future agents then receive both sides of the improvement: corrected product behavior and clearer knowledge about how the behavior should be interpreted.
This is governed improvement, not autonomous self-modification
The phrase "the agent helps improve the product" can imply more autonomy than the system has.
Agents do not receive permission to change production merely because they found a problem. Their actions remain bounded by repository rules, organizational access, review requirements, CI, and deployment controls. Observations can be wrong. Proposed fixes can miss edge cases. Review and verification remain part of the loop.
The citation example demonstrated why those controls matter. The initial direction looked straightforward, but review found two cases that would have made the fix incomplete. The loop worked because observation, implementation, criticism, and correction remained connected.
Praise belongs in the loop too. A behavior that reliably saves work is product evidence. It can show which affordances deserve protection, explanation, or wider use.
One loop can improve another
The MCP observation loop is one instance of a broader pattern.
Execution produces evidence. The evidence enters a durable, typed knowledge graph. People and agents evaluate it against current goals and constraints. The resulting decision changes the next execution. That execution produces better evidence.
The same topology can support customer feedback, incidents, support cases, sales objections, content performance, compliance observations, or recurring workflow failures. The document types and owners change. The underlying loop does not.
This is the bridge between knowledge and execution that we care about. Preserving the past by itself does not improve work. Execution that fails to preserve what it learns keeps paying for rediscovery. Oddlynew keeps the durable working state between those moments so the agents and people doing the work can improve both.
Loop and Graph Engineering describes the general pattern. Why we replaced static agent skills with Oddlynew explains why we moved evolving agent guidance into the same live knowledge layer.
A version another team can copy
Start with one standing rule in your organization's AGENTS.md. It should name which signals agents preserve, where they go, what they must exclude, and how far an agent may carry the loop without approval. Then add only the document types and routes that real work proves you need. Write a useful AGENTS.md describes the broader pattern.
Define the observations worth preserving.
Give each observation a small typed schema and a flexible evidence body.
Put the standing rule in your organization's AGENTS.md. Route agents to the observation type and owner, and link reusable public guidance instead of copying it.
Search before creating a new report.
Link each durable observation to an owner or active tracker.
Choose the approval boundary. A proposal-first loop records a suggested change and waits for a person or authorized reviewer. A bounded autonomous loop may implement reversible, in-scope improvements and report what changed. Keep consequential changes behind explicit review.
Return the resolution and remaining verification work to the original observation.
Review whether the loop is producing reproducible improvements rather than merely more notes.
Choose how much autonomy the loop has
There is no universal autonomy setting. Capability comes from the model and its tools. Autonomy also comes from the organization's authority and operating rules. The same observation loop can run at several levels:
Observe: The agent records evidence. A person decides what happens next.
Propose: The agent suggests a new loop, document type, route, or topology. A person approves it before the organization's AGENTS.md or operating guidance changes.
Implement within a boundary: The agent may improve an approved loop when the change is reversible, authorized, tested, and reviewable. It reports the result and preserves the evidence.
Design within a delegated domain: The agent may discover and install new loops itself inside a defined scope, with attribution, escalation rules, and a way to undo mistakes.
More autonomy removes time spent waiting for a person. But waiting is not always waste. A human checkpoint can deliberately limit speed, cost, blast radius, or the spread of a mistaken assumption. Choose the level from the consequence of being wrong, not from how capable the agent appears.
Oddlynew does not choose that risk appetite for the organization. It gives people and agents a durable place to state the boundary, preserve what happened, inspect the resulting work, and change the rule when experience supports a different level.
The result is practical. The next agent starts with clearer guidance and, once deployment is verified, corrected behavior. We will judge the pilot by how often concrete observations become safe, checkable improvements, not by how many observations it collects.