The Best AI Memory Optimisation Was Data We Stopped Sending

Imagine a fictional user whose preferred training days changed this afternoon. The application database can store that preference directly. A summary might explain that the person is adapting a routine around a new commute. Both can be useful, but they have different jobs and different refresh rules.

If the summary remains the only place the agent reads training days, the coach can keep suggesting an old schedule until the next summary rebuild. If the application supplies the current preference separately, a slightly older summary need not overwrite it. The important design decision is the source of truth and conflict policy, not whether the memory uses JSON or prose.

Scroll the table horizontally to see all columns.

The historical memory implementation sent structured profile information into a model and asked for memory output that included those facts again. The later documented approach assembled those portions directly from database reads and merged them with the model’s synthesised sections. The system still had memory; it stopped paying the model to copy some of it.

This is a useful optimisation pattern beyond fitness. An order status, subscription tier or confirmed appointment time should usually come from the service that owns it. A generated explanation may help a person understand the state. It should not become the authority for the state simply because it is convenient to put everything into one prompt.

Another lesson in the memory history is that a profile column was not always the right representation of the most recent recorded measurement. A person can log a new value without editing their profile. Reading the convenient profile field can therefore produce stale advice even immediately after rebuilding memory.

I would write a source contract for each important fact: where it comes from, which timestamp establishes freshness, what happens if it is missing and what takes precedence during a disagreement. A fallback is reasonable when the authoritative record is absent. Silently preferring an older fallback when a newer valid record exists is a different behaviour and deserves a test.

A workout question does not always require the same information as a nutrition question. The documented system introduced specialist-specific memory extraction rather than injecting a full memory blob into every agent. This reduces irrelevant material and makes it easier to see why a particular fact was available to a decision.

Selection is not merely deletion. An injury constraint may matter to a workout answer even when it is stored under a broader profile category. A dietary exclusion may matter throughout a meal-planning exchange. I would define required constraints before choosing an aggressive token target. A shorter request that omits a necessary restriction is an unsuccessful optimisation.

A nightly memory build is not a real-time update guarantee. Directly assembled facts can be current at the build step while the assembled memory is later reused. A request-time fetch can be fresher, but it adds dependencies and latency. Neither strategy is universally correct.

The product should decide which facts are cheap to refresh, which tolerate a delay and which must be checked immediately before an action. For example, a fictional schedule summary may tolerate yesterday’s narrative, while the availability of a session being edited needs a current check. Documenting those boundaries is more valuable than saying the system has “persistent memory” without explaining when it changes.

If a user says, “That preference is wrong,” the system needs more than a sympathetic sentence. It needs to identify which record is being corrected, whether the person is authorised to change it and which derived summaries should be invalidated or rebuilt. Otherwise the same obsolete fact may reappear next session.

Memory deletion needs the same care. Removing a visible chat message may not remove a separately stored summary or embedding. I would map those stores and explain the actual deletion behaviour rather than assume that one button erases every derivative. This is a general design requirement; it is not a claim that this article certifies our full retention or deletion pipeline.

Our historical documents contain token estimates and extrapolated dollar savings. Those are useful clues, but I would not present them as a current bill or a guaranteed reduction. Model prices, traffic mixes, tool rounds and caching change the calculation.

For a meaningful comparison, hold the task set constant and record whether the answer respected constraints, used current facts and completed the requested action. Then inspect provider-reported usage, the share estimated locally and latency. A system that uses fewer tokens but causes a second clarification round may cost more per completed task. The right denominator is useful work, not merely calls.

Suppose Maya, an invented user, asks for a Friday session after changing her availability. The authoritative state says Friday is free. A narrative summary still mentions a Thursday routine. The safe assembly marks current availability separately and prevents the old narrative from overriding it.

The specialist receives the scheduling constraints and relevant training context, rather than the entire account history. Before saving, it checks the actual plan being edited. After the operation, the UI refreshes that plan. This example shows how memory selection, tool reads and frontend state cooperate; memory alone does not solve every stale-data problem.

Should every fact be fetched live? Not necessarily. It depends on change frequency, failure cost and request budget. The important thing is an explicit freshness policy and a sensible failure state when the source is unavailable.

Is retrieval the same as memory? Retrieval is one way to obtain context. Memory describes information retained for later use. A retrieved document, a current database fact and a generated summary need different authority and validation rules.

Should I publish my actual memory prompts to explain the architecture? You can teach the trade-offs with an independent example. Exact prompts and private data structures are unnecessary. The transferable lesson is choosing what the model needs to infer and what the application already knows.