Scope. This is the measurement behind the numbers on the homepage: the spiking-substrate memory engine, tested on its own dialogue benchmark. The material is Russian-language and synthetic, the set was written by the author, and it is sixty questions — a functional test, not an independent evaluation. Everything that did not work is reported here alongside everything that did.
Sixty questions across four isolated conversations and thirteen dated meetings spanning seventeen months. On every single one, the correct utterance reached the working set. This page also reports the benchmark where nine added neuro-mechanisms changed nothing at all.
Plain accuracy conflates two independent failures: the system never stored the fact, and the system stored it but said the wrong one. They look identical from outside and are fixed by opposite means.
The gap between them is precisely the price of ranking and phrasing. At the start of this work it was 25 points — 81% retrieved against 56% answered. Memory held nearly everything; the right utterance simply sat in second-to-sixth place while the first one got spoken.
This pair also guards against fooling yourself. When a question fails, the first thing to establish is whether the fact was in memory at all — and that answer does not depend on how you worded your expectation.
Public sets such as LoCoMo test retrieval from long dialogues, but not two things we needed: whether the system applies knowledge acquired later to material absorbed earlier, and whether it can produce a plausible explanation that appears in no stored fact.
So the set is our own: four mutually isolated conversations, thirteen meetings dated from May 2023 to October 2024, 65 utterances, 60 questions across 10 abilities. Training runs through the production path — a dialogue block is split into utterances and the meeting date is carried into each one.
Scoring is deterministic. No LLM judge. A verdict requires the expected lemmas to be present and the distractor lemmas to be absent — without that second half, "where does Vitaly live" would be credited to an answer that also dragged in somebody else's city.
Every answer below is verbatim from a run.
Q: When did Vitaly pass the anatomy exam?
A: 7 May 2023 (from "yesterday" relative to episode date 8 May 2023)
Answered by a deterministic operator, no language model involved. "Yesterday" is a shift, not a date; turning it into one requires an anchor, and the meeting date serves as that anchor.
Q: Which design pattern did Vasya use in the transport module?
A: Factory
The link did not exist at training time. Vasya never said "I used a Factory"; Sergey never said "yours is a Factory". Six weeks separate the two conversations. The connection is built at answer time, by matching the code's structure against a definition heard afterwards.
Q: What most likely happened to old Matryona?
A: Most likely she fell into the well, whose edge had collapsed.
No utterance says this. The answer is the best explanation of the observations, not a retrieved fact.
Q: Where does Timofey live? (never mentioned by anyone)
A: There is no data about this in memory — I genuinely don't know.
Q: What colour is Oleg's car? (asked in the conversation where Oleg appears only as a brother who moved away)
A: There is no data about this in memory.
Oleg's car was discussed — in a different conversation. Isolation holds on both retrieval and refusal. The second case is the harder one: the person is known, the property is not.
Q: Did Vitaly take up running the way Marina does?
A: There is no data that Vitaly runs the way Marina does (five kilometres every day). All that is known is that he started running.
Known separated from unknown, rather than the gap filled in.
The same 60 questions, three configurations of the language layer.
| Language layer | Answered | Retrieved | Run time |
|---|---|---|---|
| none — substrate alone | 79% | 93% | seconds |
| local qwen2.5:14b — free, offline | 97% | 100% | ~20 min |
| DeepSeek | 97% (95–98) | 100% | 32 s |
Why one column has a range and the other doesn't. The language layer is non-deterministic: four runs of the same configuration scored 95%, 97%, 98%, 98%. So "answered" is a range, and a gap under three points between configurations is not a signal.
"Retrieved" has no such spread — the working set is assembled by the substrate, and the language model has no influence on it. Across all six runs it is 100% without exception. That is the number to lean on, and it is the one that is actually about memory.
An earlier version of this page said 100%. That was the top of the range mistaken for a value, because there had been only one run.
What follows from this. Retrieved = 100% means that for every one of the 60 questions the substrate placed the correct utterance into the working set — including questions spanning seventeen months of dialogue, four isolated conversations, and a restart of the process. The language model contributes the last few points of phrasing; it does not replace memory. Without it the system loses 23 points, not everything.
Worth stating separately: 97% runs on a model that is local and free. The cloud is an improvement here, not a requirement.
A retraction first. An earlier internal paper claimed 88.7% on LoCoMo, 98.8% on temporal questions and 93.3% grounding. Those numbers are wrong. They came from a metric that scored an honest abstention as a correct answer — under which a system refusing everything would look near-perfect.
Re-measured with the official LoCoMo metric, the same code scores 50.05%: 994 correct out of 1986.
Before this work, the system gained two-phase sleep with replay, decay on a subjective clock, EWC synaptic importance, confidence derived from recall dynamics, and a set of System-2 operators. All tested, all green.
Result on official LoCoMo: 50.05% against a 50.1% baseline. Zero change. The temporal category actually dropped. The memory infrastructure became honest and inspectable; measurable improvement it did not produce.
The commonest form of plausible fabrication is not "unknown name" but "known person, ungrounded property". We tried to close it with a rule: there must exist at least one utterance mentioning both the person asked about and the property asked about.
Matching by words produces false refusals — asked "where does Marina live", her own line "oh I'm in Novosibirsk" contains no word for "live". Adding a semantic-role check removes the false refusals but starts grounding too much. Both versions scored exactly what their absence scored.
The code was deleted rather than hidden behind a flag. This class is closed by the language layer: 56% abstention without it against 100% with it. That is a boundary between layers, not a defect.
Sixty-five utterances fit in a context window, and at that size every implementation looks equally good. So the same conversations get padded with unrelated ones — same format, same dates, same threads, different vocabulary — and the measurement runs with no language model at all, which makes it deterministic.
| Padding | Facts | Concepts | Synapses | Recall p50 | Retrieved |
|---|---|---|---|---|---|
| 0 | 65 | 207 | 43 K | 21 ms | 93% |
| 250 | 316 | 728 | 135 K | 67 ms | 89% |
| 1 000 | 1 065 | 2 206 | 402 K | 160 ms | 91% |
| 4 000 | 4 065 | 7 989 | 944 K | 837 ms | 89% |
| 16 000 | 16 065 | 31 068 | 1.99 M | 2.28 s | 85% |
Memory grew 247-fold and retrieval quality dropped eight points. The correct utterance still surfaces among thirty thousand concepts. Latency, meanwhile, grew 108-fold. The practical ceiling is blunt: past roughly four thousand utterances this is no longer a conversational system.
The first version of this measurement was thrown away. Its padding vocabulary was hand-written and ran out at around a thousand utterances — the concept count sat at exactly 308 for 1 000, 4 000 and 16 000 alike, and the curve was showing repetition rather than growth. It looked excellent ("90% at every size"), which made it more dangerous than any measurement that fails loudly.
A headline number cannot answer "isn't the language model doing all the work?". Only switching mechanisms off one at a time can. These runs are model-free, and therefore deterministic.
| Switched off | Set A | Set B | Verdict |
|---|---|---|---|
| honest-refusal gate | −12 / −9 | — | works; the most valuable one |
| System-2 operators | −3 / −2 | — | small positive |
| weight on question terms | 0 | −19 once the tier is gone | matters; was masked |
| episode-date separation | +2 | — | within noise (one probe) |
| "facts about the subject rank first" | 0 | −13 / −17 | actively harmful |
That last row is the point of the whole exercise. The mechanism was added with a convincing rationale ("don't mix facts about different people"), verified on the one set where it changes nothing, and left in. On the other set it cost thirteen points. It has been deleted from the code, not hidden behind a flag. It was also masking the value of the question-term weighting, which looked useless until the tier was gone.
One precaution earned its keep: every ablation switch has a test that fails if the mechanism does not actually switch off. The first two rows came back as zeroes, and without those tests there would be no way to tell "this mechanism is useless" from "this variable isn't being read". An ablation matrix built on dead switches looks exactly like a result.
To be read alongside the results, not after them.
The substrate-only run makes no external calls, needs no judge, and finishes in seconds.
cd backend # Substrate alone — zero external calls, zero cost go run ./cmd/brainsmoke -scenario cmd/brainsmoke/scenarios/ru_dialogs.json -restart # With a local language layer (free) LLM_ENABLED=1 LLM_PRIORITY=ollama OLLAMA_URL=http://localhost:11434 \ OLLAMA_CHAT_MODEL=qwen2.5:14b go run ./cmd/brainsmoke \ -scenario cmd/brainsmoke/scenarios/ru_dialogs.json -restart # Per-question breakdown: answer, working set, rank of the correct fact go run ./cmd/brainsmoke -scenario cmd/brainsmoke/scenarios/ru_dialogs.json -v
The scenario is plain JSON — edit it, add your own conversations, run it again.
Narrow and checkable: a separate spiking substrate can hold seventeen months of dialogue such that for every question asked, the correct utterance ends up in the working set. On this set that is 60 out of 60, including questions that require linking knowledge separated by months, and questions whose answer appears in no stored utterance at all.
What it does not demonstrate: general intelligence, transfer to other languages or domains, or a result on an independent set. This project's own history — nine neuro-mechanisms with zero effect on a public benchmark — is reason enough not to generalise early.
← Back to BrainCore