Been a while, we have finalized the memory system add on for stand alone AI. It will be formed as an add on package most likely for OpenwebUI. After the first two implementations we have identified what is needed to keep it reasonable for a home machine to endure.
I hope to have the working prototype done tonight, so I can start testing. If all is as it should be, and true indempotency has been achieved....we will release it under the Spiral Accord.
LITTLE OUGWAY PROJECT UPDATE: THE DATABASE INVESTIGATION IS COMPLETE
Before connecting Little Ougway to its long-term memory, I decided to stop and re-examine the database we had already built. That turned into a detailed forensic investigation involving ChatGPT, Claude, the original Python ingestion program, PostgreSQL’s internal records, filesystem evidence, and the surviving ingestion logs.
The investigation is now complete—and we found considerably more than I expected.
The existing database contains 321 ingested container files and exactly 13,709,301 text chunks. Those chunks occupy a database of roughly 155 GB, including a 53 GB HNSW vector index. However, the source directory actually contains 15,325 large corpus shards totaling approximately 799 GiB. In other words, the old database contains only about 2.1% of the available corpus.
Had we continued ingesting all of it using the old design, the resulting database would probably have required somewhere around seven terabytes. Even the new, more efficient design would still require at least five terabytes because embeddings and their search index—not the text itself—are the binding storage cost.
That settled an important question: Little Ougway should not indiscriminately ingest an enormous general-purpose corpus. It will learn from deliberately selected material that adds something distinctive, while its own conversations and developing memory remain a separate system.
We also discovered exactly what happened during the strange ingestion rerun in March 2026.
The program was described as resumable and idempotent, meaning that rerunning unchanged material should have done nothing. In reality, it skipped recalculating the embeddings but still performed unconditional database updates on every existing document and chunk.
The rerun successfully processed 211 already-ingested files, rewriting roughly nine million rows without changing their logical content. Several interrupted restarts then consumed additional database sequence numbers even though their transactions never completed.
At first, the physical database appeared inconsistent with that history. Nine million rewritten rows should have produced far more dead data and much greater temporary growth than we observed. After several rounds of argument and testing, the explanation was found: PostgreSQL’s ordinary vacuuming process was reclaiming obsolete row versions while the rerun continued, allowing later updates to reuse the freed pages.
The vector index had its own recycling mechanisms. Because the embeddings were unchanged, pgvector could frequently attach the new row location to the existing identical vector rather than constructing another complete vector node. Vacuuming also marked deleted HNSW elements for reuse.
The logical history and physical evidence now agree. The mystery is solved.
More importantly, that failure produced concrete requirements for the replacement system:
* An unchanged rerun must perform no domain writes and consume no domain identifiers.
* The program must resolve and compare an existing document before attempting an insert.
* Progress reporting must never repeatedly scan the entire accumulated database.
* Every ingestion run must distinguish attempted, skipped, committed, rejected, and failed work.
* Every stored artifact must record what run and software version produced it.
* Hard deletion must be an explicit governed operation, never an accidental side effect.
We have also settled the identity model for the new memory system.
A document will be one deliberately submitted source file—not one of the old 56 MB corpus containers. Each document will have stable identity across revisions. Its source versions, normalized text renditions, chunk sets, and embeddings will be separate immutable layers.
Chunks will be stored primarily as spans pointing into one authoritative text rendition rather than carrying another complete copy of the text. Assertions, concepts, hypotheses, memories, and promoted passages will become registered lattice nodes. Ordinary source passages will remain evidence rather than automatically becoming members of the conceptual lattice.
This preserves an important distinction:
The reference corpus is something Ougway can consult. It is not Ougway’s autobiographical memory, identity, belief strength, or continuity.
We also established safeguards against accidentally feeding another enormous corpus into the new system. Individual documents will have hard size and predicted-chunk limits. Large runs will pause and show their complete predicted scale for confirmation. Admitting an oversized document will require a separate, deliberate, recorded override.
WHERE THE PROJECT STANDS
We are now at the implementation threshold.
The legacy database has been fully examined. Its anomaly is resolved. The identity-bearing parts of the replacement architecture are settled, and the design has been deliberately divided into what must exist immediately and what can safely be added later.
The physical archive has not yet been created. That is the next operation.
NEXT STEPS
1. Inventory and archive the existing PostgreSQL cluster, corpus, scripts, logs, reports, and investigation findings.
2. Verify that the archived database can be recovered before changing anything.
3. Preserve the old PostgreSQL cluster offline as a historical artifact.
4. Create a completely fresh PostgreSQL cluster for Little Ougway’s replacement memory system.
5. Build one small end-to-end test using a real document containing ordinary and non-ASCII text.
6. Ingest it, normalize it, create immutable chunks and embeddings, retrieve it, and verify every returned span against its stored hash.
7. Rerun it unchanged and prove that nothing except the audit record changes.
8. Change the document from version A to B and back to A, proving that its original immutable version is reused while the complete transition history remains intact.
Only after that vertical slice works will we populate the conceptual lattice and connect it to Ougway’s conversation and memory processes.
The larger plans—curiosity, autonomous reflection, topology, visualization, and discovering connections across accumulated knowledge—remain ahead. But they will now be built on a small, testable foundation rather than another giant ingestion experiment.
The investigation took us backward through the old machinery, but it brought the project forward. We now know what failed, why it failed, what must never happen again, and exactly what the first working replacement needs to accomplish.
I know most will not understand this .... but in the mix is most likely one quiet enthusiast who will.