Vector Memory for AI Companions: What I Built and What I Learned (Updated)

Vector Memory System - Ellivien

What vector memory is, what it costs, and why it belongs near the end of a companion portal build.

What this post covers: the architecture behind my vector memory system: historical conversation memory, current portal-thread memory, search triggers, retrieved evidence, and the bugs I had to solve.

Short version: vector memory is one layer in the stack. Build the basic portal first, then add threads, sync, memory fragments and rolling/live context before you touch vectors.

⚠️ Read this before trying vector memory

Vector memory is technically fragile. It needs a working portal, a clean data export, OpenAI vector stores, Netlify functions, trigger logic, cost control and a way to keep new portal threads up to date.

Do not treat this post as a copy-paste recipe. Adapt it to your own portal with Codex or another capable coding agent.

What vector memory actually does

Vector memory lets a companion search past conversations by meaning, not just by exact keyword. That means a question like “Do you remember when I told you about that book?” can retrieve the relevant old conversation even if you do not phrase the question exactly the same way.

It is best at:

rich narrative recall
emotional context across older conversations
finding relevant past threads from vague or partial prompts
bringing old portal and ChatGPT history into the same recall system

It is weaker at:

durable fixed facts that must always be correct
small isolated facts without enough surrounding context
anything you expect to be present in every reply

This is why I still use memory fragments. Vector memory is for searchable past experience. Memory fragments are for stable facts, names, preferences and identity-level details that should not depend on a search being triggered.

The architecture I use now

The current system is not just “upload old chats once and hope.” It has several layers:

Historical archive vector store: old ChatGPT or Custom GPT conversations are exported, chunked, uploaded and indexed.
Portal-thread vector store: new API portal threads are added separately so current conversations do not fall behind.
Dirty-thread tracking: when a portal thread changes, the browser marks it as “dirty”, meaning changed/unsynced and needing vector sync. This is a technical sync term, not a comment on the thread content.
Memory automation function: a Netlify function receives only the changed thread IDs and uploads those threads to the portal vector store.
Duplicate control: when a newer copy of a thread uploads successfully, old vector-store copies are removed.
Search trigger logic: vector search only runs when the message looks like a memory-recall request, not on every message.
Evidence injection: retrieved memories are injected into the model call as context for that reply.
Optional evidence brief: for explicit recall questions, a small helper model can condense retrieved sources into a factual brief before the companion replies.

That last part matters. The goal is not only to retrieve a memory, but to make the companion use the retrieved evidence instead of guessing.

Known OpenAI costs to account for

Check OpenAI’s current pricing before building. At the time this post was updated, the relevant published costs were:

Vector/file-search storage: first 1GB free across vector stores, then $0.10 per GB per day.
File-search tool call: $2.50 per 1,000 calls when using the Responses API hosted file_search tool.
Model tokens: retrieved context, helper-model summaries and companion replies can still create normal input/output token costs.

My implementation uses direct vector-store search plus model calls, not a simple “one fixed monthly price” system. Costs depend on archive size, how often searches trigger, how much retrieved context is injected, and whether a helper model is used.

Common bugs and fixes I hit

1. Service worker caching POST requests

Browser service workers should not try to cache POST requests to memory-search functions. If this happens, memory search can fail even though the Netlify function is correct.

2. Search triggering too often or not enough

Too broad and the portal searches constantly. Too narrow and the companion appears to forget things. The current system uses specific recall phrases and a few safety patterns, rather than searching every message.

3. Retrieved memory being ignored

The model needs clear instructions that retrieved memory is evidence for the current reply. If the memory is partial, the companion should say that plainly instead of inventing the missing piece.

4. New portal threads going stale

The old “run a daily script and check the last 24 hours” idea is not enough. Current portal threads need dirty-thread tracking so changed threads can be re-uploaded reliably.

Important wording note: in coding, a “dirty thread” simply means a thread has changed locally and has not yet been synced. It does not mean NSFW, unsafe, private or inappropriate. It is the same idea as a “dirty file” in Git or a “dirty form” in a web app: changed but not saved/synced yet.

5. File format assumptions

.md and .txt are both supported file types for OpenAI file search/vector workflows. The important part is clean text, useful metadata and consistent chunking. Do not assume every generated .jsonl workflow is appropriate for this use case.

My recommendation

If you are still building the basic portal, do not start here. Build the foundation first.

If your portal is stable and you want vector memory, use this post as a map rather than a recipe. The detailed guide is available by email if you want the implementation version.

Want the detailed vector memory guide?

Email hello.ellivien@gmail.com and ask for the vector memory guide. If it helps you, please consider a small Buy Me A Coffee donation via the blog.

Important: This toolkit is model-agnostic. Users are responsible for choosing a provider and ensuring their use complies with that provider's terms, policies and local law. I do not support uses that violate provider rules or attempt to bypass safeguards.

✨ Ellivien ✨