I wanted to know whether Hindsight is real software or a thin wrapper around a vector store. The agent memory server from Vectorize went from about 20,000 GitHub stars in mid-August to more than 52,000 this week, and that kind of growth makes me suspicious. It is not a wrapper. Hindsight agent memory ships its own PostgreSQL, two local models and a background worker, and on a CPU-only box with Ollama it took 16 minutes to store three short meeting notes. Recall, on the other hand, came back in about a second without calling the LLM at all.
These notes cover what I set up, what broke, and a 20-question recall test against plain vector search over the same notes. Is a local Ollama model enough, or do you need a GPU or a hosted LLM behind it? You’ll have numbers to decide by the end. I ran everything in October 2026 on an Ubuntu 26.04 LTS VM in my lab with Hindsight 1.1.0 and Ollama in Docker, 8 CPU threads and no GPU, and every log line below comes from that box.
Finding out what Hindsight agent memory actually does
Hindsight is a memory server that agents talk to over HTTP. You hand it text with retain, ask it questions with recall, and ask it to reason over what it knows with reflect. The part that makes it more than a vector database is retain. An LLM reads each note, pulls out facts, dates and entities, and links them, so “Sofia is now the on-call DBA” becomes a dated fact tied to Sofia instead of a chunk of text with an embedding. The Hindsight repository is MIT licensed, and its benchmark claims (LongMemEval leadership, “fastest-growing”) are published by the vendor, so treat them as claims.
The startup log was the quickest way to see whether there was substance behind the README. I expected a Python API in front of someone else’s database, and I was wrong. On first start the container brought up all of this by itself:
- an embedded PostgreSQL (pg0) with pgvector, schema-migrated on first start
- a local
BAAI/bge-small-en-v1.5embedder (384 dimensions, CPU) - a local
ms-marco-MiniLM-L-6-v2cross-encoder for reranking - a task worker that runs consolidation in the background
- a Next.js control plane on port 9999 and an MCP endpoint per bank
According to the docs, recall runs four searches in parallel (vectors, BM25 keywords, the entity graph and a time filter), merges them and reranks with the cross-encoder. None of that needs the LLM, which is why recall stayed fast on a box where the LLM was slow. Consolidation is the other LLM consumer. After each retain, the worker merges related facts into “observations”, and that turned out to be the expensive part on CPU.
Only the API and the control plane are published, and I bound both to 127.0.0.1 on my lab box. Everything else stays on the Compose network.
Everything inside the dashed box runs from one Compose file. The hindsight container holds the API, the UI, the two small models and the database, while Ollama is a second container that only Hindsight talks to. If you already run Ollama on the host, you can drop that service and point the base URL at the host instead, although I didn’t test that variant.
Sizing the box before you start
How big does the box need to be? The LLM decides that, not Hindsight. The Hindsight container sat at 936 to 965 MiB of RAM the whole time, which matches the vendor’s own hardware table (1.5 GB minimum, 2 GB recommended for the full image). Ollama with a 4B model and a 16,384-token context held 5.0 GB. What hurt was speed: every retain and every consolidation pass is an LLM call, and on CPU those calls take minutes.
My lab box and what the stack used on it.
- Ubuntu 26.04 LTS VM, 8 vCPU (Intel i7-6700 host), 16 GB RAM, no GPU
- Docker Engine with the Compose plugin
- about 15 GB of disk: 12 GB for the two images, 2.5 GB for the model
- outbound HTTPS on first start (ghcr.io, Docker Hub, the Ollama registry, Hugging Face)
Docker reported the Ollama image at 8.41 GB and the Hindsight image at 3.79 GB, and together they took 12 GB under /var/lib/containerd. Treat that spec as a floor for following along, not a recommendation. If your agent retains more than a handful of notes an hour, put the LLM on a GPU or point Hindsight at a hosted model; the memory server itself fits comfortably in 2 GB. If Docker isn’t on the machine yet, the Docker CE install on Ubuntu 26.04 covers it, and nothing in the stack depends on the host distro beyond Docker itself, although 26.04 is the only one I ran it on. Your user needs to be in the docker group for the commands below to work without sudo. I ended up with these versions.
docker --version; docker compose versionBoth report a current release.
Docker version 29.9.2, build a82e528
Docker Compose version v5.6.1Any recent Docker Engine with the Compose v2 plugin should do, since the stack uses nothing newer than shm_size and named volumes. The checks below also use jq, and the Python client needs a virtual environment, which Ubuntu server images don’t ship the module for. Install both:
sudo apt update && sudo apt install -y python3-venv jqThat’s the Ubuntu package name; on another distro, install jq and whatever package provides Python’s venv module there.
Choosing a model Ollama can run on CPU
Hindsight’s default for the ollama provider is gemma3:12b, which is a lot to ask of eight CPU threads, because a 12B model has three times the weights of a 4B one to push through the CPU for every token, and every retain already waits on generated JSON. So I benchmarked two 4B models first with the same prompt, using ollama run --verbose inside the container.
| Model | Size | Prompt eval | Generation | Total for one answer |
|---|---|---|---|---|
| qwen3:4b-instruct | 2.5 GB | 32.09 tokens/s | 7.18 tokens/s | 19.6 s |
| gemma3:4b | 3.4 GB | 37.04 tokens/s | 8.62 tokens/s | 27.4 s |
Gemma generated slightly faster but wrote longer answers, so qwen3:4b-instruct finished first and is smaller. It is also the non-thinking variant, which matters here because a thinking model spends its token budget on reasoning before it writes the JSON Hindsight asks for. I only ran the full Hindsight tests with qwen3:4b-instruct, so I can’t tell you how Gemma does at fact extraction. The Ollama models cheat sheet lists other small models if you want to try your own pair.
Writing the Compose file
The upstream README starts from a single docker run with an OpenAI key. I wanted Ollama next to it with nothing leaving the box, so both go in one Compose project. Create a directory for it:
mkdir -p ~/hindsight && cd ~/hindsightOpen a new compose.yaml there.
vim compose.yamlThis is the exact file my lab ended on, after the timeout fixes described further down:
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
environment:
OLLAMA_KEEP_ALIVE: 24h
OLLAMA_CONTEXT_LENGTH: "16384"
volumes:
- ollama:/root/.ollama
hindsight:
image: ghcr.io/vectorize-io/hindsight:latest
container_name: hindsight
restart: unless-stopped
depends_on:
- ollama
shm_size: 1g
ports:
- "127.0.0.1:8888:8888"
- "127.0.0.1:9999:9999"
environment:
HINDSIGHT_API_LLM_PROVIDER: ollama
HINDSIGHT_API_LLM_BASE_URL: http://ollama:11434/v1
HINDSIGHT_API_LLM_MODEL: ${HINDSIGHT_MODEL:-qwen3:4b-instruct}
HINDSIGHT_API_LLM_OLLAMA_NUM_CTX: "16384"
HINDSIGHT_API_LLM_TIMEOUT: "1800"
HINDSIGHT_API_LLM_MAX_CONCURRENT: "1"
HINDSIGHT_API_REFLECT_WALL_TIMEOUT: "1800"
HINDSIGHT_API_WORKER_ID: box1
volumes:
- hindsight-data:/home/hindsight/.pg0
volumes:
ollama:
hindsight-data:A few of those lines aren’t obvious, and each one came from something I hit or read in the configuration reference.
127.0.0.1:on both ports, because auth is off by defaultOLLAMA_CONTEXT_LENGTHandOLLAMA_NUM_CTXat 16384, since consolidation prompts reach 6,500 tokensLLM_TIMEOUTandREFLECT_WALL_TIMEOUTat 1800, up from 120 and 300 secondsLLM_MAX_CONCURRENT: "1", down from 32, so calls queue instead of splitting the CPUWORKER_ID, so the worker keeps its identity across restarts
The HINDSIGHT_MODEL default lets you swap models without editing the file. The hindsight-data volume holds the embedded database, so it’s the one to back up; the ollama volume only holds model weights you can pull again.
Pulling the model and starting the stack
Start Ollama on its own first and pull the model. Hindsight sends a test request to the model while it starts (Verifying connection in the log), so the model should already be there when it does:
docker compose up -d ollama
docker exec ollama ollama pull qwen3:4b-instructThen bring up Hindsight. The image is about 3.8 GB and took 6 minutes 20 seconds to pull on my connection, and the first start downloads the embedder and reranker from Hugging Face:
docker compose up -d
until curl -sf http://127.0.0.1:8888/health >/dev/null; do sleep 3; done
docker logs hindsight 2>&1 | grep -E "LLM:|Connection verified|Embeddings: local"The until loop waits for the API to answer before reading the log, because the first start spends a while downloading models. If it hasn’t returned after a few minutes, press Ctrl+C and read docker logs hindsight, which is where startup errors such as a wrong model name or an unreachable Ollama show up. You want to see the provider and model you set, the local embedder loading and a verified connection to Ollama:
2026-10-22 10:48:17,788 - INFO - hindsight_api.config - LLM: provider=ollama, model=qwen3:4b-instruct
2026-10-22 10:48:42,001 - INFO - hindsight_api.engine.embeddings - Embeddings: local provider initialized (dim: 384, device: cpu)
2026-10-22 10:48:43,661 - INFO - hindsight_api.engine.providers.openai_compatible_llm - Connection verified: ollama/qwen3:4b-instructThe API answered its health check 36 seconds after up. Check it and the version.
curl -s http://127.0.0.1:8888/health; echo
curl -s http://127.0.0.1:8888/version | jq -r .api_versionA healthy server reports its embedded database as connected.
{"status":"healthy","database":"connected","db_acquire_ms":1.0,"db_pool_waiting":0,"db_pool_in_use":0,"db_pool_max":100,"db_pool_idle":5}
1.1.0If Connection verified is missing, Hindsight couldn’t reach Ollama or the model name doesn’t match what ollama list shows. The Ollama commands cheat sheet has the list, pull and ps commands for checking that side.
Hitting four limits before anything worked
My first compose file had none of the timeout or context settings, and every one of them was added after something failed. In the order I hit them:
- consolidation retried every 120 seconds
- consolidation still timed out at 600 seconds
- reflect rejected for a 4,096-token context
- reflect cut off at 300 seconds of wall time
None of them is a bug in the usual sense. Hindsight’s defaults assume a hosted LLM that answers in seconds, and a 4B model on eight CPU threads answers in minutes. The very first retain looked fine (114 seconds for two sentences, 2,290 input tokens and 221 output tokens). The trouble came a minute later, when the background consolidation for that bank started and the log filled with this:
2026-10-22 09:11:00,125 - WARNING - hindsight_api.engine.providers.openai_compatible_llm - Ollama connection error (attempt 1/4):
2026-10-22 09:13:01,125 - WARNING - hindsight_api.engine.providers.openai_compatible_llm - Ollama connection error (attempt 2/4): Each warning is one failed attempt at the same consolidation call, and Hindsight allows four.
“Ollama connection error (attempt 1/4)” with an empty message
Nothing was wrong with the connection. The warnings land exactly 120 seconds apart, which is the default HINDSIGHT_API_LLM_TIMEOUT, so Hindsight gave up on a call Ollama was still generating and retried it from scratch. That first consolidation needed three attempts and 306 seconds in total. I raised the timeout to 600 seconds and thought that was the end of it.
It wasn’t. Consolidation for my three-note bank sends a 6,500-token prompt and asks for a JSON answer that matches a schema, and Ollama generated those tokens at 1.3 to 1.7 per second, against 7.2 per second for free text in my benchmark. I suspect the schema constraint, the long prompt and the 16k context all played a part, but I didn’t separate them. Three attempts in a row hit the 600-second ceiling (09:38, 09:48 and 09:58 in my log). With the timeout at 1800 seconds and one LLM call at a time, the same call succeeded in 1,233 seconds: 6,568 tokens in and 1,614 out. On CPU you’re choosing between a long timeout and no observations at all.
If you’d rather not wait on consolidation at all, HINDSIGHT_API_ENABLE_AUTO_CONSOLIDATION=false stops it from running after every retain, and you trigger it yourself when the box is idle. I didn’t run with that setting, so I can’t say how it changes recall.
“request (4610 tokens) exceeds the available context size (4096 tokens)”
The first reflect call failed with HTTP 500, and the server log carried the real error from Ollama:
openai.BadRequestError: Error code: 400 - {'error': {'message': 'request (4610 tokens) exceeds the available context size (4096 tokens), try increasing it', 'type': 'invalid_request_error', 'param': None, 'code': None}}I had already set HINDSIGHT_API_LLM_OLLAMA_NUM_CTX to 16384, and ollama ps showed the model loaded at that size, so this one confused me for a while. The traceback gives it away: reflect is a tool-calling agent, and its calls went out through the OpenAI client to Ollama’s /v1 endpoint, which has no way to carry a context size. Those requests got the Ollama server default of 4,096 tokens. Setting OLLAMA_CONTEXT_LENGTH: "16384" on the Ollama container fixed it, because then every request gets the bigger window whatever endpoint it uses.
“Reflect operation timed out after 300 seconds”
With the context fixed, reflect ran into its own clock and returned HTTP 504 with {"detail":"Reflect operation timed out after 300 seconds. Consider reducing the budget or simplifying the query."}. That’s HINDSIGHT_API_REFLECT_WALL_TIMEOUT, a separate limit on the whole reflect loop, and its default of 300 seconds is far short of what a CPU needs. I set it to 1800 and the next reflect finished in 742 seconds.
After any of these edits, run docker compose up -d again. Compose recreates only the containers whose settings changed, and the Hindsight worker picks an interrupted job back up by itself (recovered 1 stale tasks from previous run in my log).
Storing memories with retain
The REST API takes a list of items, and a bank is created the first time you write to it. This stores one note with a timestamp, which Hindsight uses for its time-based search:
curl -s http://127.0.0.1:8888/v1/default/banks/scratch/memories \
-H "Content-Type: application/json" \
-d '{"items": [{"content": "The staging cluster runs Kubernetes 1.34 on three Proxmox VMs. Philip owns it.", "timestamp": "2026-10-22T09:00:00Z", "context": "lab notes"}]}' | jq .The call blocks until extraction finishes, which took 2 minutes 11 seconds here for one sentence pair. The token counts are the useful part of the reply:
{
"success": true,
"bank_id": "scratch",
"items_count": 1,
"async": false,
"operation_id": null,
"operation_ids": null,
"usage": {
"input_tokens": 2280,
"output_tokens": 229,
"total_tokens": 2509,
"cached_tokens": 0,
"thoughts_tokens": 0
}
}2,280 input tokens for a 78-character note tells you most of that is Hindsight’s own extraction prompt. If your agent can’t block for two minutes, add "async": true to the request body and Hindsight hands back an operation ID and does the work in its worker instead; I stuck with synchronous calls so I could time them.
For anything bigger than a test, the Python client is easier. Install it in a virtual environment:
python3 -m venv ~/hs-venv
~/hs-venv/bin/pip install -U hindsight-clientFor the recall test I wrote three fictional ops notes from January, March and May 2026, with facts that change over time. An on-call DBA hands over, a database gets upgraded, a replica is retired and Grafana moves to Kubernetes. Save them as corpus.py along with the 20 questions and the answer keywords each one should surface:
vim corpus.pyPaste the notes and questions.
DOCS = [
{
"document_id": "standup-2026-01-12",
"timestamp": "2026-01-12T09:00:00Z",
"context": "ops standup notes",
"content": (
"Standup, 12 January 2026. Priya Nair runs the platform team and owns the Proxmox cluster. "
"Marcus Webb is the on-call DBA this quarter. The main Postgres server, db01, runs PostgreSQL 16 "
"and backs both Grafana and the billing app. Backups of db01 go to the NAS called vault02 every "
"night at 01:30. Marcus prefers pgBackRest over pg_dump because restoring the 300 GB billing "
"database with pg_dump took four hours. The team agreed to freeze changes on the billing app "
"during the last week of every month because invoices are generated then. Grafana lives on mon01 "
"and uses Prometheus on the same host as its main data source."
),
},
{
"document_id": "incident-2026-03-04",
"timestamp": "2026-03-04T15:00:00Z",
"context": "incident review",
"content": (
"Incident review, 4 March 2026. On 2 March db01 ran out of disk at 02:10 because WAL files piled up "
"after the replication slot for the analytics replica, db02, stopped advancing. Billing was "
"read-only for 47 minutes. Marcus Webb dropped the stale slot and freed 180 GB. Action items: "
"Priya will add a Prometheus alert when replication slot lag goes above 10 GB, and Marcus will move "
"WAL archiving to vault02. The team decided db02 is no longer needed and will be retired by the "
"end of April."
),
},
{
"document_id": "handover-2026-05-20",
"timestamp": "2026-05-20T11:00:00Z",
"context": "on-call handover note",
"content": (
"Handover note, 20 May 2026. Marcus Webb left the DBA rotation and joined the security team. "
"Sofia Lindqvist is now the on-call DBA. db01 was upgraded to PostgreSQL 18 on 16 May with "
"pg_upgrade --link and the upgrade took 9 minutes of downtime. db02 was retired on 28 April as "
"planned. Sofia prefers Patroni for the next HA design and wants three nodes across two racks. "
"Grafana moved off mon01 into a Kubernetes namespace called observability; Prometheus stayed on "
"mon01. The monthly change freeze for billing still applies."
),
},
]
# (question, accepted answer keywords, category)
QUESTIONS = [
("Who is the on-call DBA now?", ["Sofia"], "current state"),
("Who was the on-call DBA in January 2026?", ["Marcus"], "temporal"),
("Which PostgreSQL version does db01 run now?", ["18"], "current state"),
("What caused the db01 outage in March?", ["replication slot", "WAL"], "fact"),
("How long was billing read-only during the incident?", ["47 minutes"], "fact"),
("Where do the db01 backups go?", ["vault02"], "fact"),
("Why does Marcus prefer pgBackRest?", ["four hours", "4 hours"], "preference"),
("Is db02 still running?", ["retired"], "current state"),
("Which team did Marcus join?", ["security"], "current state"),
("When is the billing change freeze?", ["last week"], "fact"),
("Where does Grafana run now?", ["observability", "Kubernetes"], "current state"),
("Which host runs Prometheus?", ["mon01"], "fact"),
("Who owns the Proxmox cluster?", ["Priya"], "fact"),
("Which HA tool does Sofia want for the next design?", ["Patroni"], "preference"),
("How much disk space was freed after the outage?", ["180 GB", "180GB"], "fact"),
("How much downtime did the PostgreSQL 18 upgrade cause?", ["9 minutes"], "fact"),
("What lag threshold was proposed for the replication slot alert?", ["10 GB", "10GB"], "fact"),
("Which database server backs Grafana?", ["db01"], "multi-hop"),
("What changed on the database team in May 2026?", ["Sofia", "PostgreSQL 18", "security"], "temporal"),
("Who is adding the Prometheus alert for replication lag?", ["Priya"], "fact"),
]The loader creates the bank and retains each note with its own date and document ID, timing every call:
vim retain_corpus.pyAdd this script.
import json, time, sys
from hindsight_client import Hindsight
from corpus import DOCS
BANK = sys.argv[1] if len(sys.argv) > 1 else "ops-notes"
c = Hindsight(base_url="http://127.0.0.1:8888", timeout=1800)
c.create_bank(bank_id=BANK, name="Ops team notes")
total = time.time()
for d in DOCS:
t = time.time()
r = c.retain(bank_id=BANK, content=d["content"], timestamp=d["timestamp"],
context=d["context"], document_id=d["document_id"])
print(json.dumps({"doc": d["document_id"], "chars": len(d["content"]),
"seconds": round(time.time() - t, 1), "success": r.success,
"input_tokens": r.usage.input_tokens, "output_tokens": r.usage.output_tokens}))
print(json.dumps({"retain_total_seconds": round(time.time() - total, 1)}))Run it with unbuffered output so you see each note as it finishes. My first run used plain output and printed nothing for 16 minutes, because Python buffers stdout when it isn’t a terminal.
~/hs-venv/bin/python -u retain_corpus.py ops-notesOn my box the three notes took 950 seconds between them:
{"doc": "standup-2026-01-12", "chars": 622, "seconds": 231.9, "success": true, "input_tokens": 2417, "output_tokens": 871}
{"doc": "incident-2026-03-04", "chars": 490, "seconds": 195.8, "success": true, "input_tokens": 2387, "output_tokens": 791}
{"doc": "handover-2026-05-20", "chars": 524, "seconds": 522.2, "success": true, "input_tokens": 2391, "output_tokens": 765}
{"retain_total_seconds": 950.0}The third note took more than twice as long because consolidation of the first note was running on Ollama at the same time (this run still had HINDSIGHT_API_LLM_MAX_CONCURRENT at 2 and a 600-second timeout, so your per-note times with the final file will differ). Notice the input tokens too: about 2,400 for every note whatever its length, because most of each retain call is Hindsight’s fixed extraction prompt. That’s why one 600-character note costs as much as a much longer one, and why batching notes into fewer, larger retains saves time on CPU.
Recalling and reflecting
Recall is a plain POST with a query. The type field shows whether a result is a raw fact (world) or an observation that consolidation wrote:
curl -s http://127.0.0.1:8888/v1/default/banks/ops-notes/memories/recall \
-H "Content-Type: application/json" \
-d '{"query": "Who is the on-call DBA now?", "max_tokens": 300}' \
| jq -r '.results[] | "\(.type)\t\(.text)"' | head -5Once consolidation had run, the top result resolved the handover instead of returning both DBAs side by side:
observation Marcus Webb is no longer the on-call DBA; Sofia Lindqvist is now the on-call DBA.
observation Sofia Lindqvist is now the on-call DBA.
world Sofia Lindqvist is now the on-call DBA | Involving: Sofia Lindqvist (on-call DBA)
world Marcus Webb is the on-call DBA this quarter | Involving: Marcus Webb (on-call DBA)
world Marcus Webb left the DBA rotation and joined the security team | Involving: Marcus WebbThat first line didn’t exist in any note. It’s the bank’s own conclusion from two notes four months apart, and it’s what you pay the consolidation time for. This is the same query from my shell on box1, with both containers up and the API reporting 1.1.0.
Reflect goes a step further and asks the LLM to answer from memory, so on CPU it’s slow again:
vim reflect_test.pyThe script asks three questions that need more than one fact each and times every answer:
import time, sys
from hindsight_client import Hindsight
BANK = sys.argv[1] if len(sys.argv) > 1 else "ops-notes"
c = Hindsight(base_url="http://127.0.0.1:8888", timeout=1800)
for q in [
"Who is the on-call DBA now, and who held the role before?",
"Where does Grafana run now, and did Prometheus move with it?",
"What should the team watch for in the next HA design for db01?",
]:
t = time.time()
r = c.reflect(bank_id=BANK, query=q)
print(f"Q: {q}\n[{time.time() - t:.0f} s] {r.text}\n", flush=True)Run it the same way as the loader:
~/hs-venv/bin/python -u reflect_test.py ops-notesI stopped it after two answers because each one held the LLM for so long. Both are right, and the second one gets the Grafana move that recall’s top result missed in the test below:
Q: Who is the on-call DBA now, and who held the role before?
[742 s] The on-call DBA now is Sofia Lindqvist.
Marcus Webb previously held the role before Sofia Lindqvist.
This transition occurred on 20 May 2026, when Marcus Webb left the DBA rotation and joined the security team.
The fact that Marcus Webb is no longer the on-call DBA is confirmed in multiple records from the same date (20 May 2026), including the statement that he left the DBA rotation and moved to the security team.
The role transition is also reflected in the record that states 'Sofia Lindqvist is now the on-call DBA', which was mentioned on the same date (20 May 2026).
There is no record indicating that anyone else held the role between Marcus Webb and Sofia Lindqvist, nor is there any record of a prior on-call DBA prior to Marcus Webb.
Thus, the sequence of on-call DBAs is: Marcus Webb (before 20 May 2026) → Sofia Lindqvist (from 20 May 2026 onward).
Q: Where does Grafana run now, and did Prometheus move with it?
[1507 s] Grafana currently runs in a Kubernetes namespace called "observability" and was moved off mon01 to that location. Prometheus, however, stayed on mon01 and did not move with Grafana.
This change is explicitly stated in the observation: "Grafana moved off mon01 into a Kubernetes namespace called observability; Prometheus stayed on mon01." (mentioned_at: 2026-05-20 11:00).
An earlier statement that "Grafana lives on mon01 and uses Prometheus on the same host as its main data source" (mentioned_at: 2026-01-12 09:00) is superseded by the later statement because it has a later `mentioned_at` date. The later fact provides the current status.
Therefore, Grafana now runs in the "observability" Kubernetes namespace, and Prometheus remains on mon01 without moving.
No other facts indicate a change in the Prometheus location or that it was moved with Grafana. The fact that Grafana moved off mon01 and into Kubernetes implies a separate deployment infrastructure, but no such movement is recorded for Prometheus.742 and 1,507 seconds, and the second one used most of the 1,800-second budget, so a harder question on the same box could still hit the 504. Most of that is LLM time: the server log shows each reflect ran three agent iterations, while the one memory search inside each took about 300 ms. The log also caught the 4B model failing to make a tool call on both questions (Forced step 'recall' returned no tool call again; answering without it), so each time it answered from the observations it had already pulled. A bigger model would follow the tool protocol more reliably, and on a GPU you would have the time to run one.
Measuring recall on 20 questions
I wanted a number rather than a feeling, so I asked the bank the 20 questions in corpus.py and counted a hit when one of the answer keywords appeared in a result. Hit@1 means the top result had it, hit@5 means any of the first five. As a baseline I split the same three notes into 23 sentences, embedded them with the same bge-small-en-v1.5 model Hindsight uses, and took the five nearest by cosine similarity. That’s what a minimal RAG memory would do. The recall script:
vim recall_test.pyIt pins the query time to 1 June 2026, after the last note, so “now” means the same thing on every run:
import json, time, sys, statistics
from hindsight_client import Hindsight
from corpus import QUESTIONS
BANK = sys.argv[1] if len(sys.argv) > 1 else "ops-notes"
c = Hindsight(base_url="http://127.0.0.1:8888", timeout=600)
def first_hit(texts, keys):
for i, t in enumerate(texts):
if any(k.lower() in t.lower() for k in keys):
return i + 1
return None
rows, lat = [], []
for q, keys, cat in QUESTIONS:
t = time.time()
res = c.recall(bank_id=BANK, query=q, query_timestamp="2026-06-01T09:00:00Z")
ms = (time.time() - t) * 1000
lat.append(ms)
texts = [r.text for r in res.results]
rank = first_hit(texts[:5], keys)
rows.append({"q": q, "cat": cat, "rank": rank, "ms": round(ms), "n": len(texts), "top1": texts[0] if texts else ""})
print(json.dumps(rows[-1]))
hit1 = sum(1 for r in rows if r["rank"] == 1)
hit5 = sum(1 for r in rows if r["rank"])
print(json.dumps({"questions": len(rows), "hit_at_1": hit1, "hit_at_5": hit5,
"median_ms": round(statistics.median(lat)), "max_ms": round(max(lat))}))Run it against the bank.
~/hs-venv/bin/python recall_test.py ops-notesThe baseline needs sentence-transformers and the same embedder, and both already live inside the Hindsight image, so I ran it there instead of installing PyTorch on the host. Save it next to corpus.py:
vim baseline_vector.pyOne chunk per sentence, cosine similarity, top five.
# Plain vector search over the same notes: one chunk per sentence, same bge-small embedder, top-5 cosine.
import json, re, time, statistics
from sentence_transformers import SentenceTransformer
from corpus import DOCS, QUESTIONS
sents = []
for d in DOCS:
sents += [s.strip() for s in re.split(r"(?<=[.;])\s+", d["content"]) if s.strip()]
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
emb = model.encode(sents, normalize_embeddings=True)
rows, lat = [], []
for q, keys, cat in QUESTIONS:
t = time.time()
qe = model.encode([q], normalize_embeddings=True)[0]
order = sorted(range(len(sents)), key=lambda i: -float(emb[i] @ qe))[:5]
lat.append((time.time() - t) * 1000)
texts = [sents[i] for i in order]
rank = next((i + 1 for i, s in enumerate(texts) if any(k.lower() in s.lower() for k in keys)), None)
rows.append({"q": q, "cat": cat, "rank": rank, "top1": texts[0]})
print(json.dumps(rows[-1]))
print(json.dumps({"chunks": len(sents), "questions": len(rows),
"hit_at_1": sum(1 for r in rows if r["rank"] == 1),
"hit_at_5": sum(1 for r in rows if r["rank"]),
"median_ms": round(statistics.median(lat))}))Copy both files into the container and run it with the Hugging Face hub offline, since the model is already cached there:
docker exec hindsight mkdir -p /tmp/bl
docker cp corpus.py hindsight:/tmp/bl/
docker cp baseline_vector.py hindsight:/tmp/bl/
docker exec -w /tmp/bl -e HF_HUB_OFFLINE=1 hindsight python baseline_vector.pyI ran the same 20 questions twice: once while consolidation was still working through the bank, with Ollama using every core, and once on an idle box after consolidation had written 18 observations. The last line of each run is the summary, and the plain vector baseline ran inside the Hindsight container with the same embedder:
{"questions": 20, "hit_at_1": 16, "hit_at_5": 20, "median_ms": 1058, "max_ms": 1178}
{"questions": 20, "hit_at_1": 18, "hit_at_5": 20, "median_ms": 489, "max_ms": 627}
{"chunks": 23, "questions": 20, "hit_at_1": 15, "hit_at_5": 20, "median_ms": 17}Broken down by the kind of question, the top-result hits look like this.
| Question type | Plain vector (23 sentences) | Hindsight, before consolidation | Hindsight, after consolidation |
|---|---|---|---|
| Single fact (10) | 7 | 9 | 9 |
| Current state after a change (5) | 4 | 4 | 4 |
| Time-bound (2) | 1 | 0 | 2 |
| Preference (2) | 2 | 2 | 2 |
| Two facts joined (1) | 1 | 1 | 1 |
| Hit@1 total | 15 of 20 | 16 of 20 | 18 of 20 |
| Hit@5 total | 20 of 20 | 20 of 20 | 20 of 20 |
| Median recall time | 17 ms | 1,058 ms (box busy) | 489 ms |
On three notes, every method finds every answer somewhere in the top five. The only place to tell them apart is the top result. Hindsight beat plain vector search by one question before consolidation and by three after it. Before consolidation the lead came from single facts, where the vector baseline ranked a similar-sounding sentence first (“How much disk space was freed?” matched the 300 GB billing database sentence). After consolidation, the observations also won both time-bound questions. That’s a modest lead on a corpus this small, and I’d be careful reading more into it.
One question beat all three. “Where does Grafana run now?” returned the January fact (Grafana on mon01) at the top of the vector baseline and of the first Hindsight run. After consolidation, Hindsight’s top result was “db01 backs Grafana”, which is true but answers a different question. The May note that moved Grafana into Kubernetes only showed up at rank 3 or 4. My keyword scoring is also generous: the January DBA question counted as a hit after consolidation because the top observation names Marcus, even though it describes the handover rather than saying he held the role in January.
The latency gap is real but not alarming. The baseline embeds one query and scores 23 vectors, which takes 17 ms. Hindsight runs four searches and a cross-encoder rerank over 38 memories, and half a second is the price on CPU. While Ollama was pegged by consolidation, the same recall took about a second, so a busy LLM on the same box slows recall too even though recall never calls it.
Looking around the control plane
Port 9999 is bound to localhost, so from a laptop I reached it through an SSH tunnel (ssh -L 9999:127.0.0.1:9999 user@your-server) and opened http://localhost:9999. The first page lists the banks with their memory counts, and opening one shows the memory graph, the documents it came from and a breakdown of memory types and links.
After consolidation the bank held 38 memories: 20 facts from the three notes and 18 observations that consolidation wrote on top of them, joined by 316 links, 232 of them temporal. The Entities view shows what retain pulled out of plain prose. People, hosts, tools and even the billing change freeze became nodes, with colour showing how recently each one came up:
The UI also has Recall and Reflect pages that call the same endpoints as the curl commands above, which is handy for checking what a bank returns before you wire an agent to it. Nothing in it needs a login while auth is off, which is one more reason to keep port 9999 on localhost.
What I still don’t know
Hindsight is real software with a real design, and on a CPU it’s usable for one thing: an agent that writes a few notes an hour and reads its memory often. Recall never waits on the LLM. Anything that writes in volume needs a GPU or a hosted model behind it, because three notes took 16 minutes to store, and consolidation then kept Ollama busy for another 67 minutes of wall clock, half an hour of it on attempts that timed out. These are the questions my lab didn’t answer:
- how long retain and consolidation take on a GPU with the same model
- whether the default
gemma3:12bextracts better facts than a 4B model - how recall holds up with thousands of notes instead of three
- how the API key extension behaves when the port is exposed
The last one matters before anyone binds port 8888 to a LAN address. The configuration reference documents a built-in API key extension (HINDSIGHT_API_TENANT_EXTENSION plus HINDSIGHT_API_TENANT_API_KEY), but I kept both ports on 127.0.0.1 and never turned it on. If you’re feeding Hindsight from a coding agent, the MCP server setup for Claude Code is the natural next step, since every bank already exposes an MCP endpoint at /mcp/<bank_id>/. For a memory layer you build yourself, the Ollama and pgvector RAG pipeline is the plain vector baseline this test compared against, written out in full.
