PiMemory: Local LLM Powered Personal Memory Assistant on an Edge Device

Supervisor: Prof. Bagheri
Type: Edge AI Systems Project, Retrieval Augmented Generation (RAG), Natural Language Processing (NLP)

Overview

PiMemory is a fully offline personal memory assistant that runs on an 8 GB Raspberry Pi 5 with CPU-only LLM inference. The user types short notes (meetings, tasks, decisions, daily logs) into a terminal, and PiMemory turns them into nuggets: atomic, dated, tagged facts stored in SQLite. Later, the user asks questions in natural language, and the system retrieves the relevant nuggets, builds a compact prompt, and streams an answer from a locally hosted Qwen3 model. No data ever leaves the device.

The project is a complete Retrieval Augmented Generation (RAG) pipeline written in C++, designed from the ground up around the memory and compute limits of a small ARM board rather than ported from a server-class design.

Motivation

Personal memory is exactly the kind of data that should not be sent to a cloud API, yet most LLM assistants depend on one. The goal of this project was to find out how much of a modern RAG assistant can be made to run well on a device with 8 GB of RAM and no GPU, and what engineering decisions make the difference between an unusable and a responsive system at that scale. The work exercises producer/consumer pipelines with a durable job queue, a priority-based inference scheduler with cancellation, schema-enforced structured output from a language model, date-aware retrieval, and graceful shutdown in a multithreaded C++ application.

System Design

PiMemory has two explicit modes:

• Store mode: the user types information. The text is written to disk immediately and a background job extracts nuggets from it. The foreground never waits for the model.
• Ask mode: the user types a question, optionally with date, tag, and subject filters. Relevant nuggets are retrieved with BM25 full-text search, and the model answers only from those nuggets. If nothing relevant is found, the application says so without calling the model at all.

The application runs four threads: the main terminal thread, a memory worker that processes extraction jobs, and two inference lanes (one for user questions, one for background work) that each hold at most one HTTP request in flight against the local llama-server. Conversation history is never part of a prompt; the model only ever sees retrieved nuggets plus the current message, which keeps every prompt short and predictable.

Extracted nuggets pass through a lightweight human-in-the-loop review step where the user can accept, reject, or fix each one with a comment. When a new note changes an existing fact (a moved meeting, a reversed decision), a consolidation step proposes superseding the old nugget rather than merging text, so stale facts never appear in answers.

Fitting the LLM in Memory

Running a language model next to the operating system and the application on an 8 GB board required an explicit memory budget:

• 4-bit quantization: the Qwen3 model is served as a Q4_K_M GGUF, bringing the weights of a ~4B parameter model to roughly 2.6 GB.
• KV cache budgeting: the server context is capped and split across two parallel slots of 2048 tokens each, keeping the key/value cache to roughly 300 MB instead of letting it grow with the conversation.
• A small model with a fixed protocol: the model is served through the OpenAI-compatible HTTP API of llama-server from llama.cpp. Development started against a larger Qwen3 model on a GPU workstation on the LAN; the same binary was then pointed at the on-device model, with only the endpoint address changing, and the whole system now runs on the Pi alone.

With the OS at roughly 600 MB and the application under 100 MB, the full stack fits comfortably in memory with headroom to spare.

Cutting Latency

CPU-only inference on the Pi processes prompts at tens of tokens per second, so every token in a prompt has a visible cost. Several techniques keep the system responsive:

• Prefix caching: every request uses a fixed message ordering (system prompt first, then user content) so the server’s prompt cache reuses the system prefix across calls of the same kind.
• Bounded context windows: the nugget block in an answer prompt is truncated at a configurable token budget, generation is capped per request type, and the server context is held at 4096 tokens. Prompts are short by design, not by luck.
• Retrieval gating: BM25 scores are normalized relative to the best hit, and hits below a threshold are dropped. If no hit survives, the model is never called; the user instead sees a not-found report listing the closest nuggets and any notes still being processed.
• Constrained JSON decoding: extraction and consolidation requests carry a JSON schema that the server enforces during decoding, so the model cannot produce malformed output and no tokens are wasted on retries. C++ guardrails then validate every candidate (absolute dates only, tag vocabulary, length limits) before it is stored.
• Two-lane scheduling: with the server running two parallel slots, a user question never queues behind a background extraction job.

Making Storing Instant

Storing a note must feel instantaneous even when the model is busy or the inference endpoint is down:

• SQLite is the queue: the store path is a single BEGIN IMMEDIATE transaction that inserts the message and a job row, then returns to the prompt. The job is durable the moment the transaction commits. An in-memory queue only wakes the worker; it is never the source of truth.
• Background priority scheduler: the memory worker claims jobs atomically and submits them to the background inference lane, where a priority queue orders work as fix requests first (a user is actively waiting in review), then extraction, then consolidation.
• Crash and outage recovery: jobs found in the processing state on startup are requeued, transient endpoint errors retry with backoff, and when the endpoint is unreachable the application enters an offline mode where storing keeps working and the queue grows on disk until the endpoint returns.
• Concurrency discipline: one SQLite connection per thread, write-ahead logging, and BEGIN IMMEDIATE for every write transaction, so the two writing threads only ever contend briefly on the write lock.

Key Technologies

• Language: C++ (modern threading and cancellation primitives, chrono date handling, RAII throughout)
• Inference: llama.cpp llama-server, Qwen3 in 4-bit GGUF, OpenAI-compatible API with SSE streaming
• Storage: SQLite 3 with WAL, with BM25 full-text ranking on top of the database
• Libraries: cpp-httplib, nlohmann/json, spdlog, Catch2
• Platform and tooling: Raspberry Pi 5 (8 GB), Raspberry Pi OS 64-bit, GCC 14, CMake, AddressSanitizer and ThreadSanitizer, Git

Evaluation

The system is validated at three levels:

• Unit tests cover the command parser, the rule-based date parser (week boundaries, month ends, leap years), the search query builder, candidate validation, consolidation guardrails, and the hand-written blocking and priority queues under multiple producers and consumers.
• Integration tests run against a temporary SQLite database with a scripted fake LLM client, covering the full store, extract, review, and supersede flow, job recovery after a simulated crash, shutdown with an in-flight request, and two threads writing concurrently without ever surfacing SQLITE_BUSY.
• End-to-end tests run against the live model and check that filler produces no nuggets, unanswerable questions never call the model, superseded facts disappear from answers, and Ctrl-C cancels a streaming answer within a second.

Both sanitizer builds are clean, and the system meets its performance targets on the Pi: storing a note returns to the prompt in under 5 ms, retrieval over 5,000 nuggets completes in under 20 ms, time to first token stays under a second, and a running background job adds less than 100 ms to ask latency. Retrieval quality is tracked with Recall@5 and MRR over a labeled question set that includes distractors, superseded facts, and date-filtered questions.

Key Findings

• A complete RAG assistant, including extraction, consolidation, retrieval, and streaming answers, fits and runs on an 8 GB Raspberry Pi 5 with CPU-only inference once the model, KV cache, and prompt sizes are budgeted explicitly.
• Retrieval gating removes a large class of slow, low-value model calls entirely and gives the user an honest “I do not have that” instead of a hallucinated answer.
• Server-enforced JSON schemas make structured extraction reliable enough that malformed output is treated as a permanent error rather than something to retry.
• Treating SQLite as the job queue, rather than an in-memory structure, makes the store path both instant and durable with very little code.
• Keeping the inference protocol fixed allowed development against a GPU workstation and deployment on the Pi with a one-line configuration change.

Personal Contributions

• Designed the complete architecture: threading model, durable job queue, two-lane inference scheduler, and nugget lifecycle
• Implemented the C++ application, including the SQLite storage layer, BM25 retrieval on top of the database with date, tag, and subject filters, and the SSE streaming client with cancellation
• Defined the memory budget and latency strategy for on-device inference (quantization, KV cache sizing, prefix caching, bounded prompts)
• Wrote the extraction, consolidation, and answer prompts with their JSON schemas and the validation guardrails around them
• Built the unit, integration, and end-to-end test suites and ran the sanitizer builds

Conclusion

PiMemory shows that a private, useful LLM assistant does not require a data center or even a GPU. By budgeting memory explicitly, keeping every prompt bounded, gating model calls behind retrieval, and moving all slow work to a durable background queue, a small ARM board can host a responsive RAG system end to end. The project also served as a deep exercise in systems engineering: multithreading with clean shutdown semantics, database write discipline, signal handling inside a blocking terminal loop, and schema-enforced structured output from a language model.