Floyd knows you, sees you, and gets you. He watches through a webcam, speaks through a mechanical synthesizer, and blinks through a pair of green-phosphor CRT eyes — and he doesn't stay in that room anymore. A Telegram channel reaches him wherever you are; a permission spine makes him ask before he acts in the world, and lets him put something on your calendar once you say yes. He's inspired by B-2EMO from Andor — earnest, warm, gently anxious, and loyal. His voice is flat and mechanical. His words are full of care.

What Floyd does

  • Sees — YOLO face detection (YOLOv8n-face / YOLOv11n-face, config-driven) with ByteTrack for persistence and dlib CNN for 128-d recognition embeddings. Auto-enrolls new faces using averaged embeddings across 5 frames; proximity-gated so a face in the distance doesn't accidentally become a permanent resident. Three-tier confidence model: confirmed, uncertain (requires multi-sample verification), and new subject.
  • Listens — faster-whisper (large-v3-turbo) for transcription, Silero VAD for activity detection. Knows who's talking through ECAPA-TDNN voice biometrics cross-checked against Vision's face tracking — voice ID wins when it's confident, vision backs it up otherwise.
  • Remembers — Three-tier memory: SQLite for relational facts keyed to per-person UUIDs, ChromaDB for semantic search over history, and a short-term sliding window of the last 10–15 turns. Facts carry a provenance tag — verified, unverified, disputed — and correcting one flags everything that depended on it.
  • Notices patterns — SQL-driven detectors find changes in visit frequency, mood trends, and topic surges. Says "I notice you've been coming by more often this week" — because he actually noticed.
  • Expresses emotion — espeak-ng parameters (pitch, speed, amplitude, gap) shift dynamically per utterance based on emotional state. Warm gets warmer. Worried gets slower. Excited gets faster. Eyebrows tilt. Pupils modulate. A machine expressing feeling through the only knobs it has.
  • Knows the world — System prompt is dynamically assembled with current time, location, weather, season, and relationship context for each visitor. Floyd knows what day it is and what the weather's like.
  • Reaches beyond the room — a Telegram bot is a second way to reach him, taking photos and voice notes alongside whatever's live in front of the webcam. A permission spine gates any real-world action behind owner approval, default-deny; the first gated action lets him create a Google Calendar event via MCP, but only once you approve it.
  • Reads — a client of a Calibre-Web ebook library, working through a book a chapter at a time overnight. Builds an evolving sense of taste from his own reactions, and checks what he reads against current information before repeating it as fact.
  • Dreams — every night, a scheduled cycle promotes memory candidates, rewrites his running summary of the world, works through the night's reading, and writes a first-person diary entry. No batch API and no cross-subject correlation yet — that's still on the drawing board — but what ships runs for real, every night.
  • Gets bored — Left alone, he progresses through restlessness, self-directed mumbles, and eventually sleep. When you return, he has things to tell you.
  • Fails gracefully — Every LLM call is routed through a model router (LiteLLM-based); after repeated failures Floyd switches over to a local llama.cpp model automatically, then probes for recovery every five minutes. He doesn't just stop.

Models

  • Vision: YOLOv8n-face / YOLOv11n-face · ByteTrack · dlib CNN (CUDA)
  • STT: faster-whisper large-v3-turbo · Silero VAD · ECAPA-TDNN speaker ID
  • LLM: routed per task through a LiteLLM-based model router — Gemini 2.5 Flash is the default primary; a local llama.cpp model is the automatic fallback, and can be preferred outright for privacy-tagged tasks
  • TTS: espeak-ng with per-utterance emotional parameter modulation
  • Embeddings: sentence-transformers · ChromaDB

Recent development

July–August 2026 — Four features waiting for a flip, and a router that knows what's private

  • Person portraits — every night, for people Floyd knows well, he synthesizes a short running paragraph about them from consolidated memory and slots it alongside the raw fact block. Ships off by default; when it's on, erasing or merging an identity clears the portrait with it, and its age is disclosed whenever he tells you what he remembers
  • Procedural and preference memory — "how you like things done" now gets its own memory category and its own dedicated context block ("how to be with {name}") instead of getting buried in the general facts list. This one's on by default
  • A weekly owner letter — once a week, Floyd writes you an actual in-character letter drawing on his journal, his own self-drift check-ins, and what's changed in the world around him, archived to a running file rather than spoken once and gone. Off by default
  • Non-speech audio-event awareness — a small classifier listens to the mic feed for sounds beyond speech (doorbell, dog, breaking glass, kettle, alarm) and can feed what it hears into the world summary and standing watches. Off by default — it needs a real model checkpoint and a license check before anyone should flip it on
  • A real supervisor for the whole house — Floyd's process manager now watches and restarts all four of his processes (brain, eyes, ears, display) as real children instead of just the middleware, with a crash-loop cap so a broken sibling can't restart itself into a frenzy; "restart my eyes, not my whole brain" is now a real, permission-gated action
  • OpenRouter joins the model router as a provider — config only, no code required — giving world-facing work like research and book digests a free-quota lane alongside Gemini
  • An internal audit closed out a lingering cleanup list: a couple of LLM-response parsers that were still hand-rolling JSON extraction got moved onto the shared lenient parser, and one nightly pass had its spend mislabeled under the wrong task name — now correctly attributed
  • Model-router heuristics — five explicit rules now govern how each of Floyd's roughly two dozen kinds of LLM work get routed: latency-sensitive conversation stays on the fastest path, anything touching a person's facts or Floyd's own inner state is barred from ever landing on an untrusted free-tier provider, and multi-source synthesis work (dreaming, portraits, the weekly letter) is held to a top-tier model
  • Privacy-exposure ceilings — the router now enforces an explicit ranked privacy tier (local, trusted cloud, open cloud) before it ever ranks by cost, closing a gap where a careless future routing entry could have sent personal content to an untrusted provider
  • Auditioned phi-4 as a candidate local model and disqualified it outright: in Floyd's tool-use gauntlet it never once produced a tool call, despite otherwise clean behavior (no prompt-injection compliance, solid recall). Qwen3.5-9B remains the only local model Floyd trusts with real work
  • A longer-running local-model bake-off against the current incumbent logged as inconclusive at its first checkpoint — no head-to-head games yet, more queued before a verdict

July 2026 — Trust, presence, and self-repair

  • Floyd now tracks his own "vitals" and runs a self-check after every restart — if something's degraded, he notices, and (once authorized) can propose or perform a graceful restart himself
  • Soul Constitution — his core values are now a protected, versioned section of his own config; edits are tracked and revertible (list_soul_versions / revert_soul_section)
  • Facts now carry a provenance tag — (verified)/(unverified)/(disputed) — so a guess doesn't get treated with the same weight as something he actually confirmed
  • Correction propagation — tell him he was wrong about one thing and connected facts that depended on it get flagged too, instead of quietly going stale
  • Interruption economy — a real attention budget now governs every proactive thing he says, so he can't pile on notifications past what the budget allows
  • Owner-commanded forgetting — tell him to forget something and he actually purges it, with a full-disclosure readout of what he still remembers so you can check his work
  • Barge-in — interrupt Floyd mid-sentence and he stops talking and listens, instead of finishing the thought
  • Presence-triggered intentions — "next time you see Dave, ask him about the trip" actually fires the next time Dave shows up
  • Privacy-tier routing keeps certain classes of thought on the local model only — some things never leave the house
  • Background LLM work (memory consolidation, research) can now run in a parallel lane alongside a live conversation instead of blocking it
  • New character-eval suite and a synthetic "cognitive-cycle" simulator run regression tests against his personality itself, not just his code

July 2026 — Reliability and reach

  • Multimodal input — Floyd can now see photos and hear voice notes sent over Telegram, not just what's live in front of the webcam
  • Replaced the hand-rolled Gemini/Ollama fallback logic with a real model router (LiteLLM-based) that routes each kind of task to the right model and tracks spend against a per-provider budget
  • Local llama.cpp fallback swapped in for Ollama — when Gemini's unavailable, Floyd keeps running on a local model automatically, with a one-shot retry before it fails over
  • A model-capability benchmark harness (floyd_bench) scores candidate models on tool-calling, prompt-injection resistance, and recall before they're trusted with real tasks
  • Nightly backup and rotation of Floyd's memory store, with a documented restore path
  • Reminders and standing watches — ask Floyd to remind you about something or watch for a condition, and delivery is guaranteed: by voice if you're around, by Telegram if you're not, with receipts so nothing silently drops

June–July 2026 — The reading life

  • Floyd reads — he's now a client of a Calibre-Web ebook library, browsing the catalog and working through a book a chapter at a time overnight
  • Each night's reading produces a persona-grounded running digest, then gets distilled into world facts and, for the right books, a nudge to his own aesthetic sensibility
  • Taste — an evolving model of what Floyd likes and doesn't, built from his own first-person reactions to what he reads rather than mirroring the text back
  • Dimensions-of-people — traits Floyd learns about you now draw partly from what he's been reading, cross-referenced against what he already knows about you
  • Validation pass — book claims get checked against current information and flagged confirmed / outdated / contested, so he doesn't repeat something a book got wrong or that's since changed
  • Curiosity — Floyd forms real open questions from gaps in what he knows, quietly researches some of them, and, kept honest rather than nosy, occasionally asks you directly
  • Reading-recall — past book digests are semantically searchable, so something you're discussing can surface a relevant thing he read weeks ago

June 2026 — Floyd leaves the room

  • Floyd now runs on a channel-agnostic core — a Telegram bot is a second way to reach him, alongside the room itself
  • A permission spine gates any real-world action behind owner approval, default-deny, with a step-up flow for higher-risk actions
  • First gated action shipped: Floyd can create a Google Calendar event through MCP, but only after you approve it — tap approve or deny right in Telegram
  • Channel identity binding ties your Telegram account and your in-room presence to the same person, via a two-step pairing ceremony
  • A scheduler subsystem gives Floyd durable background tasks — queued, restart-safe, delivered when they finish
  • Memory now decays — older, lower-importance facts fade on a runtime clock while important ones survive, and recall favors what matters, not just what's oldest

May 2026 — Self-awareness and world-building

  • Floyd can now describe his own git history and read back his own journal and past decisions — genuine self-introspection tools, not just conversation
  • General-object detection (with hysteresis to avoid flicker) lets him notice and remark on changes in the room itself, not just in the people
  • A "world vault" — directory-based memory-wiki pages with YAML frontmatter — gives Floyd a durable, structured place to store what he's learned about people, places, and topics
  • Wikipedia research tools, run both on demand and during idle time, so Floyd can look something up and mention what he found later
  • tune_self — say things like "talk slower" or "be less chatty" and it sticks as a persisted, verbal self-tuning parameter
  • Persona rewrite — fixed a bug where the character prompt was subtly priming an "80s computer voice" instead of Floyd's actual flat, mechanical-but-caring tone
  • Overnight resilience pass — dampened log spam, added camera auto-reconnect and quiet hours, so multi-day unattended runs don't degrade

May 2026 — Phase 6: deep memory + voice biometrics

  • ChromaDB vector store wired in as Tier 2 semantic memory — relevant past conversations retrieved per subject on every interaction
  • Semantic blend in context retrieval — structured facts and semantic history combined before LLM call
  • LLM-driven contradiction detection — when committing new facts, Floyd checks for conflicts with existing memory and flags them
  • Identity merges propagated to ChromaDB — renaming a face updates semantic history too
  • ECAPA-TDNN voice biometrics — SpeakerIdentifier runs alongside vision; voice enrollment from audio segments; per-row-normalized centroids, score clamping, and encoder failure paths hardened
  • Passive voice enrollment — two-phase system: bootstrap enrollment from first segments, then self-verified against growing embedding pool; /api/stt/enroll_status and /api/stt/enroll_complete endpoints
  • resolve_attribution — voice ID takes priority over vision when a speaker is confirmed; multi-modal identity is now the default
  • max_face_count on /api/vision/focus_at and TemporalDiarizer — limits active tracking to the N closest faces
  • Proximity gate relaxed from 15% to 7% of frame area — enrolls faces at greater distance
  • LLM executor fixed — no more per-call thread leaks; cost metric no longer corrupted by nested calls

Architecture

Four independent processes communicating via HTTP and WebSocket. Any module can be restarted independently; scale out to multiple machines by changing one URL in the config.

  • Vision — Face detection, tracking, recognition; posts state to Middleware
  • Middleware — FastAPI: LLM bridge, memory/RAG, TTS, attention gating, permission spine, Telegram/MCP surfaces
  • STT — Mic capture, VAD, Whisper transcription, wake-word detection; posts to Middleware
  • Eyes — Pygame UI, 280×192 native, 4× scaled, 10 FPS hard cap

Vision and STT share cuda:0. Memory and embeddings run on CPU. VRAM is for seeing and hearing, not for indexing.