\documentclass[11pt]{article}

\usepackage[margin=1in]{geometry}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{longtable}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{microtype}

\hypersetup{
  colorlinks=true,
  linkcolor=blue,
  urlcolor=blue,
  pdftitle={VideoMemory Technical Report},
  pdfauthor={Mark Ogata}
}

\setlist[itemize]{leftmargin=1.5em}
\setlist[enumerate]{leftmargin=1.5em}
\setlength{\parskip}{0.6em}
\setlength{\parindent}{0pt}
\emergencystretch=3em

\title{VideoMemory Technical Report}
\author{Mark Ogata}
\date{May 15, 2026}

\begin{document}
\maketitle

\begin{abstract}
VideoMemory is an agent-agnostic video ingestion and monitoring layer. Its goal
is to let any capable agent consume multiple live video streams without
implementing camera discovery, stream management, local filtering, model
selection, evidence capture, or hardware-specific recovery logic. This report
summarizes the current system as raw material for a future master's thesis. The
central framing is cost-aware, limited-hardware video monitoring for agents:
many streams produce many frames, most frames are uninteresting, and naive
vision-language-model inference on every frame is too expensive and too slow.
VideoMemory addresses this through a core HTTP service, task-scoped natural
language monitors, readiness diagnostics, binary monitors, semantic filtering,
usage accounting, evidence capture, webhook delivery, and concrete adapters
such as Claude Code and OpenClaw.
\end{abstract}

\section{Executive Summary}

VideoMemory is motivated by a practical limitation in current agent systems.
Agents can call tools, read files, inspect screenshots, and reason over a
single image, but they do not naturally watch the world over time. A normal
request-response vision call can answer ``what is in this image?'' It does not
solve ``watch these streams and tell my agent when a condition becomes true.''
VideoMemory fills that gap by turning video streams into task-scoped,
evidence-backed events.

The project should not be understood primarily as a Claude plugin or a single
camera demo. Claude Code is one integration surface. The deeper system is an
agent-agnostic video ingestion and eventing layer. Any agent should be able to
register or discover streams, create natural-language visual tasks, receive
events, inspect readiness, and retrieve the evidence associated with a trigger.

The hard constraint is cost and hardware. A system that sends every frame from
every stream to a cloud VLM is not viable for large numbers of streams, limited
local compute, or low-power hardware such as Raspberry Pi. VideoMemory therefore
needs cheap local stages before expensive reasoning: stream sampling, stale
frame dropping, binary monitors, semantic filters, usage accounting, and
structured readiness diagnostics.

\section{System Goal}

The intended system shape is:

\begin{verbatim}
many video streams
  -> VideoMemory stream ingestion
  -> local sampling, filtering, readiness checks, and buffering
  -> selective binary or VLM evaluation
  -> task state and aligned evidence
  -> generic event delivery
  -> any external agent decides what to do next
\end{verbatim}

The key abstraction is a natural-language monitoring task attached to a video
source. The agent states the visual condition. VideoMemory manages the stream
and produces state changes or events when that condition is observed.

The system should be evaluated by whether it makes video streams
agent-addressable in a practical way. That means supporting more than one
camera, avoiding naive model-per-frame processing, exposing useful readiness
diagnostics, and producing event evidence that an external agent can trust.

\section{Design Principles}

\subsection{Agent Agnostic By Default}

VideoMemory core should not be tied to one agent runtime. The core interface is
HTTP plus event delivery. Claude Code, OpenClaw, and future agents should be
adapters over the same task and event contract.

This distinction matters for the future thesis. The reusable contribution is
not a Claude-specific plugin. The reusable contribution is stream ingestion,
filtering, monitoring, readiness, evidence, and event delivery in a form that
any agent can consume.

\subsection{Perception And Action Are Separate}

VideoMemory owns the visual condition. The external agent owns the follow-up
action. In a request like ``tell me when I hold my phone up,'' VideoMemory
stores and monitors the condition ``a phone is visibly held up.'' The agent
stores the instruction ``tell the user.'' This keeps VideoMemory's correctness
boundary focused on device state, frame state, model results, task state,
evidence, and event delivery.

\subsection{Cost And Hardware Constraints Are First-Class}

Multiple streams make naive VLM processing unrealistic. If every frame from
every stream is sent to a cloud model, the system will hit cost, latency, and
throughput limits. VideoMemory therefore needs:

\begin{itemize}
  \item stream sampling rather than processing every frame;
  \item stale and redundant frame dropping;
  \item binary monitors for simple conditions;
  \item semantic filters before richer VLM calls;
  \item usage accounting for latency and estimated cost;
  \item low-power deployment paths such as Raspberry Pi.
\end{itemize}

\subsection{Readiness Is Not Task Creation}

A task row existing in the database does not prove that monitoring is working.
A monitor also needs a registered device, a running ingestor, fresh frames,
camera permission, model readiness where required, and an event path if the
agent expects a wakeup.

VideoMemory therefore exposes device readiness separately from task creation.
The readiness payload reports whether the device exists, whether an ingestor is
running, whether a frame is available, frame age, browser camera freshness,
binary monitor status, semantic filter status, and warnings.

\subsection{Evidence Must Match The Event}

If a task note says that a visual condition occurred, the saved frame and video
clip must correspond to the same event. A later live snapshot is not adequate
evidence. This became concrete in a bug where saved still images matched the
model input frame but saved videos were built later from the live rolling
buffer, causing drift when VLM callbacks were slow. The fix was to snapshot the
evidence buffer when a chunk is queued and build note videos from that queued
snapshot.

\section{Implemented Architecture}

\subsection{Core Service}

The core service is a Flask application in:

\begin{verbatim}
flask_app/app.py
\end{verbatim}

It serves the web UI, JSON APIs, OpenAPI schema, device pages, task pages,
settings pages, ingestor debug surfaces, semantic preview routes, capture
endpoints, and readiness endpoints. The direct local startup command is:

\begin{verbatim}
uv run flask_app/app.py
\end{verbatim}

The default local service address is:

\begin{verbatim}
http://localhost:5050
\end{verbatim}

\subsection{Input Sources}

VideoMemory currently supports or documents these input modes:

\begin{itemize}
  \item USB and built-in cameras through local device discovery;
  \item RTSP and HTTP video streams as network sources;
  \item browser camera frames posted into VideoMemory through a browser bridge;
  \item Android phone snapshot streams served as \texttt{/snapshot.jpg};
  \item Raspberry Pi USB camera deployments.
\end{itemize}

This variety matters because an agent-facing video system cannot assume one
clean camera environment. Browser permission, local device locks, mobile
network reachability, and low-power Linux camera behavior are different
engineering problems.

\subsection{Task Model}

The central application object is a task. A task has an input identifier
(\texttt{io\_id}), a natural-language task description, optional agent
identifier (\texttt{bot\_id}), monitor type, semantic filter configuration,
status, notes, and optional evidence. Agents create tasks with:

\begin{verbatim}
POST /api/tasks
\end{verbatim}

The minimum body is:

\begin{verbatim}
{
  "io_id": "0",
  "task_description": "Watch for a phone visibly held up in the user's hand."
}
\end{verbatim}

Optional fields include:

\begin{itemize}
  \item \texttt{monitor\_type};
  \item \texttt{save\_note\_frames};
  \item \texttt{save\_note\_videos};
  \item \texttt{semantic\_filter\_keywords};
  \item semantic filter tuning fields.
\end{itemize}

\subsection{Monitor Types}

VideoMemory separates simple visual triggers from richer note generation.

\texttt{monitor\_type: "binary"} is intended for fast done/not-done conditions.
It is the correct shape for tasks like ``wake me when a human is visible'' or
``tell me when the dog appears.'' The documented binary monitor uses a local
true/false path, defaults to a threshold of 0.5, and requires 2 true votes out
of the last 3 evaluated frames before marking the task done.

\texttt{monitor\_type: "general"} is the chunked VLM monitor. It writes richer
task notes and is useful when the user needs description, explanation, or
context. It may require a configured cloud or local model provider.

The one-off caption endpoint:

\begin{verbatim}
POST /api/caption_frame
\end{verbatim}

is for immediate questions about the current frame, not long-running
monitoring.

\subsection{Semantic Filtering}

Semantic filtering gates frames before the richer VLM path. Task creation can
accept semantic keywords, and debug/status routes expose semantic preview,
semantic pass streams, queue size, dropped semantic frames, pass-frame age, and
latest evaluation timestamp. This is central to the cost-aware framing: local
filtering should reduce expensive model calls when most frames are irrelevant.

A previous local monitor used the \texttt{dino\_clip\_adapter} backend with
keywords \texttt{phone}, \texttt{smartphone}, \texttt{hand}, and \texttt{person},
absolute threshold mode, threshold 0.3, \texttt{max} reduction, and ensemble
off.

\subsection{Evidence Model}

VideoMemory can save frames and video clips associated with task notes. Event
payloads can include:

\begin{itemize}
  \item \texttt{note\_frame\_api\_url};
  \item \texttt{note\_video\_api\_url}.
\end{itemize}

Agents should use these saved URLs when responding to a detected event. They
should not take a fresh snapshot later and present it as evidence for the
earlier trigger.

\subsection{Readiness Diagnostics}

The endpoint:

\begin{verbatim}
GET /api/device/{io_id}/readiness
\end{verbatim}

returns structured readiness information. It reports device existence, ingestor
existence, running state, frame availability, frame age, browser camera status,
binary monitor status, semantic filter status, and warnings. Common warnings
include no fresh browser frames, no active ingestor, no captured frame,
unregistered device ID, and macOS local camera permission problems.

\subsection{Model Providers And Usage Accounting}

The repository includes provider adapters for Google, OpenAI, Anthropic,
OpenRouter, Mistral, and local vLLM-style usage. The settings surface supports
keys such as \texttt{GOOGLE\_API\_KEY}, \texttt{OPENAI\_API\_KEY},
\texttt{OPENROUTER\_API\_KEY}, \texttt{ANTHROPIC\_API\_KEY}, and
\texttt{VIDEO\_INGESTOR\_MODEL}.

Usage accounting is implemented in:

\begin{verbatim}
videomemory/system/usage.py
\end{verbatim}

It normalizes model names and tracks input tokens, output tokens, total tokens,
latency, success status, and estimated cost. The pricing table includes Gemini,
OpenAI, Anthropic, OpenRouter/free models, and \texttt{local-vllm} at zero
external API cost.

\section{Agent Integration}

\subsection{Generic Contract}

The generic contract is documented in:

\begin{verbatim}
docs/agent-integration-contract.md
\end{verbatim}

VideoMemory core owns device discovery, stream registration, task lifecycle,
video ingestion, VLM-based detection, and task notes. The external agent owns
conversation, orchestration, policy, memory, authentication, authorization,
tool planning, and follow-up actions.

The stable API includes health checks, devices, captures, preview, readiness,
network device registration, tasks, settings, OpenAPI schema, and skill
documents. This makes VideoMemory a service agents call, not an agent runtime
itself.

\subsection{Event Delivery}

For ``when X happens, do Y'' workflows, X belongs in VideoMemory and Y belongs
in the external agent. Event delivery can happen through a generic webhook
receiver, OpenClaw-compatible hooks, or a Claude Code channel.

The generic webhook settings include:

\begin{verbatim}
VIDEOMEMORY_OPENCLAW_WEBHOOK_URL
VIDEOMEMORY_OPENCLAW_WEBHOOK_TOKEN
VIDEOMEMORY_SELF_BASE_URL
\end{verbatim}

The setting name still contains \texttt{OPENCLAW} because it originated in that
integration, but the value can point at any compatible receiver.

\subsection{Claude Code Adapter}

The Claude Code integration proves push-style wakeups into a running agent
session. Its event flow is:

\begin{verbatim}
VideoMemory monitor task
  -> task note / detection
  -> POST http://127.0.0.1:8791/videomemory-event
  -> Claude Code channel
  -> running Claude Code session receives a videomemory channel event
\end{verbatim}

The public install path is:

\begin{verbatim}
claude auth login
claude plugin marketplace add https://github.com/Clamepending/videomemory
claude plugin install videomemory@videomemory
\end{verbatim}

Claude is a strong adapter and demo surface, but the future thesis should
present it as one adapter over the general VideoMemory service boundary.

\subsection{OpenClaw And Package Distribution}

The package under \texttt{videomemory-package/} is published as
\texttt{@clamepending/videomemory} and currently declares version 0.1.9. It
provides CLI binaries named \texttt{videomemory} and
\texttt{videomemory-openclaw}, bundled scripts, hooks, skills, and plugin
metadata.

\section{Concrete Engineering Milestones}

\subsection{Core HTTP Service And API}

The core service exposes health, device discovery, capture, preview, caption,
task lifecycle, readiness, settings, and OpenAPI endpoints. It is implemented
in the Flask app and documented in the README and agent integration contract.

\subsection{Browser Camera Bridge}

The browser camera bridge handles local camera permission and browser-based
frame posting. It is used by the Claude flow through:

\begin{verbatim}
http://127.0.0.1:5050/browser-camera/facetime
\end{verbatim}

Readiness diagnostics include browser camera freshness, and the README notes
that the camera tab must remain open while the monitor is active.

\subsection{Android Snapshot Stream}

The Android app runs a small HTTP server on the phone and serves the latest
camera frame as a JPEG from \texttt{/snapshot.jpg}. It uses CameraX, requires
Android 7+ and camera permission, and needs the phone and VideoMemory host to
be mutually reachable on a LAN or through Tailscale.

\subsection{Binary Monitor}

The binary monitor is the primary local path for simple visual triggers. It is
important because it avoids requiring a cloud model key for simple wakeup
conditions and supports the limited-hardware/cost-aware project framing.

\subsection{Semantic Filtering}

Semantic filtering provides a local triage layer before rich VLM calls. It is
configured by task keywords and debugged through semantic preview/status
routes. It is implemented but still needs systematic multi-stream evaluation.

\subsection{Evidence Alignment Fix}

The evidence alignment bug was fixed and pushed to \texttt{main} as commit
\texttt{57eb342}. The root cause was time drift between the frame used for the
model input and the rolling buffer used later to build the note video. The fix
snapshotted the evidence buffer when the chunk was queued.

Key files were:

\begin{verbatim}
videomemory/system/stream_ingestors/evidence.py
videomemory/system/stream_ingestors/video_stream_ingestor.py
tests/test_video_stream_ingestor_detection_callbacks.py
\end{verbatim}

\subsection{Local Status And Phone-Held-Up Monitor}

A local operational check used \texttt{http://127.0.0.1:5050} as the base URL
and observed devices including FaceTime HD Camera, OBS Virtual Camera, and
Browser FaceTime Camera. A phone-held-up monitor used semantic keywords for
phone, smartphone, hand, and person. The monitor remaining active with no task
note meant the event had not yet fired. This is a concrete example of why task
setup is not the same as successful detection.

\subsection{Raspberry Pi Camera Contention Fix}

On a Raspberry Pi deployment, \texttt{/dev/video0} existed and standalone
OpenCV could read frames once the Flask app was stopped. The failure was not
missing hardware. The issue was contention: preview/capture tried to open the
USB camera independently instead of reusing the active ingestor. The durable
fix was to reuse a shared ingestor lease and keep the local ingestor warm for
preview and capture.

Successful smoke checks were:

\begin{verbatim}
GET /api/device/0/preview
POST /api/device/0/capture?format=json
\end{verbatim}

The JSON capture response should report one of the shared-ingestor sources:

\begin{verbatim}
source: "ingestor_live"
source: "shared_ingestor_warm"
\end{verbatim}

\section{Current Test Coverage}

The test suite covers provider adapters, devices, caption frame, task creation,
task lifecycle, settings reload, usage API, readiness, macOS device detection,
local camera preview, network camera keepalive, task note frames and videos,
OpenClaw integration, plugin scaffolding, alert transforms, semantic filtering,
snapshot integration, update checks, and URL utilities.

The evidence alignment work used:

\begin{verbatim}
uv run python -m unittest
\end{verbatim}

This is recorded because \texttt{pytest} was not installed in the project
virtual environment and a mismatched fallback interpreter caused noise.

\section{Cost And Scaling Model}

The scaling problem is that video streams produce far more frames than agents
can afford to inspect with expensive models. A future evaluation should report:

\begin{itemize}
  \item number of streams;
  \item sampled frames per second per stream;
  \item frames dropped before model evaluation;
  \item semantic filter pass rate;
  \item binary monitor evaluations per minute;
  \item general VLM calls per minute;
  \item model latency distribution;
  \item estimated model cost per hour;
  \item CPU, GPU, memory, and network usage;
  \item event latency from visual occurrence to agent event;
  \item false positive and false negative rates.
\end{itemize}

Existing building blocks include model usage records, semantic frame queue
status, dropped-frame counters, readiness payloads with frame age, binary
monitor status, and saved evidence URLs. What is not yet complete is a
systematic multi-stream benchmark.

\section{Strengths And Limitations}

VideoMemory has a clear service boundary. It prevents every agent from needing
to know how to open cameras, keep streams alive, filter frames, call models, and
package evidence. It also separates one-off perception from long-running
monitoring: \texttt{/api/caption\_frame} answers immediate frame questions,
while tasks and monitors represent persistent watch conditions.

The strongest current design features are readiness diagnostics and
event-aligned evidence. Readiness prevents false claims that a monitor is armed.
Event-aligned evidence turns saved frames and videos into correctness artifacts
rather than UI conveniences.

The biggest limitation is that multi-stream scale is not yet proven by a
repeatable benchmark. The architecture supports the right direction, but the
strong thesis claim still needs measured evidence for multiple streams, limited
hardware, model-call reduction, and cost per useful event.

The agent-agnostic claim is architecturally plausible because the API and
webhook contract are generic, but the most polished onboarding currently points
at Claude Code. A generic receiver demo should be made first-class.

\section{Thesis-Relevant Interpretation}

The likely thesis contribution is:

\begin{quote}
Agents need a resource-constrained, agent-agnostic video ingestion layer that
turns many unreliable live streams into structured, evidence-backed events.
\end{quote}

VideoMemory contributes an agent-facing task abstraction, a separation between
perception state and agent action state, support for multiple stream types,
cheap local paths through binary monitors and semantic filtering, readiness
diagnostics, event-aligned evidence, generic webhook delivery, and concrete
agent adapters.

The future thesis should use this report as raw material for system design and
implementation chapters. The evaluation chapter should add quantitative
multi-stream, cost, latency, and accuracy results. The discussion chapter should
use the evidence alignment bug, Raspberry Pi camera contention bug, and
readiness/task distinction as concrete failure-driven design lessons.

\section{Recommended Next Engineering Steps}

The next work should strengthen the central claim:

\begin{enumerate}
  \item Build a repeatable multi-stream benchmark that measures sampling,
  filter pass rate, model-call rate, latency, resource usage, and cost.
  \item Add a generic webhook receiver demo so the agent-agnostic path is as
  visible as the Claude path.
  \item Produce evidence bundles containing task metadata, readiness state,
  note text, frame/video evidence, timestamps, model outputs, and cost records.
  \item Compare binary monitoring, semantic-filtered VLM monitoring, and
  general VLM notes on the same workload.
  \item Run a limited-hardware benchmark on Raspberry Pi or equivalent hardware.
\end{enumerate}

\section{Artifact Locations}

\begin{verbatim}
VideoMemory repository:
/Users/mark/Desktop/projects/videomemory

Hugo report page:
/Users/mark/Desktop/projects/markogata/content/sharing/
  videomemory-masters-thesis-artifact-outline/index.md

LaTeX source:
/Users/mark/Desktop/projects/markogata/content/sharing/
  videomemory-masters-thesis-artifact-outline/videomemory-project-report.tex
\end{verbatim}

\end{document}
