Visual Memory MCP

Ingest screenshot(s) via base64 or file path, check the visual state cache, and return state details. On cache miss, generates perceptual hashes (dHash/aHash), CLIP embeddings, and persists a new visual state entry. Use this tool AFTER capturing a screenshot and BEFORE invoking vision LLMs to avoid redundant token spend. Use analyze_screenshot instead of recall_memory when ingesting, indexing, and caching a visual capture rather than performing read-only search over historical states. For batch ingestion, pass an `items` array instead of a single screenshot/file_path. Returns visual state ID, layout description, perceptual hashes, cache hit status (is_known), and grounded interactive elements.

Compare two visual states structurally (dHash, CLIP vector, JSON diff, layout delta) or two video trajectories (keyframe similarity, divergence point). Provide state_a_id/state_b_id for state comparison, or video_a_id/video_b_id for video trajectory comparison. Use compare_states instead of manage_snapshot when computing direct visual/structural deltas between two states rather than saving or restoring checkpoints. Returns Hamming hash distance, cosine vector similarity, structural layout diff, and layout change flag.

Package keyframe IDs, dHash/CLIP fingerprints, OCR snippets, and linked state-memory node IDs into an immutable, cryptographically hashable evidence pack for compliance and audit trails. Use create_evidence_pack instead of export_trajectories when generating tamper-proof cryptographic audit proofs for compliance rather than dumping training trajectories. Returns cryptographic evidence pack with SHA-256 payload hash, keyframe fingerprints, and state-memory links.

Export visual state transition trajectories (formats: json, llava, qwen2_vl, joint). Formats: "json" (raw state steps), "llava" / "qwen2_vl" (VLM fine-tuning datasets), "joint" (interleaved workflow events correlated by session/trace ID for state-memory-mcp). Use export_trajectories instead of create_evidence_pack when exporting training data or cross-server event logs rather than cryptographic compliance packs. Returns exported trajectory array, formatted VLM training records, or joint event stream.

Purge a specific visual state, its vector embeddings, and perceptual hashes from storage for privacy or memory reset. Use forget_state instead of undo_visual_mutation when permanently purging specific sensitive or obsolete states by ID rather than reverting the most recent action. Returns purge confirmation and deleted state ID.

Trace historical pathways from current state to a target state or state matching description via BFS over the transition graph. Use get_navigation_paths instead of predict_next_action when planning multi-step navigation routes across known states rather than choosing the immediate next interaction. Returns shortest path state sequence, intermediate actions, and cumulative transition confidence.

Fetch aggregated visual context: recent states, frequently accessed states, active transitions, and memory stats. Call this at session start to orient before performing UI actions. Use get_session_context instead of recall_memory when initializing session awareness rather than searching for specific visual targets. Returns recent states list, frequent states, active transitions, cache hit telemetry, and database sizing.

Manage visual memory snapshots (actions: save, diff, export, restore). "save" creates a named checkpoint of current states; "diff" compares two snapshots for visual drift/regression; "export" serializes a snapshot to a portable JSON archive; "restore" imports a snapshot from an exported archive. Use manage_snapshot instead of undo_visual_mutation when creating or restoring named point-in-time visual checkpoints rather than rolling back the immediate previous mutation. Returns snapshot metadata, comparison diff, export archive, or restore confirmation.

Manage WebM, MP4, and GIF video recordings in visual memory (actions: ingest, search, timeline). "ingest" extracts keyframes, deduplicates, and builds sequence transitions; "search" queries stored video recordings by keyword/category/tags; "timeline" retrieves the chronological keyframe timeline for a specific video_id. Use manage_video instead of analyze_screenshot when processing full video recordings and extracting temporal keyframes rather than single screenshots. Returns ingested video metadata, keyframe indices, search matches, or timeline sequence.

Manage Visual Spec baseline contracts for visual SDD (actions: set, verify, list). "set" registers a screenshot or design mockup as baseline; "verify" tests a live UI screenshot against a baseline; "list" returns all registered visual specs. Use manage_visual_spec instead of compare_states when verifying UI against approved design contracts rather than ad-hoc state comparisons. Returns baseline registration confirmation, visual verification score, or registered spec list.

Predict the best next UI action from the current visual state based on transition success rates, goal alignment, and grounded AX tree element targeting. Use predict_next_action instead of get_navigation_paths when selecting the immediate next optimal UI interaction rather than computing full path routes. Returns recommended action, target element selector, coordinates, and confidence score.

Search visual state memory by screenshot image, text query, or accessibility tree. Read-only — never creates or modifies states. Use recall_memory instead of analyze_screenshot when searching historical states by visual similarity or semantic text query without persisting new database records. Returns matching visual state records, similarity scores, layout descriptions, and grounded elements.

Log an action transition between two visual states or log a visual blocker (action_type: click, type, navigate, scroll, custom, blocker). Call this after every UI interaction to update transition statistics and success rates. When action_type is "blocker", generates a structured visual blocker payload for state-memory-mcp instead of recording a transition (from_state_id becomes the visual_state_id of the blocker). Use record_outcome instead of predict_next_action when persisting actual interaction results rather than predicting recommended next steps. Returns transition record, updated edge weights, or state-memory blocker payload.

Revert the most recent visual state ingestion or transition edge addition on the current git branch (types: state, transition, any). Use undo_visual_mutation instead of manage_snapshot when rolling back the immediate previous mutation rather than restoring an entire snapshot. Returns reverted mutation type, deleted state/edge ID, and rollback status.

Poll for a target visual state ID until it exists in memory or timeout occurs. Use this to avoid agent spin-loops when waiting for asynchronous screenshots. Use wait_for_visual_state instead of get_session_context when synchronizing with an asynchronous capture rather than inspecting existing memory state. Returns target visual state record or timeout error.