keep/PLAN.md

Development Plan for Keep

Overview

Keep is a command-line tool for managing temporary files with automatic compression, metadata generation, and querying capabilities. It includes a REST API server and supports extensible plugins for compression, metadata, and filtering.

Current Status (as of latest updates)

The project is in a functional state with core features implemented. Key components:

Core Features

• Save Mode: Stores content with compression and metadata generation. Supports tags and custom metadata via environment variables.
• Get Mode: Retrieves item content by ID or tags, with optional filtering (head, tail, grep, etc.) and binary detection.
• List Mode: Lists items with customizable table format, sorting, and filtering by tags/metadata.
• Delete Mode: Deletes items by ID.
• Info Mode: Shows detailed information about an item including metadata.
• Diff Mode: Compares two items using the diff utility.
• Status Mode: Displays system status including paths, compression engines, and configured plugins.
• Status Plugins Mode: Shows detailed plugin configurations and available options.

Compression Engines

• Built-in: LZ4, GZip, None.
• External programs: BZip2, XZ, ZStd (via CompressionEngineProgram).
• Factory system for selecting engines with fallback to default (LZ4 if available).

Meta Plugins

• Text: Detects binary/text content, counts lines/words, tracks line length statistics (max, mean, median).
• Cwd: Captures current working directory.
• User: Captures user ID, group ID, username, group name.
• Shell: Captures shell environment.
• ShellPid/KeepPid: Captures parent and current process IDs.
• Digest: Computes MD5, SHA256, SHA512 hashes (configurable).
• MagicFile: Detects MIME type, encoding, and file type using libmagic.
• Hostname: Captures short/full hostname with DNS resolution.
• Exec: Runs external programs for custom metadata.
• Env: Captures environment variables prefixed with KEEP_META_.
• Plugin registry and configuration via YAML (options/outputs mapping).

Filter Plugins

• Head/Tail/Skip: Bytes and lines (head_bytes(10), tail_lines(5), etc.).
• Grep: Regex pattern matching on lines.
• StripAnsi: Removes ANSI escape sequences.
• Chainable via filter strings (e.g., "head_lines(10)|grep(pattern=error)").

Server Mode

• REST API with endpoints: /api/status, /api/item/ (list/post), /api/item/{id}/meta, /api/item/{id}/content, /api/item/latest/*.
• Authentication: Bearer token or Basic auth (password or hashed).
• MCP (Model Context Protocol) support via SSE and JSON-RPC tools (save_item, get_item, list_items, etc.).
• OpenAPI/Swagger documentation.
• HTML pages for item listing and details.
• Streaming support for large content with offset/length parameters.
• Binary content detection and restriction.

Configuration

• YAML config file (~/.config/keep/config.yml) with overrides from env vars and CLI.
• Table customization: columns, alignment, colors, styles (ASCII/UTF8).
• Plugin configuration: enable/disable outputs, custom options.

Database

• SQLite with schema for items, tags, metadata.
• Migrations for schema evolution.
• Indexes on tags and metadata names.
• Foreign key constraints.

Testing

• Unit tests for compression engines, meta plugins, filter plugins, database operations.
• Integration tests for modes (save, get, list, etc.).
• Test helpers for temp files/DBs.

Other

• Binary detection using signatures and printable character ratio.
• Human-readable size formatting.
• Logging with configurable verbosity.
• Error handling with structured errors (CoreError).

Completed Changes

• Implemented full plugin system for meta and filters with registry via ctors.
• Added async item service for server mode with blocking task spawning.
• Enhanced server API with streaming, pagination, and MCP integration.
• Improved configuration with table styling and plugin output mapping.
• Added binary detection in get mode and API.
• Implemented diff mode with pipes and compression handling.
• Created status and status-plugins modes with detailed output.
• Added generate-config mode with commented defaults.
• Integrated libmagic for file type detection.
• Enhanced text meta plugin with line/word statistics.
• Added exec meta plugin for external commands.
• Implemented filter parsing with pest grammar.
• Added HTML UI for server with item listing and details.

Remaining/Planned Tasks

• Documentation: Update README.md with installation, usage examples, API docs.
• More Plugins: Add file size, creation time plugins if needed.
• Performance: Optimize streaming for very large files; add content caching.
• Security: Add rate limiting to API; validate input more strictly.
• Features:
  • Update mode for modifying existing items.
  • Search mode with full-text search on metadata/content.
  • Export/import database.
  • Backup/restore functionality.
• Testing: Add more integration tests for server API and MCP.
• Polish:
  • Improve error messages and user feedback.
  • Add progress indicators for large operations.
  • Support for custom compression plugins.
• Packaging: Create installation scripts, Docker support.

Potential Improvements

• Switch to a more efficient database for large datasets (e.g., SQLite with WAL mode).
• Add WebSocket support for real-time updates in server mode.
• Implement plugin hot-reloading for development.
• Add CLI completion (bash/zsh/fish).
• Support for archiving old items automatically.

Last updated: After codebase initialization and feature implementation.