From 530615a6a1913399562de8e6d2e4544631709bc5 Mon Sep 17 00:00:00 2001 From: Andrew Phillips Date: Wed, 10 Sep 2025 09:42:19 -0300 Subject: [PATCH] docs: Initial development plan for the Keep project Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) --- PLAN.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/PLAN.md b/PLAN.md index e69de29..365d646 100644 --- a/PLAN.md +++ b/PLAN.md @@ -0,0 +1,113 @@ +# Development Plan for Keep + +## Overview +Keep is a command-line tool for managing temporary files with automatic compression, metadata generation, and querying capabilities. It includes a REST API server and supports extensible plugins for compression, metadata, and filtering. + +## Current Status (as of latest updates) +The project is in a functional state with core features implemented. Key components: + +### Core Features +- **Save Mode**: Stores content with compression and metadata generation. Supports tags and custom metadata via environment variables. +- **Get Mode**: Retrieves item content by ID or tags, with optional filtering (head, tail, grep, etc.) and binary detection. +- **List Mode**: Lists items with customizable table format, sorting, and filtering by tags/metadata. +- **Delete Mode**: Deletes items by ID. +- **Info Mode**: Shows detailed information about an item including metadata. +- **Diff Mode**: Compares two items using the `diff` utility. +- **Status Mode**: Displays system status including paths, compression engines, and configured plugins. +- **Status Plugins Mode**: Shows detailed plugin configurations and available options. + +### Compression Engines +- Built-in: LZ4, GZip, None. +- External programs: BZip2, XZ, ZStd (via `CompressionEngineProgram`). +- Factory system for selecting engines with fallback to default (LZ4 if available). + +### Meta Plugins +- **Text**: Detects binary/text content, counts lines/words, tracks line length statistics (max, mean, median). +- **Cwd**: Captures current working directory. +- **User**: Captures user ID, group ID, username, group name. +- **Shell**: Captures shell environment. +- **ShellPid/KeepPid**: Captures parent and current process IDs. +- **Digest**: Computes MD5, SHA256, SHA512 hashes (configurable). +- **MagicFile**: Detects MIME type, encoding, and file type using libmagic. +- **Hostname**: Captures short/full hostname with DNS resolution. +- **Exec**: Runs external programs for custom metadata. +- **Env**: Captures environment variables prefixed with `KEEP_META_`. +- Plugin registry and configuration via YAML (options/outputs mapping). + +### Filter Plugins +- **Head/Tail/Skip**: Bytes and lines (head_bytes(10), tail_lines(5), etc.). +- **Grep**: Regex pattern matching on lines. +- **StripAnsi**: Removes ANSI escape sequences. +- Chainable via filter strings (e.g., "head_lines(10)|grep(pattern=error)"). + +### Server Mode +- REST API with endpoints: `/api/status`, `/api/item/` (list/post), `/api/item/{id}/meta`, `/api/item/{id}/content`, `/api/item/latest/*`. +- Authentication: Bearer token or Basic auth (password or hashed). +- MCP (Model Context Protocol) support via SSE and JSON-RPC tools (save_item, get_item, list_items, etc.). +- OpenAPI/Swagger documentation. +- HTML pages for item listing and details. +- Streaming support for large content with offset/length parameters. +- Binary content detection and restriction. + +### Configuration +- YAML config file (`~/.config/keep/config.yml`) with overrides from env vars and CLI. +- Table customization: columns, alignment, colors, styles (ASCII/UTF8). +- Plugin configuration: enable/disable outputs, custom options. + +### Database +- SQLite with schema for items, tags, metadata. +- Migrations for schema evolution. +- Indexes on tags and metadata names. +- Foreign key constraints. + +### Testing +- Unit tests for compression engines, meta plugins, filter plugins, database operations. +- Integration tests for modes (save, get, list, etc.). +- Test helpers for temp files/DBs. + +### Other +- Binary detection using signatures and printable character ratio. +- Human-readable size formatting. +- Logging with configurable verbosity. +- Error handling with structured errors (`CoreError`). + +## Completed Changes +- Implemented full plugin system for meta and filters with registry via ctors. +- Added async item service for server mode with blocking task spawning. +- Enhanced server API with streaming, pagination, and MCP integration. +- Improved configuration with table styling and plugin output mapping. +- Added binary detection in get mode and API. +- Implemented diff mode with pipes and compression handling. +- Created status and status-plugins modes with detailed output. +- Added generate-config mode with commented defaults. +- Integrated libmagic for file type detection. +- Enhanced text meta plugin with line/word statistics. +- Added exec meta plugin for external commands. +- Implemented filter parsing with pest grammar. +- Added HTML UI for server with item listing and details. + +## Remaining/Planned Tasks +- **Documentation**: Update README.md with installation, usage examples, API docs. +- **More Plugins**: Add file size, creation time plugins if needed. +- **Performance**: Optimize streaming for very large files; add content caching. +- **Security**: Add rate limiting to API; validate input more strictly. +- **Features**: + - Update mode for modifying existing items. + - Search mode with full-text search on metadata/content. + - Export/import database. + - Backup/restore functionality. +- **Testing**: Add more integration tests for server API and MCP. +- **Polish**: + - Improve error messages and user feedback. + - Add progress indicators for large operations. + - Support for custom compression plugins. +- **Packaging**: Create installation scripts, Docker support. + +## Potential Improvements +- Switch to a more efficient database for large datasets (e.g., SQLite with WAL mode). +- Add WebSocket support for real-time updates in server mode. +- Implement plugin hot-reloading for development. +- Add CLI completion (bash/zsh/fish). +- Support for archiving old items automatically. + +Last updated: After codebase initialization and feature implementation.