# Development Plan for Keep

## Overview
Keep is a command-line tool for managing temporary files with automatic compression, metadata generation, and querying capabilities. It includes a REST API server and supports extensible plugins for compression, metadata, and filtering.

## Current Status (as of latest updates)
The project is in a functional state with core features implemented. Key components:

### Core Features
- **Save Mode**: Stores content with compression and metadata generation. Supports tags and custom metadata via environment variables.
- **Get Mode**: Retrieves item content by ID or tags, with optional filtering (head, tail, grep, etc.) and binary detection.
- **List Mode**: Lists items with customizable table format, sorting, and filtering by tags/metadata.
- **Delete Mode**: Deletes items by ID.
- **Info Mode**: Shows detailed information about an item including metadata.
- **Diff Mode**: Compares two items using the `diff` utility.
- **Status Mode**: Displays system status including paths, compression engines, and configured plugins.
- **Status Plugins Mode**: Shows detailed plugin configurations and available options.

### Compression Engines
- Built-in: LZ4, GZip, None.
- External programs: BZip2, XZ, ZStd (via `CompressionEngineProgram`).
- Factory system for selecting engines with fallback to default (LZ4 if available).

### Meta Plugins
- **Text**: Detects binary/text content, counts lines/words, tracks line length statistics (max, mean, median).
- **Cwd**: Captures current working directory.
- **User**: Captures user ID, group ID, username, group name.
- **Shell**: Captures shell environment.
- **ShellPid/KeepPid**: Captures parent and current process IDs.
- **Digest**: Computes MD5, SHA256, SHA512 hashes (configurable).
- **MagicFile**: Detects MIME type, encoding, and file type using libmagic.
- **Hostname**: Captures short/full hostname with DNS resolution.
- **Exec**: Runs external programs for custom metadata.
- **Env**: Captures environment variables prefixed with `KEEP_META_`.
- Plugin registry and configuration via YAML (options/outputs mapping).

### Filter Plugins
- **Head/Tail/Skip**: Bytes and lines (head_bytes(10), tail_lines(5), etc.).
- **Grep**: Regex pattern matching on lines.
- **StripAnsi**: Removes ANSI escape sequences.
- Chainable via filter strings (e.g., "head_lines(10)|grep(pattern=error)").

### Server Mode
- REST API with endpoints: `/api/status`, `/api/item/` (list/post), `/api/item/{id}/meta`, `/api/item/{id}/content`, `/api/item/latest/*`.
- Authentication: Bearer token or Basic auth (password or hashed).
- MCP (Model Context Protocol) support via SSE and JSON-RPC tools (save_item, get_item, list_items, etc.).
- OpenAPI/Swagger documentation.
- HTML pages for item listing and details.
- Streaming support for large content with offset/length parameters.
- Binary content detection and restriction.

### Configuration
- YAML config file (`~/.config/keep/config.yml`) with overrides from env vars and CLI.
- Table customization: columns, alignment, colors, styles (ASCII/UTF8).
- Plugin configuration: enable/disable outputs, custom options.

### Database
- SQLite with schema for items, tags, metadata.
- Migrations for schema evolution.
- Indexes on tags and metadata names.
- Foreign key constraints.

### Testing
- Unit tests for compression engines, meta plugins, filter plugins, database operations.
- Integration tests for modes (save, get, list, etc.).
- Test helpers for temp files/DBs.

### Other
- Binary detection using signatures and printable character ratio.
- Human-readable size formatting.
- Logging with configurable verbosity.
- Error handling with structured errors (`CoreError`).

## Completed Changes
- Implemented full plugin system for meta and filters with registry via ctors.
- Added async item service for server mode with blocking task spawning.
- Enhanced server API with streaming, pagination, and MCP integration.
- Improved configuration with table styling and plugin output mapping.
- Added binary detection in get mode and API.
- Implemented diff mode with pipes and compression handling.
- Created status and status-plugins modes with detailed output.
- Added generate-config mode with commented defaults.
- Integrated libmagic for file type detection.
- Enhanced text meta plugin with line/word statistics.
- Added exec meta plugin for external commands.
- Implemented filter parsing with pest grammar.
- Added HTML UI for server with item listing and details.

## Remaining/Planned Tasks
- **Documentation**: Update README.md with installation, usage examples, API docs.
- **More Plugins**: Add file size, creation time plugins if needed.
- **Performance**: Optimize streaming for very large files; add content caching.
- **Security**: Add rate limiting to API; validate input more strictly.
- **Features**:
  - Update mode for modifying existing items.
  - Search mode with full-text search on metadata/content.
  - Export/import database.
  - Backup/restore functionality.
- **Testing**: Add more integration tests for server API and MCP.
- **Polish**:
  - Improve error messages and user feedback.
  - Add progress indicators for large operations.
  - Support for custom compression plugins.
- **Packaging**: Create installation scripts, Docker support.

## Potential Improvements
- Switch to a more efficient database for large datasets (e.g., SQLite with WAL mode).
- Add WebSocket support for real-time updates in server mode.
- Implement plugin hot-reloading for development.
- Add CLI completion (bash/zsh/fish).
- Support for archiving old items automatically.

Last updated: After codebase initialization and feature implementation.