docs: update DESIGN.md with current application state

Co-authored-by: aider (openai/andrew/openrouter/qwen/qwen3-coder) <aider@aider.chat>

DESIGN.md

@@ -47,30 +47,131 @@
 - `compression_engine/none.rs` - No compression implementation
 - `compression_engine/program.rs` - External program wrapper
 
-### Digest Functionality
-- Digest functionality is now integrated into meta plugins
-- SHA-256 and other digest algorithms are implemented as meta plugins
-- External digest programs are supported through meta plugin program wrapper
-
 ### Meta Plugin Module
 - `meta_plugin.rs` - Trait and type definitions
 - `meta_plugin/program.rs` - External program wrapper
 - `meta_plugin/digest.rs` - Internal digest implementations
+- `meta_plugin/system.rs` - System information metadata plugins
 
-### Plugins Module
+### Common Modules
+- `common/is_binary.rs` - Binary file detection utilities
+- `common/status.rs` - Status information generation
+
+### Utility Modules
 - `plugins.rs` - Shared plugin utilities
-- Contains `ProgramWriter` for external process communication
+- `args.rs` - CLI argument definitions
+
+## Command Line Interface
+
+### Modes
+- Save mode: `keep [--save]` (default when no mode specified and no IDs provided)
+- Get mode: `keep [--get] <ID|tag...>` (default when IDs provided)
+- List mode: `keep [--list] [tag...]`
+- Info mode: `keep [--info] <ID|tag...>`
+- Delete mode: `keep [--delete] <ID...>`
+- Update mode: `keep [--update] <ID> [tag...]`
+- Diff mode: `keep [--diff] <ID1> <ID2>`
+- Status mode: `keep [--status]`
+- Server mode: `keep [--server] <address:port>`
+
+### Item Options
+- `--meta KEY[=VALUE]` - Set metadata for the item, remove if VALUE not provided
+- `--digest <sha256|md5>` - Digest algorithm to use when saving items
+- `--compression <lz4|gzip|bzip2|xz|zstd|none>` - Compression algorithm to use when saving items
+- `--meta-plugins <plugin[,plugin...]>` - Meta plugins to use when saving items
+
+### General Options
+- `--dir <PATH>` - Specify the directory to use for storage
+- `--list-format <FORMAT>` - A comma separated list of columns to display with --list
+- `--human-readable` - Display file sizes with units
+- `--verbose` - Increase message verbosity
+- `--quiet` - Do not show any messages
+- `--output-format <table|json|yaml>` - Output format for info, status, and list modes
+- `--server-password <PASSWORD>` - Password for server authentication
+- `--force` - Force output even when binary data would be sent to a TTY
+
+## Data Storage
+
+### Database Schema
+- `items` table: id (primary key), ts (timestamp), size (optional), compression
+- `tags` table: id (foreign key to items), name (tag name)
+- `metas` table: id (foreign key to items), name (meta key), value (meta value)
+- Indexes on tag names and meta names for faster queries
+
+### File Storage
+- Data directory contains compressed item files named by their item ID
+- Database file stored in data directory
+- File permissions set to be private to user (umask 077)
 
 ## REST API Endpoints
 
-### Item Operations
+### Status Operations
-- `GET /api/item/latest` - Return the latest item as JSON, with metadata and content. Optional params: `tags[]` (find latest with matching tags)
+- `GET /api/status` - Get system status information
-- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags[]`
-- `GET /api/item/latest/content` - Return the raw content of the latest item using the mime type from meta, or unknown binary. Optional params: `tags[]`
-- `GET /api/item/<#>` - Return the item as JSON, with metadata and content
-- `GET /api/item/<#>/meta` - Return the item metadata as JSON
-- `GET /api/item/<#>/content` - Return the raw content of the item using the mime type from meta, or unknown binary
-- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest`, `start=0`, `count=100`, `tags[]`
-- `POST /api/item/` - Add a new item. Optional params: `tags[]` (defaults to `['none']` if empty), `meta[]`
-- `DELETE /api/item/<#>` - Delete an item
 
+### Item Operations
+- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest|oldest`, `start=0`, `count=100`, `tags[]=tag1&tags[]=tag2`
+- `POST /api/item/` - Add a new item
+- `DELETE /api/item/<#>` - Delete an item
+- `GET /api/item/latest` - Return the latest item as JSON. Optional params: `tags[]=tag1&tags[]=tag2`, `allow_binary=true|false`
+- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags[]=tag1&tags[]=tag2`
+- `GET /api/item/latest/content` - Return the raw content of the latest item. Optional params: `tags[]=tag1&tags[]=tag2`
+- `GET /api/item/<#>` - Return the item as JSON. Optional params: `allow_binary=true|false`
+- `GET /api/item/<#>/meta` - Return the item metadata as JSON
+- `GET /api/item/<#>/content` - Return the raw content of the item
+
+### Authentication
+- Bearer token authentication: `Authorization: Bearer <password>`
+- Basic authentication: `Authorization: Basic base64(keep:<password>)`
+- When no password is set, authentication is disabled
+
+## Supported Compression Types
+- LZ4 (internal implementation)
+- GZip (internal implementation)
+- BZip2 (external program)
+- XZ (external program)
+- ZStd (external program)
+- None (no compression)
+
+## Supported Meta Plugins
+- FileMagic - File type detection using file command
+- FileMime - MIME type detection using file command
+- FileEncoding - File encoding detection using file command
+- LineCount - Line count using wc command
+- WordCount - Word count using wc command
+- Cwd - Current working directory
+- Binary - Binary file detection
+- Uid - Current user ID
+- User - Current username
+- Gid - Current group ID
+- Group - Current group name
+- Shell - Shell path from SHELL environment variable
+- ShellPid - Shell process ID from PPID environment variable
+- KeepPid - Keep process ID
+- DigestSha256 - SHA-256 digest
+- DigestMd5 - MD5 digest using md5sum command
+- ReadTime - Time taken to read data
+- ReadRate - Rate of data reading
+- Hostname - System hostname
+- FullHostname - Fully qualified domain name
+
+## Testing Strategy
+- Unit tests for each module in `src/tests/`
+- Integration tests for modes
+- Database tests for CRUD operations
+- Compression engine tests for each supported format
+- Meta plugin tests for each plugin type
+- Server tests for API endpoints and authentication
+- Common utilities tests for helper functions
+
+## Binary Data Handling
+- Automatic binary detection using file signatures and heuristics
+- Prevents binary data output to TTY unless --force is used
+- Binary meta plugin analyzes content to determine if it's binary
+- API endpoints respect binary flags to prevent accidental binary transmission
+
+## Security Considerations
+- File permissions are restricted to user only (umask 077)
+- Input validation for item IDs to prevent path traversal
+- Authentication for server mode with bearer or basic auth
+- Proper resource cleanup using RAII patterns
+- Safe handling of external processes with proper stdin/stdout management