From 20c27169153bfd8bd5d4d07376973c34bf5c6f25 Mon Sep 17 00:00:00 2001 From: Andrew Phillips Date: Fri, 15 Aug 2025 13:20:53 -0300 Subject: [PATCH] docs: update DESIGN.md with current application state Co-authored-by: aider (openai/andrew/openrouter/qwen/qwen3-coder) --- DESIGN.md | 135 +++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 118 insertions(+), 17 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 2e4d026..2b64afc 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -47,30 +47,131 @@ - `compression_engine/none.rs` - No compression implementation - `compression_engine/program.rs` - External program wrapper -### Digest Functionality -- Digest functionality is now integrated into meta plugins -- SHA-256 and other digest algorithms are implemented as meta plugins -- External digest programs are supported through meta plugin program wrapper - ### Meta Plugin Module - `meta_plugin.rs` - Trait and type definitions - `meta_plugin/program.rs` - External program wrapper - `meta_plugin/digest.rs` - Internal digest implementations +- `meta_plugin/system.rs` - System information metadata plugins -### Plugins Module +### Common Modules +- `common/is_binary.rs` - Binary file detection utilities +- `common/status.rs` - Status information generation + +### Utility Modules - `plugins.rs` - Shared plugin utilities -- Contains `ProgramWriter` for external process communication +- `args.rs` - CLI argument definitions + +## Command Line Interface + +### Modes +- Save mode: `keep [--save]` (default when no mode specified and no IDs provided) +- Get mode: `keep [--get] ` (default when IDs provided) +- List mode: `keep [--list] [tag...]` +- Info mode: `keep [--info] ` +- Delete mode: `keep [--delete] ` +- Update mode: `keep [--update] [tag...]` +- Diff mode: `keep [--diff] ` +- Status mode: `keep [--status]` +- Server mode: `keep [--server] ` + +### Item Options +- `--meta KEY[=VALUE]` - Set metadata for the item, remove if VALUE not provided +- `--digest ` - Digest algorithm to use when saving items +- `--compression ` - Compression algorithm to use when saving items +- `--meta-plugins ` - Meta plugins to use when saving items + +### General Options +- `--dir ` - Specify the directory to use for storage +- `--list-format ` - A comma separated list of columns to display with --list +- `--human-readable` - Display file sizes with units +- `--verbose` - Increase message verbosity +- `--quiet` - Do not show any messages +- `--output-format ` - Output format for info, status, and list modes +- `--server-password ` - Password for server authentication +- `--force` - Force output even when binary data would be sent to a TTY + +## Data Storage + +### Database Schema +- `items` table: id (primary key), ts (timestamp), size (optional), compression +- `tags` table: id (foreign key to items), name (tag name) +- `metas` table: id (foreign key to items), name (meta key), value (meta value) +- Indexes on tag names and meta names for faster queries + +### File Storage +- Data directory contains compressed item files named by their item ID +- Database file stored in data directory +- File permissions set to be private to user (umask 077) ## REST API Endpoints -### Item Operations -- `GET /api/item/latest` - Return the latest item as JSON, with metadata and content. Optional params: `tags[]` (find latest with matching tags) -- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags[]` -- `GET /api/item/latest/content` - Return the raw content of the latest item using the mime type from meta, or unknown binary. Optional params: `tags[]` -- `GET /api/item/<#>` - Return the item as JSON, with metadata and content -- `GET /api/item/<#>/meta` - Return the item metadata as JSON -- `GET /api/item/<#>/content` - Return the raw content of the item using the mime type from meta, or unknown binary -- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest`, `start=0`, `count=100`, `tags[]` -- `POST /api/item/` - Add a new item. Optional params: `tags[]` (defaults to `['none']` if empty), `meta[]` -- `DELETE /api/item/<#>` - Delete an item +### Status Operations +- `GET /api/status` - Get system status information +### Item Operations +- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest|oldest`, `start=0`, `count=100`, `tags[]=tag1&tags[]=tag2` +- `POST /api/item/` - Add a new item +- `DELETE /api/item/<#>` - Delete an item +- `GET /api/item/latest` - Return the latest item as JSON. Optional params: `tags[]=tag1&tags[]=tag2`, `allow_binary=true|false` +- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags[]=tag1&tags[]=tag2` +- `GET /api/item/latest/content` - Return the raw content of the latest item. Optional params: `tags[]=tag1&tags[]=tag2` +- `GET /api/item/<#>` - Return the item as JSON. Optional params: `allow_binary=true|false` +- `GET /api/item/<#>/meta` - Return the item metadata as JSON +- `GET /api/item/<#>/content` - Return the raw content of the item + +### Authentication +- Bearer token authentication: `Authorization: Bearer ` +- Basic authentication: `Authorization: Basic base64(keep:)` +- When no password is set, authentication is disabled + +## Supported Compression Types +- LZ4 (internal implementation) +- GZip (internal implementation) +- BZip2 (external program) +- XZ (external program) +- ZStd (external program) +- None (no compression) + +## Supported Meta Plugins +- FileMagic - File type detection using file command +- FileMime - MIME type detection using file command +- FileEncoding - File encoding detection using file command +- LineCount - Line count using wc command +- WordCount - Word count using wc command +- Cwd - Current working directory +- Binary - Binary file detection +- Uid - Current user ID +- User - Current username +- Gid - Current group ID +- Group - Current group name +- Shell - Shell path from SHELL environment variable +- ShellPid - Shell process ID from PPID environment variable +- KeepPid - Keep process ID +- DigestSha256 - SHA-256 digest +- DigestMd5 - MD5 digest using md5sum command +- ReadTime - Time taken to read data +- ReadRate - Rate of data reading +- Hostname - System hostname +- FullHostname - Fully qualified domain name + +## Testing Strategy +- Unit tests for each module in `src/tests/` +- Integration tests for modes +- Database tests for CRUD operations +- Compression engine tests for each supported format +- Meta plugin tests for each plugin type +- Server tests for API endpoints and authentication +- Common utilities tests for helper functions + +## Binary Data Handling +- Automatic binary detection using file signatures and heuristics +- Prevents binary data output to TTY unless --force is used +- Binary meta plugin analyzes content to determine if it's binary +- API endpoints respect binary flags to prevent accidental binary transmission + +## Security Considerations +- File permissions are restricted to user only (umask 077) +- Input validation for item IDs to prevent path traversal +- Authentication for server mode with bearer or basic auth +- Proper resource cleanup using RAII patterns +- Safe handling of external processes with proper stdin/stdout management