diff --git a/AGENTS.md b/AGENTS.md index ec402a6..f8f3bca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,16 +37,3 @@ TERM=dumb cargo build --features server # With server feature - Tests in `src/tests/` mirroring `src/` structure; shared helpers in `src/tests/common/test_helpers.rs` - Key helpers: `create_temp_dir()`, `create_temp_db()`, `test_compression_engine()` - Test naming: `test__` - -## Procedures - -### Fix build problems - -1. `TERM=dumb cargo check` -2. Read affected files, fix errors, preserve functionality, don't downgrade versions -3. Prefer `write_file` for full file rewrites; repeat from step 1 - -### Fix formatting - -1. `TERM=dumb cargo fmt` -2. Continue with fix build problems procedure diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..f4ede5d --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Andrew Phillips + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index e69de29..0000000 diff --git a/README.md b/README.md index f2e3a6e..dcf1a20 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,473 @@ -# Keep - Temporary File Management with Compression and Metadata +# Keep -Keep is a command-line tool for managing temporary files with automatic compression, metadata generation, and querying capabilities. It supports various compression algorithms and metadata plugins for rich item inspection. +A command-line utility for managing temporary files with automatic compression, metadata generation, and querying. Instead of redirecting output to temporary files, pipe it into `keep` for organized storage and retrieval. + +```sh +# Instead of this: +curl -s https://api.example.com/data > /tmp/api-data.json +cat /tmp/api-data.json + +# Do this: +curl -s https://api.example.com/data | keep api-data +keep --get api-data +``` ## Features -- **Store and Retrieve**: Save content with automatic compression and retrieve by ID or tags. -- **Compression Support**: Built-in support for LZ4, GZip, and more via external programs (BZip2, XZ, ZStd). -- **Metadata Plugins**: Automatic extraction of file type, digests, hostname, user info, and custom metadata. -- **Filtering**: Apply filters (head, tail, grep, etc.) when retrieving content. -- **Querying**: List, search, and diff items with flexible formatting. -- **REST API Server**: Optional HTTP server for programmatic access. -- **Modular Design**: Extensible via plugins for compression, metadata, and filtering. +- **Store and Retrieve** - Save content with tags, retrieve by ID or tag +- **Automatic Compression** - LZ4, GZip, BZip2, XZ, ZStd support +- **Metadata Plugins** - Auto-extract file type, digests, hostname, user info, and more +- **Filters** - Apply transformations (head, tail, grep, strip ANSI) on retrieval +- **Querying** - List, search, diff items with flexible formatting +- **REST API Server** - Optional HTTP server for programmatic access +- **MCP Support** - Model Context Protocol integration for AI assistants +- **Modular Design** - Extensible plugin system for compression, metadata, and filtering ## Installation +### From Source + +Requires Rust and Cargo. + +```sh +cargo build --release +cargo install --path . +``` + +### Static Binary (Linux) + +```sh +./build-static.bash +# Binary at bin/keep +``` + +### Using Cargo + +```sh +cargo install --path . +``` + +## Quick Start + +```sh +# Save content with tags +echo "Hello, world!" | keep --save greeting + +# Retrieve by tag +keep --get greeting + +# List all items +keep --list + +# Get item details +keep --info greeting + +# Delete by tag +keep --delete greeting +``` + +## Usage + +### Save Mode + +Save stdin content with tags and metadata. + +```sh +# Basic save (auto-assigned ID) +echo "data" | keep --save + +# Save with tags +echo "data" | keep --save my-tag + +# Save with multiple tags and metadata +cat report.pdf | keep --save report --meta project=alpha --meta env=prod + +# Specify compression and digest +echo "data" | keep --save my-tag --compression gzip --digest sha256 +``` + +### Get Mode + +Retrieve items by ID or tags. Default mode when IDs are provided. + +```sh +# Get by ID +keep --get 1 +keep 1 + +# Get by tag +keep --get my-tag +keep my-tag + +# Get with filters +keep --get 1 --filters "head_lines(10)" + +# Get with metadata filter +keep --get --meta project=alpha + +# Force binary output to TTY +keep --get 1 --force +``` + +### List Mode + +List stored items with filtering and formatting. + +```sh +# List all items +keep --list + +# List by tag +keep --list my-tag + +# List with metadata filter +keep --list --meta env=prod + +# Custom column format +keep --list --list-format "id,time,size,tags" + +# JSON output +keep --list --output-format json + +# Human-readable sizes +keep --list --human-readable +``` + +### Info Mode + +Show detailed information about an item. + +```sh +keep --info 1 +keep --info my-tag +keep --info --meta key=value +``` + +### Update Mode + +Update an item's tags and metadata. + +```sh +# Replace tags +keep --update 1 new-tag + +# Update metadata +keep --update 1 --meta key=newvalue + +# Remove metadata (no value) +keep --update 1 --meta key +``` + +### Delete Mode + +Delete items by ID. + +```sh +keep --delete 1 +keep --delete 1 2 3 +``` + +### Diff Mode + +Show differences between two items. + +```sh +keep --diff 1 2 +``` + +### Status Mode + +Show system status and supported features. + +```sh +keep --status +keep --status-plugins +keep --status --verbose +``` + +## Filters + +Apply transformations to item content during retrieval. Filters are chained with `|`. + +```sh +# First 10 lines +keep --get 1 --filters "head_lines(10)" + +# Skip first 5 lines, then grep +keep --get 1 --filters "skip_lines(5)|grep(pattern=error)" + +# Strip ANSI escape codes +keep --get 1 --filters "strip_ansi" + +# Last 100 bytes +keep --get 1 --filters "tail_bytes(100)" +``` + +### Available Filters + +| Filter | Description | Parameters | +|--------|-------------|------------| +| `head_bytes(n)` | First n bytes | `count` | +| `head_lines(n)` | First n lines | `count` | +| `tail_bytes(n)` | Last n bytes | `count` | +| `tail_lines(n)` | Last n lines | `count` | +| `skip_bytes(n)` | Skip first n bytes | `count` | +| `skip_lines(n)` | Skip first n lines | `count` | +| `grep(pattern)` | Filter matching lines | `pattern` (regex) | +| `strip_ansi` | Remove ANSI escape codes | none | + +Set the `KEEP_FILTERS` environment variable to apply filters by default. + +## Configuration + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `KEEP_DIR` | Storage directory | `~/.keep` | +| `KEEP_CONFIG` | Config file path | `~/.config/keep/config.yml` | +| `KEEP_COMPRESSION` | Compression algorithm | `lz4` | +| `KEEP_META_PLUGINS` | Meta plugins to use | `env` | +| `KEEP_FILTERS` | Default filter chain | none | +| `KEEP_LIST_FORMAT` | List column format | built-in defaults | +| `KEEP_SERVER_ADDRESS` | Server bind address | `127.0.0.1` | +| `KEEP_SERVER_PORT` | Server port | `21080` | +| `KEEP_SERVER_PASSWORD` | Server password | none | +| `KEEP_SERVER_PASSWORD_HASH` | Server password hash | none | + +Any config setting can be overridden with `KEEP__` environment variables (double underscore separator). + +Capture shell commands automatically by setting `KEEP_META_command` in your shell profile (see Shell Integration). + +### Configuration File + +Default location: `~/.config/keep/config.yml` + +Generate a default configuration: + +```sh +keep --generate-config > ~/.config/keep/config.yml +``` + +```yaml +# Storage directory +dir: ~/.keep + +# List view columns +list_format: + - name: id + label: "Item" + align: right + - name: time + label: "Time" + align: right + - name: size + label: "Size" + align: right + - name: tags + label: "Tags" + align: left + +# Table styling +table_config: + style: utf8_full + content_arrangement: dynamic + +# Default compression +compression_plugin: + name: gzip + +# Default meta plugins +meta_plugins: + - name: env + - name: digest + options: + algorithm: sha256 + +# Server settings +server: + address: "127.0.0.1" + port: 21080 + password: "secret" + +human_readable: true +quiet: false +force: false +``` + +## Server Mode + +Start an HTTP REST API server for programmatic access. + +```sh +# Start server on default address (127.0.0.1:21080) +keep --server + +# Custom address and port +keep --server --server-address 0.0.0.0 --server-port 8080 + +# With authentication +keep --server --server-password mypassword +``` + +### API Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/status` | System status | +| `GET` | `/api/plugins/status` | Plugins status | +| `GET` | `/api/item/` | List items (supports `tags`, `order`, `start`, `count` params) | +| `POST` | `/api/item/` | Create item (body: raw content, query: `tags`, `metadata`) | +| `GET` | `/api/item/latest/content` | Latest item content | +| `GET` | `/api/item/latest/meta` | Latest item metadata | +| `GET` | `/api/item/{id}` | Item info by ID | +| `GET` | `/api/item/{id}/content` | Item content by ID | +| `GET` | `/api/item/{id}/meta` | Item metadata by ID | +| `GET` | `/api/item/{id}/info` | Item info by ID | +| `DELETE` | `/api/item/{id}` | Delete item by ID | +| `GET` | `/api/diff` | Diff two items (`id_a`, `id_b` params) | + +### Authentication + +```sh +# Bearer token +curl -H "Authorization: Bearer mypassword" http://localhost:21080/api/status + +# Basic auth +curl -u keep:mypassword http://localhost:21080/api/status +``` + +When no password is configured, authentication is disabled. + +### Swagger UI + +Build with the `swagger` feature to enable OpenAPI documentation: + +```sh +cargo build --features server,swagger +``` + +Swagger UI available at `/swagger`, OpenAPI spec at `/openapi.json`. + +## MCP (Model Context Protocol) + +AI assistant integration via the Model Context Protocol. Enable with the `mcp` feature. + +```sh +cargo build --features server,mcp +``` + +MCP endpoint available at `/mcp/sse` when the server is running. + +### Available Tools + +| Tool | Description | Parameters | +|------|-------------|------------| +| `save_item` | Save new content | `content`, `tags[]`, `metadata{}` | +| `get_item` | Get item by ID | `id` | +| `get_latest_item` | Get latest item | `tags[]` | +| `list_items` | List items | `tags[]`, `limit`, `offset` | +| `search_items` | Search items | `tags[]`, `metadata{}` | + +## Feature Flags + +| Feature | Default | Description | +|---------|---------|-------------| +| `magic` | Yes | File type detection via libmagic | +| `lz4` | Yes | LZ4 compression (internal) | +| `gzip` | Yes | GZip compression (internal) | +| `server` | No | HTTP REST API server | +| `mcp` | No | Model Context Protocol support | +| `swagger` | No | Swagger UI for API docs | +| `bzip2` | No | BZip2 compression (external program) | +| `xz` | No | XZ compression (external program) | +| `zstd` | No | ZStd compression (external program) | + +```sh +# Build with server and swagger +cargo build --features server,swagger + +# Build with all features +cargo build --features server,mcp,swagger,magic +``` + +## Shell Integration + +Source `profile.bash` to enable shell integration: + +```sh +source /path/to/keep/profile.bash +``` + +This provides: + +- **`keep` function** - Captures the current command in metadata automatically +- **`@` alias** - Shorthand for `keep --save` +- **`@@` alias** - Shorthand for `keep --get` + +```sh +# Save with automatic command capture +curl -s api.example.com | @ api-response + +# Quick retrieve +@@ api-response +``` + +## Compression + +Items are compressed automatically on save. Default: LZ4. + +| Algorithm | Type | Speed | Ratio | +|-----------|------|-------|-------| +| `lz4` | Internal | Fastest | Lower | +| `gzip` | Internal | Fast | Good | +| `bzip2` | External | Slow | Better | +| `xz` | External | Slowest | Best | +| `zstd` | External | Fast | Good | +| `none` | Internal | N/A | N/A | + +```sh +# Specify compression per item +echo "data" | keep --save my-tag --compression zstd + +# Set default via environment +export KEEP_COMPRESSION=gzip +``` + +External compression programs (`bzip2`, `xz`, `zstd`) must be installed on the system. + +## Meta Plugins + +Metadata is automatically extracted when saving items. + +| Plugin | Key | Description | +|--------|-----|-------------| +| `env` | `*` | Capture `KEEP_META_*` environment variables | +| `magic_file` | `file_type` | File type detection (requires `magic` feature) | +| `text` | `text_line_count`, `text_word_count` | Line and word counts | +| `user` | `uid`, `user`, `gid`, `group` | Current user info | +| `shell` | `shell` | Current shell path | +| `shell_pid` | `shell_pid` | Shell process ID | +| `keep_pid` | `keep_pid` | Keep process ID | +| `digest` | `digest_sha256`, `digest_md5` | Content digests | +| `read_time` | `read_time` | Time to read content | +| `read_rate` | `read_rate` | Data read rate | +| `hostname` | `hostname`, `hostname_short` | System hostname | +| `exec` | Custom | Run external commands for metadata | +| `cwd` | `cwd` | Current working directory | + +```sh +# Use specific plugins +echo "data" | keep --save tag --meta-plugins "digest,text,user" + +# Capture custom metadata via environment +KEEP_META_project=alpha echo "data" | keep --save tag +``` + +## License + +MIT License - see [LICENSE](LICENSE) for details. + +## Contact + +Andrew Phillips - andrew@gt0.ca diff --git a/README.org b/README.org deleted file mode 100644 index 91b04d1..0000000 --- a/README.org +++ /dev/null @@ -1,141 +0,0 @@ -#+TITLE: Keep -#+AUTHOR: Andrew Phillips - -* Introduction -Keep is a command-line utility designed to manage temporary files created on the command line. Instead of redirecting output to a temporary file (e.g., =command > ~/whatever.tmp=), you can use =keep= to handle the temporary files for you (e.g., =command | keep=). - -* Installation -To install Keep, you need to have Rust and Cargo installed on your system. You can then build and install Keep using the following commands: - -#+BEGIN_SRC sh -cargo build --release -cargo install --path . -#+END_SRC - -* Usage -Keep provides several subcommands to manage temporary files. Below are some examples of how to use Keep. - -** Saving an Item -To save an item with tags and metadata, you can use the =--save= option: - -#+BEGIN_SRC sh -echo "Hello, world!" | keep --save example --meta key=value -#+END_SRC - -** Getting an Item -To retrieve an item by its ID or by matching tags and metadata, you can use the =--get= option: - -#+BEGIN_SRC sh -keep --get 1 -keep --get example -keep --get --meta key=value -keep 1 -keep example -#+END_SRC - -** Listing Items -To list all items or filter them by tags and metadata, you can use the =--list= option: - -#+BEGIN_SRC sh -keep --list -keep --list example -keep --list --meta key=value -#+END_SRC - -** Updating an Item -To update an item's tags and metadata, you can use the =--update= option: - -#+BEGIN_SRC sh -keep --update 1 newtag --meta key=newvalue -#+END_SRC - -** Deleting an Item -To delete an item by its ID or by matching tags, you can use the =--delete= option: - -#+BEGIN_SRC sh -keep --delete 1 -keep --delete example -#+END_SRC - -** Showing Status -To show the status of directories and supported compression algorithms, you can use the =--status= option: - -#+BEGIN_SRC sh -keep --status -#+END_SRC - -** Diffing Items -To show a diff between two items by ID, you can use the =--diff= option: - -#+BEGIN_SRC sh -keep --diff 1 2 -#+END_SRC - -** Getting Information About an Item -To get detailed information about an item by its ID or by matching tags and metadata, you can use the =--info= option: - -#+BEGIN_SRC sh -keep --info 1 -keep --info example -keep --info --meta key=value -#+END_SRC - -* Configuration -Keep can be configured using environment variables and command-line options. The following environment variables are supported: - -- =KEEP_DIR=: Specify the directory to use for storage. -- =KEEP_LIST_FORMAT=: A comma-separated list of columns to display with =--list=. -- =KEEP_DIGEST=: Digest algorithm to use when saving items. -- =KEEP_COMPRESSION=: Compression algorithm to use when saving items. - -* Examples -Here are some examples of how to use Keep with different options: - -** Saving an Item with Compression and Digest -#+BEGIN_SRC sh -echo "Hello, world!" | keep --save example --meta key=value --compression gzip --digest sha256 -#+END_SRC - -** Getting an Item with Human-Readable Sizes -#+BEGIN_SRC sh -keep --get 1 --human-readable -#+END_SRC - -** Listing Items with Custom Format -#+BEGIN_SRC sh -keep --list --list-format "id,time,size,tags,meta:hostname" -#+END_SRC - -** Updating an Item with New Tags and Metadata -#+BEGIN_SRC sh -keep --update 1 newtag --meta key=newvalue -#+END_SRC - -** Deleting an Item by Tag -#+BEGIN_SRC sh -keep --delete example -#+END_SRC - -** Showing Status with Verbose Output -#+BEGIN_SRC sh -keep --status --verbose -#+END_SRC - -** Diffing Items with IDs -#+BEGIN_SRC sh -keep --diff 1 2 -#+END_SRC - -** Getting Information About an Item with Metadata -#+BEGIN_SRC sh -keep --info 1 -#+END_SRC - -* License -Keep is licensed under the MIT License. See the LICENSE file for more details. - -* Contributing -Contributions are welcome! Please open an issue or submit a pull request on the GitHub repository. - -* Contact -For more information, please contact Andrew Phillips at andrew@gt0.ca.