asp/keep
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: add HTTPS/TLS server support via rustls

Browse Source 
Add optional TLS support for the server using axum-server with the
tls-rustls feature. When --server-cert and --server-key are provided
(and tls feature is enabled), the server binds with TLS instead of
plain HTTP.

Changes:
- Add axum-server dependency with optional tls-rustls feature
- New 'tls' feature flag (independent of 'server')
- --server-cert/--server-key CLI args gated behind tls feature
- ServerConfig extended with cert_file/key_file fields
- Conditional TLS/HTTP binding in server mod.rs
- Fix PathBuf::to_str().unwrap() panic risk -> to_string_lossy()
- Update README.md and DESIGN.md with TLS documentation
This commit is contained in:
Andrew Phillips
2026-03-12 22:18:42 -03:00
parent 237a581429
commit bee980605f
8 changed files with 569 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

448
README.md
View File

@@ -1,27 +1,53 @@
# Keep
A command-line utility for managing temporary files with automatic compression, metadata generation, and querying. Instead of redirecting output to temporary files, pipe it into `keep` for organized storage and retrieval.
A command-line utility for storing and retrieving temporary data with automatic compression, metadata extraction, and querying. Pipe any output into `keep` for organized storage — no more losing data in `/tmp` files with cryptic names.
```sh
# Instead of this:
curl -s https://api.example.com/data > /tmp/api-data.json
cat /tmp/api-data.json
# Do this:
curl -s https://api.example.com/data | keep api-data
curl -s https://api.example.com/data | keep --save api-data
keep --get api-data
```
## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Usage](#usage)
- [Save Mode](#save-mode)
- [Get Mode](#get-mode)
- [List Mode](#list-mode)
- [Info Mode](#info-mode)
- [Update Mode](#update-mode)
- [Delete Mode](#delete-mode)
- [Diff Mode](#diff-mode)
- [Status Mode](#status-mode)
- [Filters](#filters)
- [Compression](#compression)
- [Meta Plugins](#meta-plugins)
- [Configuration](#configuration)
- [Client/Server Mode](#clientserver-mode)
- [Server Mode](#server-mode)
- [Client Mode](#client-mode)
- [API Endpoints](#api-endpoints)
- [MCP (Model Context Protocol)](#mcp-model-context-protocol)
- [Shell Integration](#shell-integration)
- [Feature Flags](#feature-flags)
- [License](#license)
## Features
- **Store and Retrieve** - Save content with tags, retrieve by ID or tag
- **Automatic Compression** - LZ4, GZip, BZip2, XZ, ZStd support
- **Metadata Plugins** - Auto-extract file type, digests, hostname, user info, and more
- **Filters** - Apply transformations (head, tail, grep, strip ANSI) on retrieval
- **Querying** - List, search, diff items with flexible formatting
- **REST API Server** - Optional HTTP server for programmatic access
- **MCP Support** - Model Context Protocol integration for AI assistants
- **Modular Design** - Extensible plugin system for compression, metadata, and filtering
- **Store and retrieve** — Save content with tags, retrieve by ID or tag
- **Automatic compression** — LZ4, GZip, BZip2, XZ, ZStd support
- **Metadata plugins** — Auto-extract file type, digests, hostname, user info, and more
- **Filters** — Apply transformations (head, tail, grep, strip ANSI) on retrieval
- **Querying** — List, search, diff items with flexible formatting
- **Client/server architecture** — Optional HTTP server with streaming support
- **MCP support** — Model Context Protocol integration for AI assistants
- **Modular design** — Extensible plugin system for compression, metadata, and filtering
## Installation
@@ -31,6 +57,11 @@ Requires Rust and Cargo.
```sh
cargo build --release
```
### Install via Cargo
```sh
cargo install --path .
```
@@ -41,22 +72,29 @@ cargo install --path .
# Binary at bin/keep
```
### Using Cargo
### Build with Server/Client Features
```sh
cargo install --path .
# Server only
cargo build --release --features server
# Client only (for connecting to a remote keep server)
cargo build --release --features client
# Server + client + all optional features
cargo build --release --features server,tls,client,swagger,mcp
```
## Quick Start
```sh
# Save content with tags
# Save content with a tag
echo "Hello, world!" | keep --save greeting
# Retrieve by tag
keep --get greeting
# List all items
# List all stored items
keep --list
# Get item details
@@ -66,6 +104,28 @@ keep --info greeting
keep --delete greeting
```
### Real-World Examples
```sh
# Save API response
curl -s https://api.github.com/repos/user/repo | keep --save repo-info
# Save test output with metadata
npm test 2>&1 | keep --save test-results --meta project=myapp --meta env=staging
# Chain commands: process and store
cat data.csv | sort | uniq | keep --save cleaned-data
# Diff two versions
keep --diff 1 5
# Get first 20 lines of an item
keep --get 1 --filters "head_lines(20)"
# List items from a specific project
keep --list --meta project=myapp
```
## Usage
### Save Mode
@@ -73,22 +133,24 @@ keep --delete greeting
Save stdin content with tags and metadata.
```sh
# Basic save (auto-assigned ID)
# Save (auto-assigned ID, no tag)
echo "data" | keep --save
# Save with tags
# Save with a tag
echo "data" | keep --save my-tag
# Save with multiple tags and metadata
cat report.pdf | keep --save report --meta project=alpha --meta env=prod
# Specify compression and digest
# Specify compression and digest algorithm
echo "data" | keep --save my-tag --compression gzip --digest sha256
```
Tags and metadata make items easy to find later. Tags are simple identifiers; metadata is key-value pairs.
### Get Mode
Retrieve items by ID or tags. Default mode when IDs are provided.
Retrieve items by ID or tags. This is the default mode when IDs are provided.
```sh
# Get by ID
@@ -99,13 +161,13 @@ keep 1
keep --get my-tag
keep my-tag
# Get with filters
# Get with filters applied
keep --get 1 --filters "head_lines(10)"
# Get with metadata filter
# Get by metadata filter
keep --get --meta project=alpha
# Force binary output to TTY
# Force binary output to TTY (override safety check)
keep --get 1 --force
```
@@ -120,16 +182,16 @@ keep --list
# List by tag
keep --list my-tag
# List with metadata filter
# Filter by metadata
keep --list --meta env=prod
# Custom column format
keep --list --list-format "id,time,size,tags"
# JSON output
# JSON output for scripting
keep --list --output-format json
# Human-readable sizes
# Human-readable file sizes
keep --list --human-readable
```
@@ -154,7 +216,7 @@ keep --update 1 new-tag
# Update metadata
keep --update 1 --meta key=newvalue
# Remove metadata (no value)
# Remove a metadata key
keep --update 1 --meta key
```
@@ -193,7 +255,7 @@ Apply transformations to item content during retrieval. Filters are chained with
# First 10 lines
keep --get 1 --filters "head_lines(10)"
# Skip first 5 lines, then grep
# Skip first 5 lines, then grep for errors
keep --get 1 --filters "skip_lines(5)|grep(pattern=error)"
# Strip ANSI escape codes
@@ -201,6 +263,9 @@ keep --get 1 --filters "strip_ansi"
# Last 100 bytes
keep --get 1 --filters "tail_bytes(100)"
# Complex chain
keep --get 1 --filters "skip_lines(10)|grep(pattern=TODO)|head_lines(5)"
```
### Available Filters
@@ -216,7 +281,61 @@ keep --get 1 --filters "tail_bytes(100)"
| `grep(pattern)` | Filter matching lines | `pattern` (regex) |
| `strip_ansi` | Remove ANSI escape codes | none |
Set the `KEEP_FILTERS` environment variable to apply filters by default.
Set `KEEP_FILTERS` to apply a default filter chain to all retrievals.
## Compression
Items are compressed automatically on save. Default: LZ4.
| Algorithm | Type | Speed | Ratio |
|-----------|------|-------|-------|
| `lz4` | Internal | Fastest | Lower |
| `gzip` | Internal | Fast | Good |
| `bzip2` | External | Slow | Better |
| `xz` | External | Slowest | Best |
| `zstd` | External | Fast | Good |
| `none` | Internal | N/A | N/A |
```sh
# Specify compression per item
echo "data" | keep --save my-tag --compression zstd
# Set default via environment
export KEEP_COMPRESSION=gzip
```
External compression programs (`bzip2`, `xz`, `zstd`) must be installed on the system.
## Meta Plugins
Metadata is automatically extracted when saving items.
| Plugin | Key | Description |
|--------|-----|-------------|
| `env` | `*` | Capture `KEEP_META_*` environment variables |
| `magic_file` | `file_type` | File type detection (requires `magic` feature) |
| `text` | `text_line_count`, `text_word_count` | Line and word counts |
| `user` | `uid`, `user`, `gid`, `group` | Current user info |
| `shell` | `shell` | Current shell path |
| `shell_pid` | `shell_pid` | Shell process ID |
| `keep_pid` | `keep_pid` | Keep process ID |
| `digest` | `digest_sha256`, `digest_md5` | Content digests |
| `read_time` | `read_time` | Time to read content |
| `read_rate` | `read_rate` | Data read rate |
| `hostname` | `hostname`, `hostname_short` | System hostname |
| `exec` | Custom | Run external commands for metadata |
| `cwd` | `cwd` | Current working directory |
```sh
# Use specific plugins
echo "data" | keep --save tag --meta-plugins "digest,text,user"
# Capture custom metadata via environment
KEEP_META_project=alpha echo "data" | keep --save tag
# Combine environment and CLI metadata
KEEP_META_build=1234 echo "data" | keep --save tag --meta env=staging
```
## Configuration
@@ -234,11 +353,13 @@ Set the `KEEP_FILTERS` environment variable to apply filters by default.
| `KEEP_SERVER_PORT` | Server port | `21080` |
| `KEEP_SERVER_PASSWORD` | Server password | none |
| `KEEP_SERVER_PASSWORD_HASH` | Server password hash | none |
| `KEEP_SERVER_CERT` | TLS certificate file path (PEM) | none |
| `KEEP_SERVER_KEY` | TLS private key file path (PEM) | none |
| `KEEP_CLIENT_URL` | Remote keep server URL | none |
| `KEEP_CLIENT_PASSWORD` | Remote server password | none |
Any config setting can be overridden with `KEEP__<SETTING>` environment variables (double underscore separator).
Capture shell commands automatically by setting `KEEP_META_command` in your shell profile (see Shell Integration).
### Configuration File
Default location: `~/.config/keep/config.yml`
@@ -289,18 +410,35 @@ server:
address: "127.0.0.1"
port: 21080
password: "secret"
# TLS (requires tls feature)
# cert_file: /path/to/cert.pem
# key_file: /path/to/key.pem
# Client settings
client:
url: "http://localhost:21080"
password: "secret"
human_readable: true
quiet: false
force: false
```
## Server Mode
## Client/Server Mode
Start an HTTP REST API server for programmatic access.
Keep supports a client/server architecture where one machine runs a keep server and other machines connect as clients. This is useful for:
- Centralizing stored data across multiple machines
- Sharing items between team members
- Offloading storage to a dedicated server
- Piping data from long-running processes without local storage
### Server Mode
Start an HTTP REST API server:
```sh
# Start server on default address (127.0.0.1:21080)
# Default: 127.0.0.1:21080
keep --server
# Custom address and port
@@ -310,24 +448,165 @@ keep --server --server-address 0.0.0.0 --server-port 8080
keep --server --server-password mypassword
```
#### HTTPS / TLS
Build with the `tls` feature to enable HTTPS:
```sh
cargo build --release --features server,tls
```
Provide a TLS certificate and private key (both PEM format):
```sh
# Via CLI flags
keep --server \
--server-cert /path/to/cert.pem \
--server-key /path/to/key.pem
# Via environment variables
export KEEP_SERVER_CERT=/path/to/cert.pem
export KEEP_SERVER_KEY=/path/to/key.pem
keep --server
# Via config file (config.yml)
server:
cert_file: /path/to/cert.pem
key_file: /path/to/key.pem
```
When cert and key are provided, the server listens with HTTPS. Without them, it falls back to plain HTTP. The port is controlled by `--server-port` (default: 21080).
**Self-signed certificates** (for development):
```sh
# Generate a self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
-days 365 -nodes -subj "/CN=localhost"
# Start server with self-signed cert
keep --server --server-cert cert.pem --server-key key.pem
# Connect client with HTTPS
keep --client-url https://localhost:21080 --save my-tag
```
The server accepts data from both dumb clients (raw HTTP/curl) and smart clients (the keep CLI).
#### Server Query Parameters
The server supports query parameters that control processing:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `tags` | none | Comma-separated tags |
| `metadata` | none | JSON-encoded metadata |
| `compress` | `true` | `false` = client already compressed, store as-is |
| `meta` | `true` | `false` = client handles metadata, skip server-side plugins |
| `decompress` | `true` | `false` = return raw compressed bytes on GET |
When using a smart client, these are set automatically. For curl, the server handles everything by default.
#### Example: Curl as a Dumb Client
```sh
# Save (server handles compression and metadata)
curl -X POST -d "my data" http://localhost:21080/api/item/?tags=my-tag
# Retrieve (server decompresses)
curl http://localhost:21080/api/item/1/content
# Save compressed (client handles compression, server skips)
gzip -c data.txt | curl -X POST -d @- "http://localhost:21080/api/item/?compress=false&tags=my-tag"
```
### Client Mode
The keep CLI can connect to a remote server as a smart client. Build with the `client` feature:
```sh
cargo build --release --features client
```
```sh
# Set server URL via flag or environment
keep --client-url http://server:21080 --save my-tag
export KEEP_CLIENT_URL=http://server:21080
# With authentication
keep --client-url http://server:21080 --client-password mypassword --save my-tag
export KEEP_CLIENT_PASSWORD=mypassword
```
#### How Client Mode Works
Client mode uses **local plugins** and **remote storage**:
1. **Save**: Local compression and metadata plugins run on the client; compressed data streams to the server
2. **Get**: Server sends raw compressed data; client decompresses locally and applies filters
3. **Other operations** (list, info, delete, diff): Delegated directly to the server
This means client behavior is consistent with local mode — the same compression settings and filters apply.
#### Streaming Architecture
Client save uses a 3-thread streaming pipeline for constant memory usage regardless of data size:
```
┌──────────────┐ OS pipe ┌────────────────┐
│ Reader thread ├──────────────────┤ Streamer thread│
│ │ (compressed │ │
│ stdin → tee │ bytes) │ pipe → POST │
│ → hash │ │ (chunked) │
│ → compress│ │ │
└──────────────┘ └────────────────┘
│ │
▼ ▼
stdout + Server stores blob
SHA-256 digest
```
- **Reader thread**: Reads stdin, tees output to stdout, computes SHA-256, compresses data, writes to OS pipe
- **Streamer thread**: Reads compressed bytes from pipe, streams to server via chunked HTTP POST
- **Main thread**: After streaming completes, sends computed metadata (digest, hostname, size) to server
Memory usage is O(PIPESIZE) — typically 64KB — regardless of how much data is being stored.
#### Example: Remote Pipeline
```sh
# On a build server, pipe logs to a central keep server
make build 2>&1 | keep --client-url http://logserver:21080 \
--save build-logs \
--meta project=myapp \
--meta branch=$(git branch --show-current)
# Retrieve from any machine
keep --client-url http://logserver:21080 --get build-logs
# List recent builds from a specific project
keep --client-url http://logserver:21080 --list --meta project=myapp
```
### API Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/status` | System status |
| `GET` | `/api/plugins/status` | Plugins status |
| `GET` | `/api/item/` | List items (supports `tags`, `order`, `start`, `count` params) |
| `POST` | `/api/item/` | Create item (body: raw content, query: `tags`, `metadata`) |
| `GET` | `/api/plugins/status` | Plugin status |
| `GET` | `/api/item/` | List items (`tags`, `order`, `start`, `count` params) |
| `POST` | `/api/item/` | Create item (body: raw content, params: `tags`, `metadata`, `compress`, `meta`) |
| `GET` | `/api/item/latest/content` | Latest item content |
| `GET` | `/api/item/latest/meta` | Latest item metadata |
| `GET` | `/api/item/{id}` | Item info by ID |
| `GET` | `/api/item/{id}/content` | Item content by ID |
| `GET` | `/api/item/{id}/meta` | Item metadata by ID |
| `GET` | `/api/item/{id}/info` | Item info by ID |
| `POST` | `/api/item/{id}/meta` | Add metadata to existing item (body: JSON object) |
| `DELETE` | `/api/item/{id}` | Delete item by ID |
| `GET` | `/api/diff` | Diff two items (`id_a`, `id_b` params) |
### Authentication
#### Authentication
```sh
# Bearer token
@@ -339,7 +618,7 @@ curl -u keep:mypassword http://localhost:21080/api/status
When no password is configured, authentication is disabled.
### Swagger UI
#### Swagger UI
Build with the `swagger` feature to enable OpenAPI documentation:
@@ -369,28 +648,6 @@ MCP endpoint available at `/mcp/sse` when the server is running.
| `list_items` | List items | `tags[]`, `limit`, `offset` |
| `search_items` | Search items | `tags[]`, `metadata{}` |
## Feature Flags
| Feature | Default | Description |
|---------|---------|-------------|
| `magic` | Yes | File type detection via libmagic |
| `lz4` | Yes | LZ4 compression (internal) |
| `gzip` | Yes | GZip compression (internal) |
| `server` | No | HTTP REST API server |
| `mcp` | No | Model Context Protocol support |
| `swagger` | No | Swagger UI for API docs |
| `bzip2` | No | BZip2 compression (external program) |
| `xz` | No | XZ compression (external program) |
| `zstd` | No | ZStd compression (external program) |
```sh
# Build with server and swagger
cargo build --features server,swagger
# Build with all features
cargo build --features server,mcp,swagger,magic
```
## Shell Integration
Source `profile.bash` to enable shell integration:
@@ -401,9 +658,9 @@ source /path/to/keep/profile.bash
This provides:
- **`keep` function** - Captures the current command in metadata automatically
- **`@` alias** - Shorthand for `keep --save`
- **`@@` alias** - Shorthand for `keep --get`
- **`keep` function** — Captures the current command in metadata automatically
- **`@` alias** — Shorthand for `keep --save`
- **`@@` alias** — Shorthand for `keep --get`
```sh
# Save with automatic command capture
@@ -413,55 +670,34 @@ curl -s api.example.com | @ api-response
@@ api-response
```
## Compression
## Feature Flags
Items are compressed automatically on save. Default: LZ4.
| Algorithm | Type | Speed | Ratio |
|-----------|------|-------|-------|
| `lz4` | Internal | Fastest | Lower |
| `gzip` | Internal | Fast | Good |
| `bzip2` | External | Slow | Better |
| `xz` | External | Slowest | Best |
| `zstd` | External | Fast | Good |
| `none` | Internal | N/A | N/A |
| Feature | Default | Description |
|---------|---------|-------------|
| `magic` | Yes | File type detection via libmagic |
| `lz4` | Yes | LZ4 compression (internal) |
| `gzip` | Yes | GZip compression (internal) |
| `server` | No | HTTP REST API server |
| `tls` | No | HTTPS/TLS server support (requires `server`) |
| `client` | No | HTTP client for remote server |
| `mcp` | No | Model Context Protocol support |
| `swagger` | No | Swagger UI for API docs |
| `bzip2` | No | BZip2 compression (external program) |
| `xz` | No | XZ compression (external program) |
| `zstd` | No | ZStd compression (external program) |
```sh
# Specify compression per item
echo "data" | keep --save my-tag --compression zstd
# Server with Swagger UI
cargo build --features server,swagger
# Set default via environment
export KEEP_COMPRESSION=gzip
```
# Server with HTTPS
cargo build --features server,tls
External compression programs (`bzip2`, `xz`, `zstd`) must be installed on the system.
# Client only
cargo build --features client
## Meta Plugins
Metadata is automatically extracted when saving items.
| Plugin | Key | Description |
|--------|-----|-------------|
| `env` | `*` | Capture `KEEP_META_*` environment variables |
| `magic_file` | `file_type` | File type detection (requires `magic` feature) |
| `text` | `text_line_count`, `text_word_count` | Line and word counts |
| `user` | `uid`, `user`, `gid`, `group` | Current user info |
| `shell` | `shell` | Current shell path |
| `shell_pid` | `shell_pid` | Shell process ID |
| `keep_pid` | `keep_pid` | Keep process ID |
| `digest` | `digest_sha256`, `digest_md5` | Content digests |
| `read_time` | `read_time` | Time to read content |
| `read_rate` | `read_rate` | Data read rate |
| `hostname` | `hostname`, `hostname_short` | System hostname |
| `exec` | Custom | Run external commands for metadata |
| `cwd` | `cwd` | Current working directory |
```sh
# Use specific plugins
echo "data" | keep --save tag --meta-plugins "digest,text,user"
# Capture custom metadata via environment
KEEP_META_project=alpha echo "data" | keep --save tag
# Everything
cargo build --features server,tls,client,mcp,swagger,magic
```
## License