keep/README.md

# Keep

A command-line utility for storing and retrieving temporary data with automatic compression, metadata extraction, and querying. Pipe any output into `keep` for organized storage — no more losing data in `/tmp` files with cryptic names.

```sh
# Instead of this:
curl -s https://api.example.com/data > /tmp/api-data.json

# Do this:
curl -s https://api.example.com/data | keep --save api-data
keep --get api-data
```

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Usage](#usage)
  - [Save Mode](#save-mode)
  - [Get Mode](#get-mode)
  - [List Mode](#list-mode)
  - [Info Mode](#info-mode)
  - [Update Mode](#update-mode)
  - [Delete Mode](#delete-mode)
  - [Diff Mode](#diff-mode)
  - [Status Mode](#status-mode)
- [Filters](#filters)
- [Compression](#compression)
- [Meta Plugins](#meta-plugins)
- [Configuration](#configuration)
- [Client/Server Mode](#clientserver-mode)
  - [Server Mode](#server-mode)
  - [Client Mode](#client-mode)
  - [API Endpoints](#api-endpoints)
- [Shell Integration](#shell-integration)
- [Feature Flags](#feature-flags)
- [License](#license)

## Features

- **Store and retrieve** — Save content with tags, retrieve by ID or tag
- **Automatic compression** — LZ4, GZip, BZip2, XZ, ZStd support
- **Metadata plugins** — Auto-extract file type, digests, hostname, user info, and more
- **Filters** — Apply transformations (head, tail, grep, strip ANSI) on retrieval
- **Querying** — List, search, diff items with flexible formatting
- **Client/server architecture** — Optional HTTP server with streaming support
- **Modular design** — Extensible plugin system for compression, metadata, and filtering

## Installation

### From Source

Requires Rust and Cargo.

```sh
cargo build --release
```

### Install via Cargo

```sh
cargo install --path .
```

### Static Binary (Linux)

```sh
./build-static.bash
# Binary at bin/keep
```

### Build with Server/Client Features

```sh
# Server only
cargo build --release --features server

# Client only (for connecting to a remote keep server)
cargo build --release --features client

# Server + client + all optional features
cargo build --release --features server,client,swagger
```

## Quick Start

```sh
# Save content with a tag
echo "Hello, world!" | keep --save greeting

# Retrieve by tag
keep --get greeting

# List all stored items
keep --list

# Get item details
keep --info greeting

# Delete by tag
keep --delete greeting
```

### Real-World Examples

```sh
# Save API response
curl -s https://api.github.com/repos/user/repo | keep --save repo-info

# Save test output with metadata
npm test 2>&1 | keep --save test-results --meta project=myapp --meta env=staging

# Chain commands: process and store
cat data.csv | sort | uniq | keep --save cleaned-data

# Diff two versions
keep --diff 1 5

# Get first 20 lines of an item
keep --get 1 --filters "head_lines(20)"

# List items from a specific project
keep --list --meta project=myapp
```

## Usage

### Save Mode

Save stdin content with tags and metadata.

```sh
# Save (auto-assigned ID, no tag)
echo "data" | keep --save

# Save with a tag
echo "data" | keep --save my-tag

# Save with multiple tags and metadata
cat report.pdf | keep --save report --meta project=alpha --meta env=prod

# Specify compression and digest algorithm
echo "data" | keep --save my-tag --compression gzip --digest sha256
```

Tags and metadata make items easy to find later. Tags are simple identifiers; metadata is key-value pairs.

### Get Mode

Retrieve items by ID or tags. This is the default mode when IDs are provided.

```sh
# Get by ID
keep --get 1
keep 1

# Get by tag
keep --get my-tag
keep my-tag

# Get with filters applied
keep --get 1 --filters "head_lines(10)"

# Get by metadata filter
keep --get --meta project=alpha

# Force binary output to TTY (override safety check)
keep --get 1 --force
```

### List Mode

List stored items with filtering and formatting.

```sh
# List all items
keep --list

# List by tag
keep --list my-tag

# Filter by metadata
keep --list --meta env=prod

# Custom column format
keep --list --list-format "id,time,size,tags"

# JSON output for scripting
keep --list --output-format json

# Human-readable file sizes
keep --list --human-readable
```

### Info Mode

Show detailed information about an item.

```sh
keep --info 1
keep --info my-tag
keep --info --meta key=value
```

### Update Mode

Update an item's tags and metadata.

```sh
# Replace tags
keep --update 1 new-tag

# Update metadata
keep --update 1 --meta key=newvalue

# Remove a metadata key
keep --update 1 --meta key
```

### Delete Mode

Delete items by ID.

```sh
keep --delete 1
keep --delete 1 2 3
```

### Diff Mode

Show differences between two items.

```sh
keep --diff 1 2
```

### Status Mode

Show system status and supported features.

```sh
keep --status
keep --status-plugins
keep --status --verbose
```

## Filters

Apply transformations to item content during retrieval. Filters are chained with `|`.

```sh
# First 10 lines
keep --get 1 --filters "head_lines(10)"

# Skip first 5 lines, then grep for errors
keep --get 1 --filters "skip_lines(5)|grep(pattern=error)"

# Strip ANSI escape codes
keep --get 1 --filters "strip_ansi"

# Last 100 bytes
keep --get 1 --filters "tail_bytes(100)"

# Complex chain
keep --get 1 --filters "skip_lines(10)|grep(pattern=TODO)|head_lines(5)"
```

### Available Filters

| Filter | Description | Parameters |
|--------|-------------|------------|
| `head_bytes(n)` | First n bytes | `count` |
| `head_lines(n)` | First n lines | `count` |
| `tail_bytes(n)` | Last n bytes | `count` |
| `tail_lines(n)` | Last n lines | `count` |
| `skip_bytes(n)` | Skip first n bytes | `count` |
| `skip_lines(n)` | Skip first n lines | `count` |
| `grep(pattern)` | Filter matching lines | `pattern` (regex) |
| `strip_ansi` | Remove ANSI escape codes | none |

Set `KEEP_FILTERS` to apply a default filter chain to all retrievals.

## Compression

Items are compressed automatically on save. Default: LZ4.

| Algorithm | Type | Speed | Ratio |
|-----------|------|-------|-------|
| `lz4` | Internal | Fastest | Lower |
| `gzip` | Internal | Fast | Good |
| `bzip2` | External | Slow | Better |
| `xz` | External | Slowest | Best |
| `zstd` | External | Fast | Good |
| `none` | Internal | N/A | N/A |

```sh
# Specify compression per item
echo "data" | keep --save my-tag --compression zstd

# Set default via environment
export KEEP_COMPRESSION=gzip
```

External compression programs (`bzip2`, `xz`, `zstd`) must be installed on the system.

## Meta Plugins

Metadata is automatically extracted when saving items.

| Plugin | Key | Description |
|--------|-----|-------------|
| `env` | `*` | Capture `KEEP_META_*` environment variables |
| `magic_file` | `file_type` | File type detection (requires `magic` feature) |
| `text` | `text_line_count`, `text_word_count` | Line and word counts |
| `user` | `uid`, `user`, `gid`, `group` | Current user info |
| `shell` | `shell` | Current shell path |
| `shell_pid` | `shell_pid` | Shell process ID |
| `keep_pid` | `keep_pid` | Keep process ID |
| `digest` | `digest_sha256`, `digest_md5` | Content digests |
| `read_time` | `read_time` | Time to read content |
| `read_rate` | `read_rate` | Data read rate |
| `hostname` | `hostname`, `hostname_short` | System hostname |
| `exec` | Custom | Run external commands for metadata |
| `cwd` | `cwd` | Current working directory |

```sh
# Use specific plugins
echo "data" | keep --save tag --meta-plugins "digest,text,user"

# Capture custom metadata via environment
KEEP_META_project=alpha echo "data" | keep --save tag

# Combine environment and CLI metadata
KEEP_META_build=1234 echo "data" | keep --save tag --meta env=staging
```

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `KEEP_DIR` | Storage directory | `~/.keep` |
| `KEEP_CONFIG` | Config file path | `~/.config/keep/config.yml` |
| `KEEP_COMPRESSION` | Compression algorithm | `lz4` |
| `KEEP_META_PLUGINS` | Meta plugins to use | `env` |
| `KEEP_FILTERS` | Default filter chain | none |
| `KEEP_LIST_FORMAT` | List column format | built-in defaults |
| `KEEP_SERVER_ADDRESS` | Server bind address | `127.0.0.1` |
| `KEEP_SERVER_PORT` | Server port | `21080` |
| `KEEP_SERVER_USERNAME` | Server Basic auth username | `keep` |
| `KEEP_SERVER_PASSWORD` | Server password | none |
| `KEEP_SERVER_PASSWORD_HASH` | Server password hash | none |
| `KEEP_SERVER_JWT_SECRET` | JWT secret for token auth | none |
| `KEEP_SERVER_JWT_SECRET_FILE` | Path to JWT secret file | none |
| `KEEP_SERVER_MAX_BODY_SIZE` | Maximum POST body size in bytes (0=unlimited) | unlimited |
| `KEEP_SERVER_CERT` | TLS certificate file path (PEM) | none |
| `KEEP_SERVER_KEY` | TLS private key file path (PEM) | none |
| `KEEP_CLIENT_URL` | Remote keep server URL | none |
| `KEEP_CLIENT_USERNAME` | Remote server username | `keep` |
| `KEEP_CLIENT_PASSWORD` | Remote server password | none |
| `KEEP_CLIENT_JWT` | JWT token for remote server | none |

Any config setting can be overridden with `KEEP__<SETTING>` environment variables (double underscore separator).

### Configuration File

Default location: `~/.config/keep/config.yml`

Generate a default configuration:

```sh
keep --generate-config > ~/.config/keep/config.yml
```

```yaml
# Storage directory
dir: ~/.keep

# List view columns
list_format:
  - name: id
    label: "Item"
    align: right
  - name: time
    label: "Time"
    align: right
  - name: size
    label: "Size"
    align: right
  - name: tags
    label: "Tags"
    align: left

# Table styling
table_config:
  style: utf8_full
  content_arrangement: dynamic

# Default compression
compression_plugin:
  name: gzip

# Default meta plugins
meta_plugins:
  - name: env
  - name: digest
    options:
      algorithm: sha256

# Server settings
server:
  address: "127.0.0.1"
  port: 21080
  username: "keep"
  password: "secret"
  # Maximum POST body size in bytes (0 = unlimited)
  # max_body_size: 52428800  # 50 MB
  # JWT authentication (takes priority over password)
  # jwt_secret: "my-secret-key"
  # jwt_secret_file: /path/to/jwt_secret
  # TLS (requires tls feature)
  # cert_file: /path/to/cert.pem
  # key_file: /path/to/key.pem

# Client settings
client:
  url: "http://localhost:21080"
  username: "keep"
  password: "secret"
  # Or use JWT token
  # jwt: "eyJhbGciOiJIUzI1NiIs..."

human_readable: true
quiet: false
force: false
```

## Client/Server Mode

Keep supports a client/server architecture where one machine runs a keep server and other machines connect as clients. This is useful for:

- Centralizing stored data across multiple machines
- Sharing items between team members
- Offloading storage to a dedicated server
- Piping data from long-running processes without local storage

### Server Mode

Start an HTTP REST API server:

```sh
# Default: 127.0.0.1:21080
keep --server

# Custom address and port
keep --server --server-address 0.0.0.0 --server-port 8080

# With password authentication
keep --server --server-password mypassword

# With custom username
keep --server --server-username admin --server-password mypassword

# With JWT authentication
keep --server --server-jwt-secret my-secret-key
```

#### JWT Authentication

JWT (JSON Web Token) authentication provides permission-based access control. When a JWT secret is configured, the server validates tokens and checks permission claims for each request.

**Configuration:**

```sh
# Via CLI flag
keep --server --server-jwt-secret my-secret-key

# Via environment variable
export KEEP_SERVER_JWT_SECRET=my-secret-key
keep --server

# Via config file (config.yml)
server:
  jwt_secret: "my-secret-key"

# Via secret file (for Docker/secrets management)
keep --server --server-jwt-secret-file /path/to/secret
```

**Token format:**

JWTs must use HS256 algorithm with the following claims:

| Claim | Type | Required | Description |
|-------|------|----------|-------------|
| `sub` | string | Yes | Subject (client identifier) |
| `exp` | number | Yes | Expiration time (Unix timestamp) |
| `read` | boolean | No | Permission for GET requests (default: false) |
| `write` | boolean | No | Permission for POST/PUT requests (default: false) |
| `delete` | boolean | No | Permission for DELETE requests (default: false) |

**Permission mapping:**

| HTTP Method | Required Permission |
|-------------|-------------------|
| `GET` | `read` |
| `POST`, `PUT`, `PATCH` | `write` |
| `DELETE` | `delete` |

**Example token payload:**

```json
{
  "sub": "ci-pipeline",
  "exp": 1735689600,
  "read": true,
  "write": true,
  "delete": false
}
```

**Generating tokens:**

The server does not generate tokens — use any JWT library or tool:

```sh
# Using jwt-cli (https://github.com/mike-engel/jwt-cli)
jwt encode --secret my-secret-key \
  --exp=$(date -d '+24 hours' +%s) \
  '{"sub":"my-client","read":true,"write":true,"delete":false}'

# Using Python
python3 -c "
import jwt, time
token = jwt.encode({
    'sub': 'my-client',
    'exp': int(time.time()) + 86400,
    'read': True, 'write': True, 'delete': False
}, 'my-secret-key', algorithm='HS256')
print(token)
"
```

**Using tokens:**

```sh
# With curl
curl -H "Authorization: Bearer <jwt-token>" http://localhost:21080/api/item/

# The keep client uses --client-jwt for JWT tokens
keep --client-url http://server:21080 --client-jwt <jwt-token> --save my-tag
```

**Response codes:**

| Code | Meaning |
|------|---------|
| `200` | Authorized |
| `401` | Missing, invalid, or expired token |
| `403` | Valid token but insufficient permissions |

**Notes:**

- When `jwt_secret` is set, password authentication is disabled — all requests must present a valid JWT Bearer token
- JWT and password authentication are mutually exclusive — when both `jwt_secret` and `password` are configured, only JWT is used
- Permission fields default to `false` if omitted — tokens must explicitly grant permissions
- JWT authentication requires the `server` feature (jsonwebtoken is included automatically)

#### HTTPS / TLS

Build with the `tls` feature to enable HTTPS:

```sh
cargo build --release --features server,tls
```

Provide a TLS certificate and private key (both PEM format):

```sh
# Via CLI flags
keep --server \
  --server-cert /path/to/cert.pem \
  --server-key /path/to/key.pem

# Via environment variables
export KEEP_SERVER_CERT=/path/to/cert.pem
export KEEP_SERVER_KEY=/path/to/key.pem
keep --server

# Via config file (config.yml)
server:
  cert_file: /path/to/cert.pem
  key_file: /path/to/key.pem
```

When cert and key are provided, the server listens with HTTPS. Without them, it falls back to plain HTTP. The port is controlled by `--server-port` (default: 21080).

**Self-signed certificates** (for development):

```sh
# Generate a self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
  -days 365 -nodes -subj "/CN=localhost"

# Start server with self-signed cert
keep --server --server-cert cert.pem --server-key key.pem

# Connect client with HTTPS
keep --client-url https://localhost:21080 --save my-tag
```

The server accepts data from both dumb clients (raw HTTP/curl) and smart clients (the keep CLI).

#### Server Streaming

The server streams all data through fixed-size buffers (8192 bytes). At no point is the entire file content held in memory.

- **POST**: Body streams through the compression and storage pipeline in chunks. When `max_body_size` is exceeded, the server returns `413 PAYLOAD_TOO_LARGE` while keeping the partial item already saved through the pipeline.
- **GET**: Content streams from disk through decompression to the client using the same fixed-size buffers.
- **Diff**: Individual items are capped at 10 MB for the diff endpoint to prevent unbounded memory use.

##### Max Body Size

Control the maximum accepted body size with:

```sh
# Via CLI flag (bytes)
keep --server --server-max-body-size 52428800

# Via environment variable
export KEEP_SERVER__MAX_BODY_SIZE=52428800
keep --server

# Via config file (config.yml)
server:
  max_body_size: 52428800  # 50 MB
```

When set to `0` or omitted, no limit is enforced.

#### Server Query Parameters

The server supports query parameters that control processing:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `tags` | none | Comma-separated tags |
| `metadata` | none | JSON-encoded metadata |
| `compress` | `true` | `false` = client already compressed, store as-is |
| `meta` | `true` | `false` = client handles metadata, skip server-side plugins |
| `decompress` | `true` | `false` = return raw compressed bytes on GET |

When using a smart client, these are set automatically. For curl, the server handles everything by default.

#### Example: Curl as a Dumb Client

```sh
# Save (server handles compression and metadata)
curl -X POST -d "my data" http://localhost:21080/api/item/?tags=my-tag

# Retrieve (server decompresses)
curl http://localhost:21080/api/item/1/content

# Save compressed (client handles compression, server skips)
gzip -c data.txt | curl -X POST -d @- "http://localhost:21080/api/item/?compress=false&tags=my-tag"
```

### Client Mode

The keep CLI can connect to a remote server as a smart client. Build with the `client` feature:

```sh
cargo build --release --features client
```

```sh
# Set server URL via flag or environment
keep --client-url http://server:21080 --save my-tag
export KEEP_CLIENT_URL=http://server:21080

# With password authentication
keep --client-url http://server:21080 --client-password mypassword --save my-tag
export KEEP_CLIENT_PASSWORD=mypassword

# With custom username
keep --client-url http://server:21080 --client-username admin --client-password mypassword --save my-tag

# With JWT authentication
keep --client-url http://server:21080 --client-jwt <jwt-token> --save my-tag
export KEEP_CLIENT_JWT=<jwt-token>
```

#### How Client Mode Works

Client mode uses **local plugins** and **remote storage**:

1. **Save**: Local compression and metadata plugins run on the client; compressed data streams to the server
2. **Get**: Server sends raw compressed data; client decompresses locally and applies filters
3. **Other operations** (list, info, delete, diff): Delegated directly to the server

This means client behavior is consistent with local mode — the same compression settings and filters apply.

#### Streaming Architecture

Client save uses a 3-thread streaming pipeline for constant memory usage regardless of data size:

```
┌──────────────┐     OS pipe      ┌────────────────┐
│ Reader thread ├──────────────────┤ Streamer thread│
│              │  (compressed     │                │
│ stdin → tee  │   bytes)         │ pipe → POST    │
│    → hash    │                  │   (chunked)    │
│    → compress│                  │                │
└──────────────┘                  └────────────────┘
        │                                │
        ▼                                ▼
    stdout +                    Server stores blob
    SHA-256 digest
```

- **Reader thread**: Reads stdin, tees output to stdout, computes SHA-256, compresses data, writes to OS pipe
- **Streamer thread**: Reads compressed bytes from pipe, streams to server via chunked HTTP POST
- **Main thread**: After streaming completes, sends computed metadata (digest, hostname, size) to server

Memory usage is O(PIPESIZE) — typically 8 KB — regardless of how much data is being stored.

#### Example: Remote Pipeline

```sh
# On a build server, pipe logs to a central keep server
make build 2>&1 | keep --client-url http://logserver:21080 \
  --save build-logs \
  --meta project=myapp \
  --meta branch=$(git branch --show-current)

# Retrieve from any machine
keep --client-url http://logserver:21080 --get build-logs

# List recent builds from a specific project
keep --client-url http://logserver:21080 --list --meta project=myapp
```

### API Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/status` | System status |
| `GET` | `/api/plugins/status` | Plugin status |
| `GET` | `/api/item/` | List items (`tags`, `order`, `start`, `count` params) |
| `POST` | `/api/item/` | Create item (body: raw content, params: `tags`, `metadata`, `compress`, `meta`) |
| `GET` | `/api/item/latest/content` | Latest item content |
| `GET` | `/api/item/latest/meta` | Latest item metadata |
| `GET` | `/api/item/{id}` | Item info by ID |
| `GET` | `/api/item/{id}/content` | Item content by ID |
| `GET` | `/api/item/{id}/meta` | Item metadata by ID |
| `GET` | `/api/item/{id}/info` | Item info by ID |
| `POST` | `/api/item/{id}/meta` | Add metadata to existing item (body: JSON object) |
| `DELETE` | `/api/item/{id}` | Delete item by ID |
| `GET` | `/api/diff` | Diff two items (`id_a`, `id_b` params) |

#### Authentication

The server supports three authentication modes:

**1. Password (HTTP Basic auth):**

```sh
# Default username is "keep"
curl -u keep:mypassword http://localhost:21080/api/status

# Custom username
curl -u admin:mypassword http://localhost:21080/api/status
```

**2. JWT (permission-based):**

```sh
# Valid JWT with read permission allows GET requests
curl -H "Authorization: Bearer <jwt-token>" http://localhost:21080/api/item/
```

See [JWT Authentication](#jwt-authentication) for token format and configuration.

**3. No authentication:**

When neither password nor JWT secret is configured, authentication is disabled.

#### Swagger UI

Build with the `swagger` feature to enable OpenAPI documentation:

```sh
cargo build --features server,swagger
```

Swagger UI available at `/swagger`, OpenAPI spec at `/openapi.json`.

#### Security

The server applies the following security measures:

- **Input validation**: Item IDs are validated as positive integers; tags and metadata have length limits (256 and 128 characters respectively).
- **XSS protection**: All user-controlled data rendered into HTML pages is escaped.
- **Security headers**: Responses include `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, and `Referrer-Policy: strict-origin-when-cross-origin`.
- **CORS**: Explicit allowed headers (`Content-Type`, `Authorization`, `Accept`); no wildcard headers.
- **Path traversal**: Item IDs are validated to prevent directory traversal attacks.
- **Internal errors**: Internal error details are never exposed in HTML responses — only generic messages are shown.

## Shell Integration

Source `profile.bash` to enable shell integration:

```sh
source /path/to/keep/profile.bash
```

This provides:

- **`keep` function** — Captures the current command in metadata automatically
- **`@` alias** — Shorthand for `keep --save`
- **`@@` alias** — Shorthand for `keep --get`

```sh
# Save with automatic command capture
curl -s api.example.com | @ api-response

# Quick retrieve
@@ api-response
```

## Feature Flags

| Feature | Default | Description |
|---------|---------|-------------|
| `magic` | Yes | File type detection via libmagic |
| `lz4` | Yes | LZ4 compression (internal) |
| `gzip` | Yes | GZip compression (internal) |
| `server` | No | HTTP REST API server |
| `tls` | No | HTTPS/TLS server support (requires `server`) |
| `client` | No | HTTP client for remote server |
| `swagger` | No | Swagger UI for API docs |
| `bzip2` | No | BZip2 compression (external program) |
| `xz` | No | XZ compression (external program) |
| `zstd` | No | ZStd compression (external program) |

```sh
# Server with Swagger UI
cargo build --features server,swagger

# Server with HTTPS
cargo build --features server,tls

# Client only
cargo build --features client

# Everything
cargo build --features server,tls,client,swagger,magic
```

## License

MIT License - see [LICENSE](LICENSE) for details.

## Contact

Andrew Phillips - andrew@gt0.ca