refactor: rename plugin features with type prefix for consistency

- Plugin features now use type_ prefix (meta_magic, filter_grep, etc.)
- Added meta_all_musl and filter_all_musl for MUSL-compatible builds
- grep filter plugin made optional via filter_grep feature flag
- Removed regex crate from grep-related code, uses strip_prefix instead
- Updated CHANGELOG.md with breaking change documentation

.dockerignore

@@ -0,0 +1,5 @@
+target/
+.git/
+*.db
+keep.db
+bin/

.gitignore

@@ -1,2 +1,5 @@
 /target
 .aider*
+.crush
+keep.db
+bin/

AGENTS.md

@@ -0,0 +1,65 @@
+# Agent Configuration
+
+**IMPORTANT:** `xxx | keep | zzz` must be as performant as possible in all situations.
+
+## Build/Test Commands
+
+**IMPORTANT**: Do not run the application, start the web server, or the trunk server.
+**IMPORTANT:** Cargo commands cannot be run in parallel. Prefix all commands with `TERM=dumb`.
+
+```bash
+TERM=dumb cargo check              # Fast compile check
+TERM=dumb cargo build              # Build project
+TERM=dumb cargo test               # Run all tests
+TERM=dumb cargo test test_name     # Run specific test by name substring
+TERM=dumb cargo test -- --nocapture  # Verbose test output
+TERM=dumb cargo fmt --check        # Check formatting
+TERM=dumb cargo fmt                # Apply formatting
+TERM=dumb cargo clippy -- -D warnings  # Lint (warnings are errors)
+TERM=dumb cargo build --release    # Release build
+TERM=dumb cargo build --features server  # With server feature
+```
+
+## Code Conventions
+
+- `anyhow::Result` for error handling; `thiserror` for custom error types (`src/services/error.rs`)
+- Plugin traits: `CompressionEngine`, `FilterPlugin`, `MetaPlugin`
+- Dynamic trait objects use `clone_box()` for `Clone` on `Box<dyn Trait>`
+- Plugin registration uses `ctor` constructors at module load time
+- Filter plugins must implement `filter()`, `clone_box()`, and `options()`
+- Meta plugins extend `BaseMetaPlugin` for boilerplate reduction
+- Enum string representations: `#[strum(serialize_all = "snake_case")]`
+- Lint rules: `deny(clippy::all)`, `deny(unsafe_code)` (except `libc::umask` in main.rs)
+- Feature flags: `default = ["magic", "lz4", "gzip"]`; optional: `server`, `swagger`
+
+## Testing
+
+- Tests in `src/tests/` mirroring `src/` structure; shared helpers in `src/tests/common/test_helpers.rs`
+- Key helpers: `create_temp_dir()`, `create_temp_db()`, `test_compression_engine()`
+- Test naming: `test_<feature>_<scenario>`
+
+## Streaming Constraint
+
+**At no point should the whole file be in memory at once.** All I/O must use fixed-size buffers:
+
+- `PIPESIZE` = 8192 bytes (`src/common/mod.rs:10`)
+- Server POST body streams through `save_item_raw_streaming` via `MpscReader`
+- Server GET content streams via streaming reader (not `read_to_end`)
+- When `max_body_size` is exceeded, return `413` but keep the partial item (nonfatal by design)
+- Filter/meta plugins use `PIPESIZE`-sized buffers
+
+## HTML Rendering
+
+- Use `html_escape` crate for all user-controlled data in HTML pages
+- `esc()` for text content, `esc_attr()` for HTML attributes
+- Security headers middleware: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin`
+
+## Changelog
+
+The project uses [Keep a Changelog](https://keepachangelog.com/). The changelog lives at `CHANGELOG.md` in the project root.
+
+- **Always update `CHANGELOG.md`** when making changes that affect users (new features, breaking changes, bug fixes, etc.)
+- Add entries under the `[Unreleased]` section using these categories: `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, `Security`
+- Keep descriptions concise and user-focused — what changed from the user's perspective, not implementation details
+- Commit changelog updates in the same commit as the feature/fix they document
+- Before releasing a new version, move `[Unreleased]` entries to a versioned section (e.g., `[0.2.0] - YYYY-MM-DD`) and add a new empty `[Unreleased]` above it

CHANGELOG.md

@@ -0,0 +1,107 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+- New `filter_grep` feature to optionally include the grep filter plugin (regex-based line filtering). Disabling this feature removes the `regex` crate and its ~800 KiB dependency stack from the binary.
+- New `meta_all_musl` feature for all MUSL-compatible meta plugins (excludes `meta_magic` which requires libmagic)
+- New `filter_all_musl` feature for all MUSL-compatible filter plugins
+- Database index on `items(ts)` column for faster ORDER BY sorting
+- Server API `ItemInfo` now includes `file_size` — actual filesystem-reported size of the item data file
+
+### Changed
+
+- CLI args now feature-gated: `--server` and related options hidden when built without `server` feature; `--client-*` options hidden when built without `client` feature. Run `--help` only shows relevant options.
+- `server` Cargo feature now includes TLS support by default (`axum-server`); `tls` feature removed
+- Clap `conflicts_with_all` removed from all mode args — exclusivity now handled by implicit `group("mode")`
+- Filter plugins check size before loading content into memory (prevents OOM on large inputs)
+- Status page pre-allocates collections with known capacities (meta plugins, compression info)
+- `#[inline]` on HTML escape helper functions (`esc`, `esc_attr`) for hot path performance
+- Removed `once_cell` crate (replaced with `std::sync::LazyLock` from Rust 1.80)
+- Removed `lazy_static` crate (replaced with `std::sync::LazyLock`)
+
+### Breaking
+
+- Plugin feature flags renamed with type prefix for consistency:
+  - `magic` → `meta_magic`
+  - `infer` → `meta_infer`
+  - `tree_magic_mini` → `meta_tree_magic_mini`
+  - `tokens` → `meta_tokens`
+  - `grep` → `filter_grep`
+  - `all-meta-plugins` → `meta_all`
+  - `all-filter-plugins` → `filter_all`
+
+### Fixed
+
+- CLI help text typo: "metatdata" → "metadata" in `--get` and `--info` descriptions
+
+### Refactored
+
+- Added module-level documentation to `services/` module
+
+### Documentation
+
+- README.md: Fixed compression table — zstd is native (not external), "none" renamed to "raw"
+- DESIGN.md: Updated schema to reflect current `items` table columns and meta plugin inventory
+
+## [0.1.0] - 2026-03-21
+
+### Added
+
+- Streaming tar-based export (`--export`) producing `.keep.tar` archives without loading entire files into memory
+- Streaming tar-based import (`--import`) extracting `.keep.tar` archives with new IDs
+- Server endpoints `GET /api/export` and `POST /api/import`
+- ID-based filtering for `--list` (`keep -l 1 2 3` lists specific items by ID)
+- Server API accepts optional `ids` query parameter on `GET /api/item/`
+- `--ids-only` flag for `--list` mode for scripting
+- `infer` and `tree_magic_mini` meta plugins for MIME type detection
+- Native `zstd` compression plugin as default
+- Configurable compression via `--compression` flag
+- Export/import modes with format detection (JSON, YAML, binary)
+- `XDG_CONFIG_HOME` support for default config file location
+- `XDG_DATA_HOME` support for default storage location
+- Tilde (`~`) expansion in config file paths
+
+### Changed
+
+- `CompressionType::None` renamed to `CompressionType::Raw` (with `"none"` as alias for backward compatibility)
+- `items.size` column renamed to `items.uncompressed_size`
+- Added `items.compressed_size` column tracking compressed file size on disk
+- Added `items.closed` column tracking whether an item is fully written
+- Default `list_format` in config now matches CLI default (7 vs 5 columns)
+- All filter plugins share deduplicated option implementations
+
+### Refactored
+
+- Extracted `spawn_body_reader()` and `check_binary_content()` helpers for streaming uploads
+- Extracted `yaml_value_to_string()` helper for meta plugins
+- Extracted `item_path()` helper in `ItemService` to reduce path duplication
+- Unified `get_item_meta_name`/`value` to take `&str` instead of `String`
+- Shared `ItemInfo` struct between client and server
+- Compression service now returns `Result` types instead of panicking via `.expect()`
+- `ApiResponse::ok()` and `ApiResponse::empty()` constructors
+- `meta_filter()` helper on `Settings` for consistent filtering
+- Added `tag_names()` method on `ItemWithMeta`
+- `filter_clone_box!` macro for filter plugin cloning
+
+### Fixed
+
+- Panic guards in diff, compression engine, and spawned threads
+- Pre-existing borrow errors in export handler and `TryFrom` implementation
+- TOCTOU race in `stream_raw_content_response`
+- Swallowed write errors in meta plugins (digest, magic_file, exec)
+- Truncated uploads (413) now properly store compressed data
+- `term::stderr().unwrap()` panic in `item_service`
+- `.unwrap()` panics in compression engine `Read`/`Write` impls
+- Client API errors now propagate to user instead of being swallowed
+- Import endpoint returns 413 on `max_body_size` instead of truncating
+- `keep --list` uses `list_format` from config in all modes
+- All tables respect `table_config` from settings
+- `DisplayListItem` struct removed (was unused)
+- `#[serde(alias = "size")]` on `ImportMeta` for backward compatibility

Cargo.toml

@@ -2,53 +2,129 @@
 name = "keep"
 version = "0.1.0"
 edition = "2024"
+rust-version = "1.85"
 description = "Keep and manage temporary files with automatic compression and metadata generation"
 readme = "README.md"
+license = "MIT"
+repository = "https://gitea.gt0.ca/asp/keep"
+keywords = ["cli", "files", "compression", "metadata"]
 categories = ["command-line-utilities"]
 
-[[test]]
-name = "tests"
-path = "src/tests.rs"
-
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
-
 [dependencies]
-anyhow = "1.0.72"
-clap = { version = "4.3.10", features = ["derive", "env"] }
-directories = "5.0.1"
-lazy_static = "1.4.0"
-libc = "0.2.147"
-log = "0.4.19"
-rusqlite = { version = "0.29.0", features = ["bundled", "array", "chrono"] }
-rusqlite_migration = "1.0.2"
-stderrlog = "0.5.4"
-strum_macros = "0.25"
-strum = { version = "0.25", features = ["derive"] }
-prettytable-rs = "0.10.0"
-chrono = "0.4.26"
-gethostname = "0.4.3"
-humansize = "2.1.3"
-enum-map = "2.6.1"
-inventory = "0.3"
-is-terminal = "0.4.9"
-term = "0.7.0"
-lz4_flex = "0.11.1"
-flate2 = { version = "1.0.27", features = ["zlib-ng-compat"] }
-regex = "1.9.5"
-nix = "0.26.2"
-sha2 = "0.10.0"
-local-ip-address = "0.5.5"
-dns-lookup = "2.0.2"
-uzers = "0.11.3"
-serde = { version = "1.0.219", features = ["derive"] }
-serde_json = "1.0.142"
-serde_yaml = "0.9.34"
-tokio = { version = "1.0", features = ["full"] }
-axum = "0.7"
-tower = "0.4"
-tower-http = { version = "0.5", features = ["cors", "fs", "trace"] }
+anyhow = "1.0"
+axum = { version = "0.8", optional = true }
+derive_more = { version = "2.0", features = ["full"] }
+smart-default = "0.7"
+thiserror = "2.0"
+base64 = "0.22"
+chrono = { version = "0.4", features = ["serde"] }
+clap = { version = "4.6", features = ["derive", "env"] }
+clap_complete = "4"
+command-fds = "0.3"
+config = "0.15"
+ctor = "0.2"
+directories = "6.0"
+dns-lookup = "3.0"
+enum-map = "2.7"
+flate2 = { version = "1.0", features = ["zlib-ng-compat"], optional = true }
+futures = "0.3"
+gethostname = "1.0"
+humansize = "2.1"
+async-stream = "0.3"
 hyper = { version = "1.0", features = ["full"] }
+http-body-util = "0.1"
+inventory = "0.3"
+is-terminal = "0.4"
+libc = "0.2"
+local-ip-address = "0.6"
+log = "0.4"
+lz4_flex = { version = "0.12", optional = true }
+zstd = { version = "0.13", optional = true }
+magic = { version = "0.13", optional = true }
+infer = { version = "0.19", optional = true }
+tree_magic_mini = { version = "3.2", optional = true }
+nix = { version = "0.30", features = ["fs", "process"] }
+comfy-table = "7.2"
+pwhash = "1.0"
+regex = { version = "1.10", optional = true }
+ringbuf = "0.4"
+rusqlite = { version = "0.37", features = ["bundled", "array", "chrono"] }
+rusqlite_migration = "2.3"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+serde_yaml = "0.9"
+sha2 = "0.10"
+md5 = "0.7"
+subtle = "2.6"
+env_logger = "0.11"
+strfmt = "0.2"
+strum = { version = "0.27", features = ["derive"] }
+term = "1.2"
+tokio = { version = "1.0", features = ["full"] }
+tokio-stream = "0.1"
+tokio-util = "0.7"
+tower = { version = "0.5", optional = true }
+tower-http = { version = "0.6", features = ["cors", "fs", "trace"], optional = true }
+utoipa = { version = "5.4", features = ["axum_extras"], optional = true }
+utoipa-swagger-ui = { version = "9.0", features = ["axum"], optional = true }
+uzers = "0.12"
+which = "8.0"
+xdg = "2.5"
+strip-ansi-escapes = "0.2"
+tar = "0.4"
+pest = "2.8"
+pest_derive = "2.8"
+dirs = "6.0"
+similar = { version = "2.7", default-features = false, features = ["text"] }
+html-escape = "0.2"
+ureq = { version = "3", features = ["json"], optional = true }
+os_pipe = { version = "1", optional = true }
+axum-server = { version = "0.8", features = ["tls-rustls"], optional = true }
+jsonwebtoken = { version = "10", optional = true, features = ["aws_lc_rs"] }
+tiktoken-rs = { version = "0.9", optional = true }
+tempfile = "3.3"
+
+[features]
+# Default features include core compression engines plugins that support MUSL
+default = [
+        "client",
+        "gzip",
+        "filter_grep",
+        "meta_infer",
+        "lz4",
+        "meta_tokens",
+        "meta_tree_magic_mini",
+        "zstd"
+]
+
+# Server feature (includes axum and TLS/HTTPS via axum-server; rustls already available via client/ureq)
+server = ["dep:axum", "dep:tower", "dep:tower-http", "dep:utoipa", "dep:jsonwebtoken", "dep:axum-server"]
+
+# Compression features
+gzip = ["flate2"]
+lz4 = ["lz4_flex"]
+bzip2 = []
+xz = []
+zstd = ["dep:zstd"]
+
+# Meta plugin features
+meta_magic = ["dep:magic"]
+meta_infer = ["dep:infer"]
+meta_tree_magic_mini = ["dep:tree_magic_mini"]
+meta_tokens = ["dep:tiktoken-rs"]
+meta_all = ["meta_magic", "meta_infer", "meta_tree_magic_mini", "meta_tokens"]
+meta_all_musl = ["meta_infer", "meta_tree_magic_mini", "meta_tokens"]
+
+# Filter plugin features
+filter_grep = ["dep:regex"]
+filter_all = ["filter_grep"]
+filter_all_musl = ["filter_grep"]
+
+# Swagger UI feature
+swagger = ["dep:utoipa-swagger-ui"]
+
+# Client feature (HTTP client for remote server)
+client = ["dep:ureq", "dep:os_pipe"]
 
 [dev-dependencies]
-tempfile = "3.3.0"
-
+rand = "0.9"

DESIGN.md

@@ -31,8 +31,9 @@
 - `modes/info.rs` - Show detailed item information
 - `modes/diff.rs` - Compare two items
 - `modes/status.rs` - Show system status and capabilities
-- `modes/server.rs` - REST HTTP server mode with OpenAPI documentation
-- `modes/common.rs` - Shared utilities for all modes
+- `modes/server.rs` - REST HTTP/HTTPS server mode with OpenAPI documentation
+- `modes/client.rs` - Client mode for remote server (streaming save, local decompression)
+- `modes/common.rs` - Shared utilities for all modes (OutputFormat, table creation, `print_serialized`, `build_path_table`, `ensure_default_tag`, `render_item_info_table`, `render_list_table_with_format`)
 
 ### Database Module
 - `db.rs` - SQLite database operations
@@ -47,17 +48,195 @@
 - `compression_engine/none.rs` - No compression implementation
 - `compression_engine/program.rs` - External program wrapper
 
-### Digest Functionality
-- Digest functionality is now integrated into meta plugins
-- SHA-256 and other digest algorithms are implemented as meta plugins
-- External digest programs are supported through meta plugin program wrapper
-
 ### Meta Plugin Module
-- `meta_plugin.rs` - Trait and type definitions
+- `meta_plugin.rs` - Trait and type definitions, `SaveMetaFn` callback type
 - `meta_plugin/program.rs` - External program wrapper
 - `meta_plugin/digest.rs` - Internal digest implementations
+- `meta_plugin/system.rs` - System information metadata plugins
 
-### Plugins Module
+**SaveMetaFn Architecture**: Meta plugins are decoupled from direct DB access via a `SaveMetaFn` callback (`Arc<Mutex<dyn FnMut(&str, &str) + Send>>`). The callback is injected at `MetaService` construction and propagated to all plugins via `BaseMetaPlugin`. This enables:
+- **Local mode**: Callback collects metadata into a `Vec`, written to DB after plugins finish
+- **Client mode**: Callback collects into a `HashMap`, sent to server after streaming completes
+- **Server mode**: Callback collects into a `Vec`, written to DB after plugins finish (same as local)
+
+### Common Modules
+- `common/is_binary.rs` - Binary file detection utilities
+- `common/status.rs` - Status information generation
+- `common/mod.rs` - `PIPESIZE` constant (8192), `stream_copy()` streaming utility
+
+### Client Module
+- `client.rs` - HTTP client wrapper (ureq-based, supports streaming POST)
+- `modes/client/save.rs` - 3-thread streaming save with local meta plugins (stdin → tee → compress → meta plugins → pipe → HTTP POST)
+- `modes/client/get.rs` - Get with server-side raw fetch + local decompression
+- `modes/client/list.rs` - List delegation to server
+- `modes/client/info.rs` - Info delegation to server
+- `modes/client/delete.rs` - Delete delegation to server
+- `modes/client/diff.rs` - Diff delegation to server
+- `modes/client/status.rs` - Status delegation to server
+- `modes/client/update.rs` - Update delegation to server (sends plugin names/metadata/tags)
+
+### Utility Modules
 - `plugins.rs` - Shared plugin utilities
-- Contains `ProgramWriter` for external process communication
+- `args.rs` - CLI argument definitions
 
+## Command Line Interface
+
+### Modes
+- Save mode: `keep [--save]` (default when no mode specified and no IDs provided)
+- Get mode: `keep [--get] <ID|tag...>` (default when IDs provided)
+- List mode: `keep [--list] [tag...]`
+- Info mode: `keep [--info] <ID|tag...>`
+- Delete mode: `keep [--delete] <ID...>`
+- Update mode: `keep [--update] <ID> [tag...]`
+- Diff mode: `keep [--diff] <ID1> <ID2>`
+- Status mode: `keep [--status]`
+- Server mode: `keep [--server] <address:port>`
+
+### Item Options
+- `--meta KEY[=VALUE]` - Set metadata for the item, remove if VALUE not provided
+- `--digest <sha256|md5>` - Digest algorithm to use when saving items
+- `--compression <lz4|gzip|bzip2|xz|zstd|none>` - Compression algorithm to use when saving items
+- `--meta-plugins <plugin[,plugin...]>` - Meta plugins to use when saving items
+
+### General Options
+- `--dir <PATH>` - Specify the directory to use for storage
+- `--list-format <FORMAT>` - A comma separated list of columns to display with --list
+- `--human-readable` - Display file sizes with units
+- `--verbose` - Increase message verbosity
+- `--quiet` - Do not show any messages
+- `--output-format <table|json|yaml>` - Output format for info, status, and list modes
+- `--server-password <PASSWORD>` - Password for server authentication
+- `--server-cert <PATH>` - TLS certificate file (PEM) for HTTPS server
+- `--server-key <PATH>` - TLS private key file (PEM) for HTTPS server
+- `--force` - Force output even when binary data would be sent to a TTY
+
+### Client Options (requires `client` feature)
+- `--client-url <URL>` - Remote keep server URL
+- `--client-password <PASSWORD>` - Remote server password
+
+## Data Storage
+
+### Database Schema
+- `items` table: id (primary key), ts (timestamp), uncompressed_size (optional), compressed_size (optional), closed (boolean), compression
+- `tags` table: id (foreign key to items), name (tag name)
+- `metas` table: id (foreign key to items), name (meta key), value (meta value)
+- Indexes on tag names and meta names for faster queries
+
+### File Storage
+- Data directory contains compressed item files named by their item ID
+- Database file stored in data directory
+- File permissions set to be private to user (umask 077)
+
+## REST API Endpoints
+
+### Status Operations
+- `GET /api/status` - Get system status information
+- `GET /api/plugins/status` - Get plugin status information
+
+### Item Operations
+- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest|oldest`, `start=0`, `count=100`, `tags=tag1,tag2`
+- `POST /api/item/` - Add a new item (body: raw content, **streamed** through fixed-size 8192-byte buffers). Query params: `tags`, `metadata` (JSON), `compress=true|false`, `meta=true|false`
+- `POST /api/item/<#>/meta` - Add metadata to an existing item (body: JSON object)
+- `POST /api/item/<#>/update` - Re-run meta plugins on stored content. Query params: `plugins` (comma-separated), `metadata` (JSON), `tags` (comma-separated, idempotent)
+- `DELETE /api/item/<#>` - Delete an item
+- `GET /api/item/latest` - Return the latest item as JSON. Optional params: `tags=tag1,tag2`, `allow_binary=true|false`
+- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags=tag1,tag2`
+- `GET /api/item/latest/content` - Return the raw content of the latest item (**streamed**). Optional params: `tags=tag1,tag2`, `decompress=true|false`
+- `GET /api/item/<#>` - Return the item as JSON. Optional params: `allow_binary=true|false`
+- `GET /api/item/<#>/meta` - Return the item metadata as JSON
+- `GET /api/item/<#>/content` - Return the raw content of the item (**streamed**). Optional params: `decompress=true|false`
+- `GET /api/diff` - Diff two items. Params: `id_a`, `id_b` (individual items capped at 10 MB)
+
+### Server Configuration
+- `max_body_size` - Maximum POST body size in bytes (default: unlimited). When exceeded, server returns `413 PAYLOAD_TOO_LARGE` while keeping the partial item already saved through the streaming pipeline. Set to `0` for unlimited.
+
+### Server Modes
+- **Plain HTTP** (default): `tokio::net::TcpListener` + `axum::serve()`
+- **HTTPS** (with `tls` feature): `axum_server::bind_rustls()` with rustls when `--server-cert` and `--server-key` are provided
+- Conditional selection at startup: cert+key present → HTTPS, otherwise → HTTP
+
+### Client/Server Protocol
+- Smart clients (keep CLI) set `compress=false` and `meta=false` on POST, handling compression and meta plugins locally
+- Dumb clients (curl) use defaults (`compress=true`, `meta=true`), server handles everything
+- Smart client update: sends `plugins` param to server, server runs plugins on stored content (avoids downloading compressed data)
+- GET responses include `X-Keep-Compression` header when `decompress=false`
+- Streaming save uses chunked transfer encoding for constant memory usage
+- **Universal streaming**: All server paths (POST, GET, diff) use `PIPESIZE` (8192) byte buffers
+- **413 partial item**: When `max_body_size` is exceeded, the server returns `413` but keeps the partial item already saved through the pipeline (nonfatal design — pipes continue normally)
+
+### Authentication
+- Bearer token authentication: `Authorization: Bearer <password>`
+- Basic authentication: `Authorization: Basic base64(keep:<password>)`
+- When no password is set, authentication is disabled
+
+## Supported Compression Types
+- LZ4 (internal implementation)
+- GZip (internal implementation)
+- BZip2 (external program)
+- XZ (external program)
+- ZStd (external program)
+- None (no compression)
+
+## Supported Meta Plugins
+
+Meta plugins collect metadata during item save. Each plugin produces one or more key-value pairs:
+
+- `magic_file` - File type detection using libmagic (when `magic` feature enabled)
+- `infer` - MIME type detection using infer crate (when `infer` feature enabled)
+- `tree_magic_mini` - MIME type detection using tree_magic_mini (when `tree_magic_mini` feature enabled)
+- `tokens` - LLM token counting using tiktoken (when `tokens` feature enabled)
+- `text` - Text analysis: line count, word count, char count, line average length
+- `digest` - SHA-256 and MD5 checksums
+- `hostname` - System hostname (full and short)
+- `cwd` - Current working directory
+- `user` - Current username and UID
+- `shell` - Shell path from SHELL environment variable
+- `shell_pid` - Shell process ID from PPID
+- `keep_pid` - Keep process ID
+- `env` - Arbitrary environment variables (via `KEEP_META_ENV_*` prefix)
+- `exec` - Execute external commands for custom metadata
+- `read_time` - Time taken to read content
+- `read_rate` - Content read rate (bytes/second)
+
+## Testing Strategy
+- Unit tests for each module in `src/tests/`
+- Integration tests for modes
+- Database tests for CRUD operations
+- Compression engine tests for each supported format
+- Meta plugin tests for each plugin type
+- Server tests for API endpoints and authentication
+- Common utilities tests for helper functions
+
+## Binary Data Handling
+- Automatic binary detection using file signatures and heuristics
+- Prevents binary data output to TTY unless --force is used
+- Binary meta plugin analyzes content to determine if it's binary
+- API endpoints respect binary flags to prevent accidental binary transmission
+
+## Security Considerations
+- File permissions are restricted to user only (umask 077)
+- Input validation for item IDs to prevent path traversal
+- Authentication for server mode with bearer or basic auth
+- TLS/HTTPS support via rustls when certificate and key are provided
+- Proper resource cleanup using RAII patterns
+- Safe handling of external processes with proper stdin/stdout management
+- **Streaming architecture**: All server I/O uses fixed-size 8192-byte buffers; no full file contents held in memory
+- **XSS protection**: All user-controlled data in HTML pages is escaped via `html-escape`
+- **Security headers**: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin`
+- **CORS**: Explicit allowed headers only (`Content-Type`, `Authorization`, `Accept`); no wildcard headers
+- **Input limits**: Tags (256 chars), metadata keys (128 chars), metadata values (4096 chars), pagination (10,000 max)
+- **Config file size**: 4 KB cap with `from_utf8_lossy` for safe UTF-8 handling
+- **Error sanitization**: Internal errors never exposed in HTML responses
+- **No `unsafe_code`**: Enforced via `#![deny(unsafe_code)]` (exceptions: `libc::umask` in main.rs, `unsafe impl Send` for `SendCookie` in magic_file.rs)
+
+## Feature Flags
+- `server` - HTTP REST API server (axum-based)
+- `tls` - HTTPS/TLS support for server (axum-server + rustls)
+- `client` - HTTP client for remote server (ureq-based, includes streaming save)
+- `swagger` - OpenAPI/Swagger UI documentation
+- `magic` - File type detection via libmagic
+- `lz4` - LZ4 compression (internal)
+- `gzip` - GZip compression (internal)
+- `bzip2` - BZip2 compression (external)
+- `xz` - XZ compression (external)
+- `zstd` - ZStd compression (external)

Dockerfile

@@ -0,0 +1,67 @@
+# Build stage
+FROM rust:1.88-slim AS builder
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    cmake \
+    curl \
+    make \
+    gcc \
+    musl-tools \
+    pkg-config \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN rustup target add x86_64-unknown-linux-musl
+
+WORKDIR /app
+
+# Copy manifests and fetch dependencies (cached layer)
+COPY Cargo.toml Cargo.lock ./
+RUN mkdir src && echo 'fn main() {}' > src/main.rs && echo '' > src/lib.rs
+RUN cargo fetch --target x86_64-unknown-linux-musl
+
+# Copy real source and build static binary
+# magic feature excluded (requires shared libmagic; fallback uses `file` command)
+COPY src/ src/
+RUN cargo build --release --target x86_64-unknown-linux-musl \
+    --no-default-features --features lz4,gzip,server,swagger,client,tls \
+    && strip target/x86_64-unknown-linux-musl/release/keep
+
+# Runtime stage - scratch since binary is fully static
+FROM scratch
+
+COPY --from=builder /app/target/x86_64-unknown-linux-musl/release/keep /keep
+COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs/
+
+EXPOSE 21080
+
+# General options
+# ENV KEEP_CONFIG=/config/config.yml
+# Mount a volume for persistent storage: -v keep-data:/data
+ENV KEEP_DIR=/data
+ENV KEEP_LIST_FORMAT="id,time,size,tags,meta:hostname"
+
+# Item options
+# ENV KEEP_COMPRESSION=lz4
+# ENV KEEP_META_PLUGINS=""
+# ENV KEEP_FILTERS=""
+
+# Server options
+ENV KEEP_SERVER_ADDRESS=0.0.0.0
+ENV KEEP_SERVER_PORT=21080
+# ENV KEEP_SERVER_USERNAME="keep"
+# ENV KEEP_SERVER_PASSWORD=""
+# ENV KEEP_SERVER_PASSWORD_HASH=""
+# ENV KEEP_SERVER_JWT_SECRET=""
+# ENV KEEP_SERVER_JWT_SECRET_FILE=/config/jwt_secret
+
+# TLS options
+# ENV KEEP_SERVER_CERT=/certs/cert.pem
+# ENV KEEP_SERVER_KEY=/certs/key.pem
+
+# Client options
+# ENV KEEP_CLIENT_URL=""
+# ENV KEEP_CLIENT_USERNAME="keep"
+# ENV KEEP_CLIENT_PASSWORD=""
+# ENV KEEP_CLIENT_JWT=""
+
+ENTRYPOINT ["/keep", "--server"]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Andrew Phillips
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

PLAN.md

@@ -1,81 +0,0 @@
-# Code Quality Issues and Fixes
-
-## Critical Issues
-
-### 1. Memory Safety & Resource Leaks - DONE
-**Files affected:** `src/modes/diff.rs`, `src/compression_engine/program.rs`
-**Functions affected:** `mode_diff()`, `CompressionEngineProgram::open()`, `CompressionEngineProgram::create()`
-**Problem example:** Raw file descriptors converted with `unsafe { std::fs::File::from_raw_fd(fd_write) }` without proper cleanup on errors
-**Fix example:** Use RAII wrappers or ensure proper cleanup in Drop implementations and error paths
-
-### 2. Error Handling Problems - DONE
-**Files affected:** `src/modes/save.rs`, `src/modes/update.rs`, `src/db.rs`
-**Functions affected:** `mode_save()`, `mode_update()`, `get_item()`, `insert_item()`
-**Problem example:** `item.id.unwrap()` can panic if item.id is None
-**Fix example:** Replace with `item.id.ok_or_else(|| anyhow!("Item missing ID"))?`
-
-### 3. Concurrency Issues - DONE
-**Files affected:** `src/modes/diff.rs`, `src/meta_plugin/digest.rs`
-**Functions affected:** `mode_diff()`, meta plugin `update()` methods
-**Problem example:** In `mode_diff()`, if writer threads panic, resources may not be cleaned up properly: `writer_thread_a.join()` only propagates panic but doesn't ensure file descriptors are closed
-**Fix example:** Use RAII guards or ensure cleanup in panic handlers: `let _fd_guard = FileDescriptorGuard::new(fd_write);`
-
-## Design Problems
-
-### 4. Database Design Issues - DONE
-**Files affected:** `src/db.rs`, `src/modes/save.rs`, `src/modes/update.rs`
-**Functions affected:** `insert_item()`, `update_item()`, `store_meta()`, `set_item_tags()`
-**Problem example:** Multiple database operations without transactions can leave partial state
-**Fix example:** Wrap related operations in `conn.transaction()` blocks
-
-### 5. Plugin Architecture Flaws
-**Files affected:** `src/meta_plugin.rs`, `src/meta_plugin/digest.rs`, `src/meta_plugin/program.rs`
-**Functions affected:** `MetaPlugin::create()`, `MetaPlugin::update()`, `MetaPlugin::finalize()`
-**Problem example:** 
-- `create()` returns dummy writer that's never used, inconsistent with actual usage pattern
-- `MetaPluginProgram::finalize()` spawns new process instead of reusing existing one
-- No validation that meta plugins produce valid output formats
-- Plugin errors are silently ignored in save operations
-**Fix example:** 
-- Remove `create()` method and rely only on `update()`/`finalize()` pattern
-- Reuse single process per plugin instance for better performance
-- Add output validation and proper error propagation
-
-### 6. Security Concerns - DONE
-**Files affected:** `src/main.rs`, `src/modes/get.rs`, `src/modes/delete.rs`
-**Functions affected:** `main()`, `mode_get()`, `mode_delete()`
-**Problem example:** Item IDs used directly in file paths without validation: `item_path.push(item_id.to_string())`
-**Fix example:** Validate item IDs are positive integers and sanitize file paths
-
-## Performance Issues
-
-### 7. Inefficient Operations
-**Files affected:** `src/modes/save.rs`, `src/compression_engine.rs`
-**Functions affected:** `mode_save()`, `CompressionEngine::size()`
-**Problem example:** Fixed BUFSIZ buffer (typically 8KB) may not be optimal for all scenarios, especially large files or fast storage
-**Fix example:** Use adaptive buffer sizing based on file size or storage characteristics, or allow configuration via environment variable
-
-### 8. I/O Problems
-**Files affected:** `src/meta_plugin/program.rs`, `src/compression_engine/program.rs`
-**Functions affected:** `MetaPluginProgram::finalize()`, `CompressionEngineProgram::open()`, `CompressionEngineProgram::create()`
-**Problem example:** Meta plugin processes can block indefinitely if they hang or produce large output without proper timeouts
-**Fix example:** Add timeouts to process operations and non-blocking I/O for meta plugins: `process.wait_timeout(Duration::from_secs(30))`
-
-## Code Quality Issues
-
-### 9. Error Messages
-**Files affected:** `src/modes/common.rs`, `src/main.rs`
-**Functions affected:** `cmd_args_digest_type()`, `cmd_args_compression_type()`, `main()`
-**Problem example:** `format!("Unknown digest type: {}", digest_name)` exposes internal terminology
-**Fix example:** `format!("Invalid digest algorithm '{}'. Use 'sha256' or 'md5'", digest_name)`
-
-### 10. Code Organization
-**Files affected:** `src/modes/save.rs`, `src/modes/diff.rs`
-**Functions affected:** `mode_save()`, `mode_diff()`
-**Problem example:** Large functions doing multiple responsibilities
-**Fix example:** Split into smaller functions:
-- `src/modes/save.rs: mode_save()` → `setup_compression_and_plugins()`, `process_input_stream()`, `finalize_meta_plugins()`, `save_item_to_database()`
-- `src/modes/diff.rs: mode_diff()` → `validate_diff_args()`, `setup_diff_pipes()`, `spawn_writer_threads()`, `execute_diff_command()`, `handle_diff_output()`
-- `src/modes/diff.rs: write_item_to_pipe()` → `open_item_reader()`, `copy_item_data()`
-
-

README.md

@@ -0,0 +1,957 @@
+# Keep
+
+A command-line utility for storing and retrieving temporary data with automatic compression, metadata extraction, and querying. Pipe any output into `keep` for organized storage — no more losing data in `/tmp` files with cryptic names.
+
+```sh
+# Instead of this:
+curl -s https://api.example.com/data > /tmp/api-data.json
+
+# Do this:
+curl -s https://api.example.com/data | keep --save api-data
+keep --get api-data
+```
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Usage](#usage)
+  - [Save Mode](#save-mode)
+  - [Get Mode](#get-mode)
+  - [List Mode](#list-mode)
+  - [Info Mode](#info-mode)
+  - [Update Mode](#update-mode)
+  - [Delete Mode](#delete-mode)
+  - [Diff Mode](#diff-mode)
+  - [Status Mode](#status-mode)
+- [Filters](#filters)
+- [Compression](#compression)
+- [Meta Plugins](#meta-plugins)
+- [Configuration](#configuration)
+- [Client/Server Mode](#clientserver-mode)
+  - [Server Mode](#server-mode)
+  - [Client Mode](#client-mode)
+  - [API Endpoints](#api-endpoints)
+- [Shell Integration](#shell-integration)
+- [Feature Flags](#feature-flags)
+- [License](#license)
+
+## Features
+
+- **Store and retrieve** — Save content with tags, retrieve by ID or tag
+- **Automatic compression** — LZ4, GZip, BZip2, XZ, ZStd support
+- **Metadata plugins** — Auto-extract file type, digests, hostname, user info, and more
+- **Filters** — Apply transformations (head, tail, grep, strip ANSI) on retrieval
+- **Querying** — List, search, diff items with flexible formatting
+- **Client/server architecture** — Optional HTTP server with streaming support
+- **Modular design** — Extensible plugin system for compression, metadata, and filtering
+
+## Installation
+
+### From Source
+
+Requires Rust and Cargo.
+
+```sh
+cargo build --release
+```
+
+### Install via Cargo
+
+```sh
+cargo install --path .
+```
+
+### Static Binary (Linux)
+
+```sh
+./build-static.bash
+# Binary at bin/keep
+```
+
+### Environment Module
+
+A TCL modulefile is provided at `modulefile`. To use it, copy or symlink the project directory into your modules path:
+
+```sh
+# Symlink into an existing module path (e.g., /usr/local/modules)
+ln -s /path/to/keep /usr/local/modules/keep
+
+# Load the module
+module load keep
+
+# Verify
+keep --status
+
+# Source the shell profile (optional, for shell integration)
+source $KEEP_BASH_PROFILE    # bash
+source $KEEP_ZSH_PROFILE     # zsh
+source $KEEP_SH_PROFILE      # sh/dash/ksh
+source $KEEP_CSH_PROFILE     # csh/tcsh
+```
+
+The modulefile prepends `keep/bin` to `PATH` and sets shell-specific profile variables:
+
+| Variable | Profile | Shell |
+|----------|---------|-------|
+| `KEEP_BASH_PROFILE` | `profile.bash` | bash |
+| `KEEP_ZSH_PROFILE` | `profile.zsh` | zsh |
+| `KEEP_SH_PROFILE` | `profile.sh` | sh, dash, ksh93, pdksh, mksh |
+| `KEEP_CSH_PROFILE` | `profile.csh` | csh, tcsh |
+
+### Shell Completion
+
+Tab completion is available for `bash`, `zsh`, `fish`, `elvish`, and `powershell`. Completions for `@` (save) and `@@` (get) are available for `bash` and `zsh` only.
+
+**Bash** — add to `~/.bashrc`:
+
+```sh
+. <(keep --generate-completion bash)
+```
+
+**Zsh** — add to `~/.zshrc`:
+
+```sh
+. <(keep --generate-completion zsh)
+```
+
+**With `profile.bash` or `profile.zsh`**: Completions for `keep`, `@` (save), and `@@` (get) are loaded automatically when sourcing the profile.
+
+### Build with Server/Client Features
+
+```sh
+# Server only
+cargo build --release --features server
+
+# Client only (for connecting to a remote keep server)
+cargo build --release --features client
+
+# Server + client + all optional features
+cargo build --release --features server,client,swagger
+```
+
+## Quick Start
+
+```sh
+# Save content with a tag (--save is optional when piping)
+echo "Hello, world!" | keep greeting
+
+# Retrieve by ID (--get is optional for numeric IDs)
+keep 1
+
+# Retrieve by tag (--get is required for tags)
+keep --get greeting
+
+# List all stored items
+keep --list
+
+# Get item details
+keep --info greeting
+
+# Delete by ID
+keep --delete 1
+```
+
+### Real-World Examples
+
+```sh
+# Save API response
+curl -s https://api.github.com/repos/user/repo | keep --save repo-info
+
+# Save test output with metadata
+npm test 2>&1 | keep --save test-results --meta project=myapp --meta env=staging
+
+# Chain commands: process and store
+cat data.csv | sort | uniq | keep --save cleaned-data
+
+# Diff two versions
+keep --diff 1 5
+
+# Get first 20 lines of an item
+keep --get 1 --filters "head_lines(20)"
+
+# List items from a specific project
+keep --list --meta project=myapp
+```
+
+## Usage
+
+### Save Mode
+
+Save stdin content with tags and metadata. The `--save` flag is optional when piping content.
+
+```sh
+# Save (auto-assigned ID, no tag)
+echo "data" | keep --save
+
+# Save with a tag (--save is optional when piping)
+echo "data" | keep --save my-tag
+echo "data" | keep my-tag
+
+# Save with multiple tags and metadata
+cat report.pdf | keep --save report --meta project=alpha --meta env=prod
+
+# Specify compression
+echo "data" | keep --save my-tag --compression gzip
+```
+
+Tags and metadata make items easy to find later. Tags are simple identifiers; metadata is key-value pairs.
+
+### Get Mode
+
+Retrieve items by ID. This is the default mode when numeric IDs are provided.
+
+```sh
+# Get by ID (no --get needed for numeric IDs)
+keep --get 1
+keep 1
+
+# Get by tag (requires --get flag)
+keep --get my-tag
+
+# Get with filters applied
+keep --get 1 --filters "head_lines(10)"
+
+# Get by metadata filter
+keep --get --meta project=alpha
+
+# Force binary output to TTY (override safety check)
+keep --get 1 --force
+```
+
+### List Mode
+
+List stored items with filtering and formatting.
+
+```sh
+# List all items
+keep --list
+
+# List by tag
+keep --list my-tag
+
+# Filter by metadata
+keep --list --meta env=prod
+
+# Custom column format
+keep --list --list-format "id,time,size,tags"
+
+# JSON output for scripting
+keep --list --output-format json
+
+# Human-readable file sizes
+keep --list --human-readable
+```
+
+### Info Mode
+
+Show detailed information about an item.
+
+```sh
+keep --info 1
+keep --info my-tag
+keep --info --meta key=value
+```
+
+### Update Mode
+
+Update an item's tags, metadata, and re-run meta plugins.
+
+```sh
+# Replace tags
+keep --update 1 new-tag
+
+# Update metadata
+keep --update 1 --meta key=newvalue
+
+# Remove a metadata key
+keep --update 1 --meta key
+
+# Re-run meta plugins on stored content
+keep --update 1 --meta-plugin digest --meta-plugin text
+```
+
+### Delete Mode
+
+Delete items by ID.
+
+```sh
+keep --delete 1
+keep --delete 1 2 3
+```
+
+### Diff Mode
+
+Show differences between two items.
+
+```sh
+keep --diff 1 2
+```
+
+### Status Mode
+
+Show system status and supported features.
+
+```sh
+keep --status
+keep --status-plugins
+keep --status --verbose
+```
+
+## Filters
+
+Apply transformations to item content during retrieval. Filters are chained with `|`.
+
+```sh
+# First 10 lines
+keep --get 1 --filters "head_lines(10)"
+
+# Skip first 5 lines, then grep for errors
+keep --get 1 --filters "skip_lines(5)|grep(pattern=error)"
+
+# Strip ANSI escape codes
+keep --get 1 --filters "strip_ansi"
+
+# Last 100 bytes
+keep --get 1 --filters "tail_bytes(100)"
+
+# Complex chain
+keep --get 1 --filters "skip_lines(10)|grep(pattern=TODO)|head_lines(5)"
+```
+
+### Available Filters
+
+| Filter | Description | Parameters |
+|--------|-------------|------------|
+| `head_bytes(n)` | First n bytes | `count` |
+| `head_lines(n)` | First n lines | `count` |
+| `tail_bytes(n)` | Last n bytes | `count` |
+| `tail_lines(n)` | Last n lines | `count` |
+| `skip_bytes(n)` | Skip first n bytes | `count` |
+| `skip_lines(n)` | Skip first n lines | `count` |
+| `grep(pattern)` | Filter matching lines | `pattern` (regex) |
+| `strip_ansi` | Remove ANSI escape codes | none |
+
+Set `KEEP_FILTERS` to apply a default filter chain to all retrievals.
+
+## Compression
+
+Items are compressed automatically on save. Default: LZ4.
+
+| Algorithm | Type | Speed | Ratio |
+|-----------|------|-------|-------|
+| `lz4` | Internal | Fastest | Lower |
+| `gzip` | Internal | Fast | Good |
+| `bzip2` | External | Slow | Better |
+| `xz` | External | Slowest | Best |
+| `zstd` | Internal | Fast | Good |
+| `raw` | Internal | N/A | N/A |
+
+```sh
+# Specify compression per item
+echo "data" | keep --save my-tag --compression zstd
+
+# Set default via environment
+export KEEP_COMPRESSION=gzip
+```
+
+External compression programs (`bzip2`, `xz`, `zstd`) must be installed on the system.
+
+## Meta Plugins
+
+Metadata is automatically extracted when saving items.
+
+| Plugin | Key | Description |
+|--------|-----|-------------|
+| `env` | `*` | Capture `KEEP_META_*` environment variables |
+| `magic_file` | `file_type` | File type detection (requires `magic` feature) |
+| `text` | `text_line_count`, `text_word_count` | Line and word counts |
+| `user` | `user_uid`, `user_name`, `user_gid`, `user_group` | Current user info |
+| `shell` | `shell` | Current shell path |
+| `shell_pid` | `shell_pid` | Shell process ID |
+| `keep_pid` | `keep_pid` | Keep process ID |
+| `digest` | `digest_sha256`, `digest_md5` | Content digests |
+| `read_time` | `read_time` | Time to read content |
+| `read_rate` | `read_rate` | Data read rate |
+| `hostname` | `hostname`, `hostname_short` | System hostname |
+| `exec` | Custom | Run external commands for metadata |
+| `cwd` | `cwd` | Current working directory |
+
+```sh
+# Use specific plugins (repeatable)
+echo "data" | keep --save tag --meta-plugin digest --meta-plugin text --meta-plugin user
+
+# Pass options to a plugin via JSON
+echo "data" | keep --save tag --meta-plugin 'tokens:{"options":{"min_length":"2"}}'
+
+# Capture custom metadata via environment
+KEEP_META_project=alpha echo "data" | keep --save tag
+
+# Combine environment and CLI metadata
+KEEP_META_build=1234 echo "data" | keep --save tag --meta env=staging
+```
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `KEEP_DIR` | Storage directory | `~/.keep` |
+| `KEEP_CONFIG` | Config file path | `~/.config/keep/config.yml` |
+| `KEEP_COMPRESSION` | Compression algorithm | `lz4` |
+| `KEEP_META_PLUGINS` | Meta plugins to use (JSON format: `name[:{json}]`, comma-separated) | `env` |
+| `KEEP_FILTERS` | Default filter chain | none |
+| `KEEP_LIST_FORMAT` | List column format | built-in defaults |
+| `KEEP_SERVER_ADDRESS` | Server bind address | `127.0.0.1` |
+| `KEEP_SERVER_PORT` | Server port | `21080` |
+| `KEEP_SERVER_USERNAME` | Server Basic auth username | `keep` |
+| `KEEP_SERVER_PASSWORD` | Server password | none |
+| `KEEP_SERVER_PASSWORD_HASH` | Server password hash | none |
+| `KEEP_SERVER_JWT_SECRET` | JWT secret for token auth | none |
+| `KEEP_SERVER_JWT_SECRET_FILE` | Path to JWT secret file | none |
+| `KEEP_SERVER_MAX_BODY_SIZE` | Maximum POST body size in bytes (0=unlimited) | unlimited |
+| `KEEP_SERVER_CERT` | TLS certificate file path (PEM) | none |
+| `KEEP_SERVER_KEY` | TLS private key file path (PEM) | none |
+| `KEEP_CLIENT_URL` | Remote keep server URL | none |
+| `KEEP_CLIENT_USERNAME` | Remote server username | `keep` |
+| `KEEP_CLIENT_PASSWORD` | Remote server password | none |
+| `KEEP_CLIENT_JWT` | JWT token for remote server | none |
+
+Any config setting can be overridden with `KEEP__<SETTING>` environment variables (double underscore separator).
+
+### Configuration File
+
+Default location: `~/.config/keep/config.yml`
+
+Generate a default configuration:
+
+```sh
+keep --generate-config > ~/.config/keep/config.yml
+```
+
+```yaml
+# Storage directory
+dir: ~/.keep
+
+# List view columns
+list_format:
+  - name: id
+    label: "Item"
+    align: right
+  - name: time
+    label: "Time"
+    align: right
+  - name: size
+    label: "Size"
+    align: right
+  - name: tags
+    label: "Tags"
+    align: left
+
+# Table styling
+table_config:
+  style: utf8_full
+  content_arrangement: dynamic
+
+# Default compression
+compression_plugin:
+  name: gzip
+
+# Default meta plugins
+meta_plugins:
+  - name: env
+  - name: digest
+    options:
+      algorithm: sha256
+
+# Server settings
+server:
+  address: "127.0.0.1"
+  port: 21080
+  username: "keep"
+  password: "secret"
+  # Maximum POST body size in bytes (0 = unlimited)
+  # max_body_size: 52428800  # 50 MB
+  # JWT authentication (takes priority over password)
+  # jwt_secret: "my-secret-key"
+  # jwt_secret_file: /path/to/jwt_secret
+  # TLS (requires tls feature)
+  # cert_file: /path/to/cert.pem
+  # key_file: /path/to/key.pem
+
+# Client settings
+client:
+  url: "http://localhost:21080"
+  username: "keep"
+  password: "secret"
+  # Or use JWT token
+  # jwt: "eyJhbGciOiJIUzI1NiIs..."
+
+human_readable: true
+quiet: false
+force: false
+```
+
+## Client/Server Mode
+
+Keep supports a client/server architecture where one machine runs a keep server and other machines connect as clients. This is useful for:
+
+- Centralizing stored data across multiple machines
+- Sharing items between team members
+- Offloading storage to a dedicated server
+- Piping data from long-running processes without local storage
+
+### Server Mode
+
+Start an HTTP REST API server:
+
+```sh
+# Default: 127.0.0.1:21080
+keep --server
+
+# Custom address and port
+keep --server --server-address 0.0.0.0 --server-port 8080
+
+# With password authentication
+keep --server --server-password mypassword
+
+# With custom username
+keep --server --server-username admin --server-password mypassword
+
+# With JWT authentication
+keep --server --server-jwt-secret my-secret-key
+```
+
+#### JWT Authentication
+
+JWT (JSON Web Token) authentication provides permission-based access control. When a JWT secret is configured, the server validates tokens and checks permission claims for each request.
+
+**Configuration:**
+
+```sh
+# Via CLI flag
+keep --server --server-jwt-secret my-secret-key
+
+# Via environment variable
+export KEEP_SERVER_JWT_SECRET=my-secret-key
+keep --server
+
+# Via config file (config.yml)
+server:
+  jwt_secret: "my-secret-key"
+
+# Via secret file (for Docker/secrets management)
+keep --server --server-jwt-secret-file /path/to/secret
+```
+
+**Token format:**
+
+JWTs must use HS256 algorithm with the following claims:
+
+| Claim | Type | Required | Description |
+|-------|------|----------|-------------|
+| `sub` | string | Yes | Subject (client identifier) |
+| `exp` | number | Yes | Expiration time (Unix timestamp) |
+| `read` | boolean | No | Permission for GET requests (default: false) |
+| `write` | boolean | No | Permission for POST/PUT requests (default: false) |
+| `delete` | boolean | No | Permission for DELETE requests (default: false) |
+
+**Permission mapping:**
+
+| HTTP Method | Required Permission |
+|-------------|-------------------|
+| `GET` | `read` |
+| `POST`, `PUT`, `PATCH` | `write` |
+| `DELETE` | `delete` |
+
+**Example token payload:**
+
+```json
+{
+  "sub": "ci-pipeline",
+  "exp": 1735689600,
+  "read": true,
+  "write": true,
+  "delete": false
+}
+```
+
+**Generating tokens:**
+
+The server does not generate tokens — use any JWT library or tool:
+
+```sh
+# Using jwt-cli (https://github.com/mike-engel/jwt-cli)
+jwt encode --secret my-secret-key \
+  --exp=$(date -d '+24 hours' +%s) \
+  '{"sub":"my-client","read":true,"write":true,"delete":false}'
+
+# Using Python
+python3 -c "
+import jwt, time
+token = jwt.encode({
+    'sub': 'my-client',
+    'exp': int(time.time()) + 86400,
+    'read': True, 'write': True, 'delete': False
+}, 'my-secret-key', algorithm='HS256')
+print(token)
+"
+```
+
+**Using tokens:**
+
+```sh
+# With curl
+curl -H "Authorization: Bearer <jwt-token>" http://localhost:21080/api/item/
+
+# The keep client uses --client-jwt for JWT tokens
+keep --client-url http://server:21080 --client-jwt <jwt-token> --save my-tag
+```
+
+**Response codes:**
+
+| Code | Meaning |
+|------|---------|
+| `200` | Authorized |
+| `401` | Missing, invalid, or expired token |
+| `403` | Valid token but insufficient permissions |
+
+**Notes:**
+
+- When `jwt_secret` is set, password authentication is disabled — all requests must present a valid JWT Bearer token
+- JWT and password authentication are mutually exclusive — when both `jwt_secret` and `password` are configured, only JWT is used
+- Permission fields default to `false` if omitted — tokens must explicitly grant permissions
+- JWT authentication requires the `server` feature (jsonwebtoken is included automatically)
+
+#### HTTPS / TLS
+
+Build with the `tls` feature to enable HTTPS:
+
+```sh
+cargo build --release --features server,tls
+```
+
+Provide a TLS certificate and private key (both PEM format):
+
+```sh
+# Via CLI flags
+keep --server \
+  --server-cert /path/to/cert.pem \
+  --server-key /path/to/key.pem
+
+# Via environment variables
+export KEEP_SERVER_CERT=/path/to/cert.pem
+export KEEP_SERVER_KEY=/path/to/key.pem
+keep --server
+
+# Via config file (config.yml)
+server:
+  cert_file: /path/to/cert.pem
+  key_file: /path/to/key.pem
+```
+
+When cert and key are provided, the server listens with HTTPS. Without them, it falls back to plain HTTP. The port is controlled by `--server-port` (default: 21080).
+
+**Self-signed certificates** (for development):
+
+```sh
+# Generate a self-signed cert
+openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
+  -days 365 -nodes -subj "/CN=localhost"
+
+# Start server with self-signed cert
+keep --server --server-cert cert.pem --server-key key.pem
+
+# Connect client with HTTPS
+keep --client-url https://localhost:21080 --save my-tag
+```
+
+The server accepts data from both dumb clients (raw HTTP/curl) and smart clients (the keep CLI).
+
+#### Server Streaming
+
+The server streams all data through fixed-size buffers (8192 bytes). At no point is the entire file content held in memory.
+
+- **POST**: Body streams through the compression and storage pipeline in chunks. When `max_body_size` is exceeded, the server returns `413 PAYLOAD_TOO_LARGE` while keeping the partial item already saved through the pipeline.
+- **GET**: Content streams from disk through decompression to the client using the same fixed-size buffers.
+- **Diff**: Individual items are capped at 10 MB for the diff endpoint to prevent unbounded memory use.
+
+##### Max Body Size
+
+Control the maximum accepted body size with:
+
+```sh
+# Via CLI flag (bytes)
+keep --server --server-max-body-size 52428800
+
+# Via environment variable
+export KEEP_SERVER__MAX_BODY_SIZE=52428800
+keep --server
+
+# Via config file (config.yml)
+server:
+  max_body_size: 52428800  # 50 MB
+```
+
+When set to `0` or omitted, no limit is enforced.
+
+#### Server Query Parameters
+
+The server supports query parameters that control processing:
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `tags` | none | Comma-separated tags |
+| `metadata` | none | JSON-encoded metadata |
+| `compress` | `true` | `false` = client already compressed, store as-is |
+| `meta` | `true` | `false` = client handles metadata, skip server-side plugins |
+| `decompress` | `true` | `false` = return raw compressed bytes on GET |
+
+The `POST /api/item/{id}/update` endpoint accepts additional parameters:
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `plugins` | none | Comma-separated plugin names to re-run on stored content |
+| `metadata` | none | JSON-encoded metadata overrides to apply |
+| `tags` | none | Comma-separated tags to add (idempotent) |
+
+When using a smart client, these are set automatically. For curl, the server handles everything by default.
+
+#### Example: Curl as a Dumb Client
+
+```sh
+# Save (server handles compression and metadata)
+curl -X POST -d "my data" http://localhost:21080/api/item/?tags=my-tag
+
+# Retrieve (server decompresses)
+curl http://localhost:21080/api/item/1/content
+
+# Save compressed (client handles compression, server skips)
+gzip -c data.txt | curl -X POST -d @- "http://localhost:21080/api/item/?compress=false&tags=my-tag"
+```
+
+### Client Mode
+
+The keep CLI can connect to a remote server as a smart client. Build with the `client` feature:
+
+```sh
+cargo build --release --features client
+```
+
+```sh
+# Set server URL via flag or environment
+keep --client-url http://server:21080 --save my-tag
+export KEEP_CLIENT_URL=http://server:21080
+
+# With password authentication
+keep --client-url http://server:21080 --client-password mypassword --save my-tag
+export KEEP_CLIENT_PASSWORD=mypassword
+
+# With custom username
+keep --client-url http://server:21080 --client-username admin --client-password mypassword --save my-tag
+
+# With JWT authentication
+keep --client-url http://server:21080 --client-jwt <jwt-token> --save my-tag
+export KEEP_CLIENT_JWT=<jwt-token>
+```
+
+#### How Client Mode Works
+
+Client mode uses **local plugins** and **remote storage**:
+
+1. **Save**: Local compression and meta plugins run on the client; compressed data streams to the server. Smart clients set `meta=false` so the server skips its own plugins.
+2. **Get**: Server sends raw compressed data; client decompresses locally and applies filters
+3. **Update**: Meta plugins run on the server to avoid downloading compressed data for re-processing
+4. **Other operations** (list, info, delete, diff): Delegated directly to the server
+
+This means client behavior is consistent with local mode — the same compression settings and filters apply.
+
+#### Streaming Architecture
+
+Client save uses a 3-thread streaming pipeline for constant memory usage regardless of data size:
+
+```
+┌───────────────────┐     OS pipe      ┌────────────────┐
+│ Reader thread     ├──────────────────┤ Streamer thread│
+│                   │  (compressed     │                │
+│ stdin → tee       │   bytes)         │ pipe → POST    │
+│    → hash         │                  │   (chunked)    │
+│    → compress     │                  │                │
+│    → meta plugins │                  │                │
+└───────────────────┘                  └────────────────┘
+        │                                  │
+        ▼                                  ▼
+    stdout +                       Server stores blob
+    computed metadata
+```
+
+- **Reader thread**: Reads stdin, tees output to stdout, computes SHA-256 via digest plugin, compresses data, runs meta plugins (hostname, text, etc.), writes to OS pipe
+- **Streamer thread**: Reads compressed bytes from pipe, streams to server via chunked HTTP POST
+- **Main thread**: After streaming completes, sends plugin-collected metadata to server
+
+Memory usage is O(PIPESIZE) — typically 8 KB — regardless of how much data is being stored.
+
+#### Example: Remote Pipeline
+
+```sh
+# On a build server, pipe logs to a central keep server
+make build 2>&1 | keep --client-url http://logserver:21080 \
+  --save build-logs \
+  --meta project=myapp \
+  --meta branch=$(git branch --show-current)
+
+# Retrieve from any machine
+keep --client-url http://logserver:21080 --get build-logs
+
+# List recent builds from a specific project
+keep --client-url http://logserver:21080 --list --meta project=myapp
+```
+
+### API Endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/status` | System status |
+| `GET` | `/api/plugins/status` | Plugin status |
+| `GET` | `/api/item/` | List items (`tags`, `order`, `start`, `count` params) |
+| `POST` | `/api/item/` | Create item (body: raw content, params: `tags`, `metadata`, `compress`, `meta`) |
+| `GET` | `/api/item/latest/content` | Latest item content |
+| `GET` | `/api/item/latest/meta` | Latest item metadata |
+| `GET` | `/api/item/{id}` | Item info by ID |
+| `GET` | `/api/item/{id}/content` | Item content by ID |
+| `GET` | `/api/item/{id}/meta` | Item metadata by ID |
+| `GET` | `/api/item/{id}/info` | Item info by ID |
+| `POST` | `/api/item/{id}/meta` | Add metadata to existing item (body: JSON object) |
+| `POST` | `/api/item/{id}/update` | Re-run meta plugins on stored content (params: `plugins`, `metadata`, `tags`) |
+| `DELETE` | `/api/item/{id}` | Delete item by ID |
+| `GET` | `/api/diff` | Diff two items (`id_a`, `id_b` params) |
+
+#### Authentication
+
+The server supports three authentication modes:
+
+**1. Password (HTTP Basic auth):**
+
+```sh
+# Default username is "keep"
+curl -u keep:mypassword http://localhost:21080/api/status
+
+# Custom username
+curl -u admin:mypassword http://localhost:21080/api/status
+```
+
+**2. JWT (permission-based):**
+
+```sh
+# Valid JWT with read permission allows GET requests
+curl -H "Authorization: Bearer <jwt-token>" http://localhost:21080/api/item/
+```
+
+See [JWT Authentication](#jwt-authentication) for token format and configuration.
+
+**3. No authentication:**
+
+When neither password nor JWT secret is configured, authentication is disabled.
+
+#### Swagger UI
+
+Build with the `swagger` feature to enable OpenAPI documentation:
+
+```sh
+cargo build --features server,swagger
+```
+
+Swagger UI available at `/swagger`, OpenAPI spec at `/openapi.json`.
+
+#### Security
+
+The server applies the following security measures:
+
+- **Input validation**: Item IDs are validated as positive integers; tags and metadata have length limits (256 and 128 characters respectively).
+- **XSS protection**: All user-controlled data rendered into HTML pages is escaped.
+- **Security headers**: Responses include `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, and `Referrer-Policy: strict-origin-when-cross-origin`.
+- **CORS**: Explicit allowed headers (`Content-Type`, `Authorization`, `Accept`); no wildcard headers.
+- **Path traversal**: Item IDs are validated to prevent directory traversal attacks.
+- **Internal errors**: Internal error details are never exposed in HTML responses — only generic messages are shown.
+
+## Shell Integration
+
+Profile scripts are provided for several shells. Source the appropriate one to enable shell integration:
+
+| Profile | Shells | Features |
+|---------|--------|----------|
+| `profile.bash` | bash | Preexec hook, wrapper function, `@`/`@@` aliases, tab completions |
+| `profile.zsh` | zsh | Preexec hook, wrapper function, `@`/`@@` aliases, tab completions |
+| `profile.sh` | sh, dash, ksh93, pdksh, mksh | Wrapper function, `@`/`@@` aliases |
+| `profile.csh` | csh, tcsh | Alias-based `keep` wrapper, `@`/`@@` aliases |
+
+```sh
+# bash
+source /path/to/keep/profile.bash
+
+# zsh
+source /path/to/keep/profile.zsh
+
+# sh, dash, ksh
+source /path/to/keep/profile.sh
+
+# csh/tcsh
+source /path/to/keep/profile.csh
+```
+
+All profiles provide:
+
+- **`@` alias** — Shorthand for `keep --save`
+- **`@@` alias** — Shorthand for `keep --get`
+
+Bash and zsh profiles additionally provide:
+
+- **`keep` function** — Captures the current command in metadata automatically
+- **Tab completion** — For `keep`, `@`, and `@@`
+
+```sh
+# Save with automatic command capture (bash/zsh)
+curl -s api.example.com | @ api-response
+
+# Quick retrieve
+@@ api-response
+```
+
+## Feature Flags
+
+| Feature | Default | Description |
+|---------|---------|-------------|
+| `magic` | Yes | File type detection via libmagic |
+| `lz4` | Yes | LZ4 compression (internal) |
+| `gzip` | Yes | GZip compression (internal) |
+| `server` | No | HTTP REST API server |
+| `tls` | No | HTTPS/TLS server support (requires `server`) |
+| `client` | No | HTTP client for remote server |
+| `swagger` | No | Swagger UI for API docs |
+| `bzip2` | No | BZip2 compression (external program) |
+| `xz` | No | XZ compression (external program) |
+| `zstd` | No | ZStd compression (external program) |
+
+```sh
+# Server with Swagger UI
+cargo build --features server,swagger
+
+# Server with HTTPS
+cargo build --features server,tls
+
+# Client only
+cargo build --features client
+
+# Everything
+cargo build --features server,tls,client,swagger,magic
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## Contact
+
+Andrew Phillips - andrew@gt0.ca

README.org

@@ -1,141 +0,0 @@
-#+TITLE: Keep
-#+AUTHOR: Andrew Phillips
-
-* Introduction
-Keep is a command-line utility designed to manage temporary files created on the command line. Instead of redirecting output to a temporary file (e.g., =command > ~/whatever.tmp=), you can use =keep= to handle the temporary files for you (e.g., =command | keep=).
-
-* Installation
-To install Keep, you need to have Rust and Cargo installed on your system. You can then build and install Keep using the following commands:
-
-#+BEGIN_SRC sh
-cargo build --release
-cargo install --path .
-#+END_SRC
-
-* Usage
-Keep provides several subcommands to manage temporary files. Below are some examples of how to use Keep.
-
-** Saving an Item
-To save an item with tags and metadata, you can use the =--save= option:
-
-#+BEGIN_SRC sh
-echo "Hello, world!" | keep --save example --meta key=value
-#+END_SRC
-
-** Getting an Item
-To retrieve an item by its ID or by matching tags and metadata, you can use the =--get= option:
-
-#+BEGIN_SRC sh
-keep --get 1
-keep --get example
-keep --get --meta key=value
-keep 1
-keep example
-#+END_SRC
-
-** Listing Items
-To list all items or filter them by tags and metadata, you can use the =--list= option:
-
-#+BEGIN_SRC sh
-keep --list
-keep --list example
-keep --list --meta key=value
-#+END_SRC
-
-** Updating an Item
-To update an item's tags and metadata, you can use the =--update= option:
-
-#+BEGIN_SRC sh
-keep --update 1 newtag --meta key=newvalue
-#+END_SRC
-
-** Deleting an Item
-To delete an item by its ID or by matching tags, you can use the =--delete= option:
-
-#+BEGIN_SRC sh
-keep --delete 1
-keep --delete example
-#+END_SRC
-
-** Showing Status
-To show the status of directories and supported compression algorithms, you can use the =--status= option:
-
-#+BEGIN_SRC sh
-keep --status
-#+END_SRC
-
-** Diffing Items
-To show a diff between two items by ID, you can use the =--diff= option:
-
-#+BEGIN_SRC sh
-keep --diff 1 2
-#+END_SRC
-
-** Getting Information About an Item
-To get detailed information about an item by its ID or by matching tags and metadata, you can use the =--info= option:
-
-#+BEGIN_SRC sh
-keep --info 1
-keep --info example
-keep --info --meta key=value
-#+END_SRC
-
-* Configuration
-Keep can be configured using environment variables and command-line options. The following environment variables are supported:
-
-- =KEEP_DIR=: Specify the directory to use for storage.
-- =KEEP_LIST_FORMAT=: A comma-separated list of columns to display with =--list=.
-- =KEEP_DIGEST=: Digest algorithm to use when saving items.
-- =KEEP_COMPRESSION=: Compression algorithm to use when saving items.
-
-* Examples
-Here are some examples of how to use Keep with different options:
-
-** Saving an Item with Compression and Digest
-#+BEGIN_SRC sh
-echo "Hello, world!" | keep --save example --meta key=value --compression gzip --digest sha256
-#+END_SRC
-
-** Getting an Item with Human-Readable Sizes
-#+BEGIN_SRC sh
-keep --get 1 --human-readable
-#+END_SRC
-
-** Listing Items with Custom Format
-#+BEGIN_SRC sh
-keep --list --list-format "id,time,size,tags,meta:hostname"
-#+END_SRC
-
-** Updating an Item with New Tags and Metadata
-#+BEGIN_SRC sh
-keep --update 1 newtag --meta key=newvalue
-#+END_SRC
-
-** Deleting an Item by Tag
-#+BEGIN_SRC sh
-keep --delete example
-#+END_SRC
-
-** Showing Status with Verbose Output
-#+BEGIN_SRC sh
-keep --status --verbose
-#+END_SRC
-
-** Diffing Items with IDs
-#+BEGIN_SRC sh
-keep --diff 1 2
-#+END_SRC
-
-** Getting Information About an Item with Metadata
-#+BEGIN_SRC sh
-keep --info 1
-#+END_SRC
-
-* License
-Keep is licensed under the MIT License. See the LICENSE file for more details.
-
-* Contributing
-Contributions are welcome! Please open an issue or submit a pull request on the GitHub repository.
-
-* Contact
-For more information, please contact Andrew Phillips at andrew@gt0.ca.

build-static.bash

@@ -2,7 +2,6 @@
 
 set -ex
 
-export RUSTFLAGS='-C target-feature=+crt-static'
-cargo build --release --target x86_64-unknown-linux-gnu
+cargo build --release --target x86_64-unknown-linux-musl
 mkdir -p bin
-cp target/x86_64-unknown-linux-gnu/release/keep ./bin/
+cp target/x86_64-unknown-linux-musl/release/keep ./bin/

docker-compose.yml

@@ -0,0 +1,32 @@
+services:
+  keep:
+    build: .
+    ports:
+      - "21080:21080"
+    volumes:
+      - keep-data:/data
+      - keep-config:/config
+    environment:
+      - KEEP_SERVER_ADDRESS=0.0.0.0
+      - KEEP_SERVER_PORT=21080
+      # - KEEP_SERVER_USERNAME=keep
+      # - KEEP_SERVER_PASSWORD=changeme
+      # - KEEP_SERVER_PASSWORD_HASH=
+      # - KEEP_SERVER_JWT_SECRET=
+      # - KEEP_SERVER_JWT_SECRET_FILE=/config/jwt_secret
+      # - KEEP_COMPRESSION=lz4
+      # - KEEP_META_PLUGINS=
+      # - KEEP_FILTERS=
+      - KEEP_CONFIG=/config/config.yml
+      # - KEEP_SERVER_CERT=/certs/cert.pem
+      # - KEEP_SERVER_KEY=/certs/key.pem
+      # - KEEP_CLIENT_USERNAME=keep
+      # - KEEP_CLIENT_JWT=""
+    restart: unless-stopped
+    # For TLS, mount certificate files:
+    # volumes:
+    #   - ./certs:/certs:ro
+
+volumes:
+  keep-data:
+  keep-config:

modulefile

@@ -14,3 +14,7 @@ set mydir       [ file normalize $mydir_base ]
 module-whatis   Keep
 
 prepend-path    PATH             $mydir/bin
+setenv KEEP_BASH_PROFILE ${mydir}/profile.bash
+setenv KEEP_ZSH_PROFILE  ${mydir}/profile.zsh
+setenv KEEP_SH_PROFILE   ${mydir}/profile.sh
+setenv KEEP_CSH_PROFILE  ${mydir}/profile.csh

profile.bash

@@ -2,28 +2,14 @@
 
 function __keep_preexec {
   KEEP_META_command="$1"
-  KEEP_META_directory=${KEEP_META_directory:-${PWD}}
-  KEEP_META_hostname=${KEEP_META_hostname:-${HOSTNAME:-$(hostname -f)}}
   KEEP_META_tty=${KEEP_META_tty:-$(tty)}
 }
 
 function __keep_preexec_init {
-  local found=false
-  local f
   for f in "${preexec_functions[@]}"; do
-    if [[ $f = __keep_preexec ]]; then
-      found=true
-      break
-    fi
+    [[ $f = __keep_preexec ]] && return
   done
-
-  if [[ $found = false ]]; then
   preexec_functions+=(__keep_preexec)
-  fi
-
-  if [[ -z $KEEP_LIST_FORMAT ]]; then
-    export KEEP_LIST_FORMAT="id,time,size,tags,meta:hostname,meta:command"
-  fi
 }
 
 function keep {
@@ -32,8 +18,6 @@ function keep {
       export KEEP_META_command
     fi
 
-    export KEEP_META_directory
-    export KEEP_META_hostname
     export KEEP_META_tty
 
     exec keep "$@"
@@ -48,4 +32,20 @@ function @@ {
   keep --get "$@"
 }
 
+# Shell completions
+. <(command keep --generate-completion bash)
+
+___keep_complete() {
+    local mode="$1"
+    COMP_WORDS=(keep "$mode" "${COMP_WORDS[@]:1}")
+    COMP_CWORD=$((COMP_CWORD + 1))
+    _keep
+}
+
+___keep_save_completion() { ___keep_complete --save; }
+___keep_get_completion() { ___keep_complete --get; }
+
+complete -F ___keep_save_completion @
+complete -F ___keep_get_completion @@
+
 __keep_preexec_init

profile.csh

@@ -0,0 +1,11 @@
+#!/bin/csh
+# Profile for csh and tcsh.
+# Preexec hooks are not available; KEEP_META_command is not set.
+
+if ( ! $?KEEP_META_tty ) then
+  setenv KEEP_META_tty `tty`
+endif
+
+alias keep 'env KEEP_META_tty=${KEEP_META_tty} command keep \!*'
+alias @ 'keep --save \!*'
+alias @@ 'keep --get \!*'

profile.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+# POSIX-compatible profile for sh, dash, ksh93, pdksh, mksh, and other POSIX shells.
+# Preexec hooks are not available in these shells; KEEP_META_command is not set.
+
+KEEP_META_tty=${KEEP_META_tty:-$(tty)}
+
+keep() {
+  export KEEP_META_tty
+  command keep "$@"
+}
+
+alias @='keep --save'
+alias @@='keep --get'

profile.zsh

@@ -0,0 +1,38 @@
+#!/bin/zsh
+
+autoload -U add-zsh-hook
+
+__keep_preexec() {
+  KEEP_META_command="$1"
+  KEEP_META_tty=${KEEP_META_tty:-$(tty)}
+}
+
+add-zsh-hook preexec __keep_preexec
+
+keep() {
+  if [[ $ZSH_SUBSHELL -le 2 ]]; then
+    export KEEP_META_command
+  fi
+  export KEEP_META_tty
+  command keep "$@"
+}
+
+alias @='keep --save'
+alias @@='keep --get'
+
+# Shell completions
+. <(command keep --generate-completion zsh)
+
+___keep_complete() {
+    local mode="$1"
+    local -a words
+    words=(keep "$mode" "${words[@]:1}")
+    ((CURRENT++))
+    _keep
+}
+
+___keep_save_completion() { ___keep_complete --save; }
+___keep_get_completion() { ___keep_complete --get; }
+
+compdef ___keep_save_completion @
+compdef ___keep_get_completion @@

src/args.rs

@@ -0,0 +1,358 @@
+use std::path::PathBuf;
+use std::str::FromStr;
+
+use clap::*;
+use clap_complete::Shell;
+
+/// Main struct for command-line arguments, parsed via Clap.
+#[derive(Parser, Debug, Clone)]
+#[command(author, version, about, long_about = None)]
+pub struct Args {
+    #[command(flatten)]
+    pub mode: ModeArgs,
+    #[command(flatten)]
+    pub item: ItemArgs,
+    #[command(flatten)]
+    pub options: OptionsArgs,
+
+    #[arg(help("A list of either item IDs or tags"))]
+    #[arg(value_parser = clap::value_parser!(NumberOrString))]
+    #[arg(required = false)]
+    pub ids_or_tags: Vec<NumberOrString>,
+}
+
+/// Struct for mode-specific arguments, defining CLI flags for different operations.
+#[derive(Parser, Debug, Clone)]
+pub struct ModeArgs {
+    #[arg(group("mode"), help_heading("Mode Options"), short, long)]
+    #[arg(help("Save an item using any tags or metadata provided"))]
+    pub save: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), short, long)]
+    #[arg(help("Get an item either by its ID or by a combination of matching tags and metadata"))]
+    pub get: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), long)]
+    #[arg(help("Show a diff between two items by ID"))]
+    pub diff: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), short, long)]
+    #[arg(help("List items, filtering on tags or metadata if given"))]
+    pub list: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), short, long)]
+    #[arg(help("Delete items either by ID or by matching tags"))]
+    #[arg(requires = "ids_or_tags")]
+    pub delete: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), short, long)]
+    #[arg(help("Get an item either by its ID or by a combination of matching tags and metadata"))]
+    pub info: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), short('u'), long)]
+    #[arg(help("Update an item's tags and metadata by ID"))]
+    pub update: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), short('S'), long)]
+    #[arg(help("Show status of directories and supported compression algorithms"))]
+    pub status: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), long)]
+    #[arg(help("Show available plugins and their configurations"))]
+    pub status_plugins: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), long)]
+    #[arg(help("Export items to a .keep.tar archive (requires IDs or tags)"))]
+    pub export: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), long, value_name("FILE"))]
+    #[arg(help("Import items from a .keep.tar archive or legacy .meta.yml file"))]
+    pub import: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(group("mode"), help_heading("Mode Options"), long)]
+    #[arg(help("Start REST HTTP server"))]
+    pub server: bool,
+
+    #[arg(group("mode"), help_heading("Mode Options"), long)]
+    #[arg(help("Generate default configuration and output to stdout"))]
+    pub generate_config: bool,
+
+    #[arg(help_heading("Mode Options"), long)]
+    #[arg(help("Generate shell completion script"))]
+    pub generate_completion: Option<Shell>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_ADDRESS"))]
+    #[arg(help("Server address to bind to"))]
+    pub server_address: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PORT"))]
+    #[arg(help("Server port to bind to"))]
+    pub server_port: Option<u16>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_CERT"))]
+    #[arg(help("Path to TLS certificate file (PEM) for HTTPS"))]
+    pub server_cert: Option<PathBuf>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_KEY"))]
+    #[arg(help("Path to TLS private key file (PEM) for HTTPS"))]
+    pub server_key: Option<PathBuf>,
+}
+
+/// Represents a meta plugin argument with optional JSON config.
+///
+/// Parsed from `name` or `name:{"options":{...},"outputs":{...}}` syntax.
+#[derive(Debug, Clone)]
+pub struct MetaPluginArg {
+    pub name: String,
+    pub options: Option<serde_json::Value>,
+}
+
+impl FromStr for MetaPluginArg {
+    type Err = anyhow::Error;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if let Some((name, json_str)) = s.split_once(':') {
+            let value: serde_json::Value = serde_json::from_str(json_str)
+                .map_err(|e| anyhow::anyhow!("Invalid JSON for meta plugin '{}': {}", name, e))?;
+            Ok(MetaPluginArg {
+                name: name.to_string(),
+                options: Some(value),
+            })
+        } else {
+            Ok(MetaPluginArg {
+                name: s.to_string(),
+                options: None,
+            })
+        }
+    }
+}
+
+/// Represents a metadata key-value argument.
+///
+/// Parsed from `key=value` (set) or `key` (delete/filter by existence).
+#[derive(Debug, Clone)]
+pub enum MetaArg {
+    /// Set metadata with a value.
+    Set { key: String, value: String },
+    /// Bare key without a value (delete in update mode, filter by existence otherwise).
+    Key(String),
+}
+
+impl MetaArg {
+    /// Returns the key.
+    pub fn key(&self) -> &str {
+        match self {
+            MetaArg::Set { key, .. } | MetaArg::Key(key) => key,
+        }
+    }
+
+    /// Returns the value if this is a Set variant.
+    pub fn value(&self) -> Option<&str> {
+        match self {
+            MetaArg::Set { value, .. } => Some(value),
+            MetaArg::Key(_) => None,
+        }
+    }
+}
+
+impl FromStr for MetaArg {
+    type Err = anyhow::Error;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if let Some((key, value)) = s.split_once('=') {
+            Ok(MetaArg::Set {
+                key: key.to_string(),
+                value: value.to_string(),
+            })
+        } else {
+            Ok(MetaArg::Key(s.to_string()))
+        }
+    }
+}
+
+/// Struct for item-specific arguments, such as compression and plugins.
+#[derive(Parser, Debug, Clone)]
+pub struct ItemArgs {
+    #[arg(help_heading("Item Options"), short, long, env("KEEP_COMPRESSION"))]
+    #[arg(help("Compression algorithm to use when saving items"))]
+    pub compression: Option<String>,
+
+    #[arg(
+        help_heading("Item Options"),
+        short('M'),
+        long = "meta-plugin",
+        value_parser = clap::value_parser!(MetaPluginArg),
+        env("KEEP_META_PLUGINS")
+    )]
+    #[arg(help("Meta plugin to use (repeatable): name or name:{json}"))]
+    pub meta_plugins: Vec<MetaPluginArg>,
+
+    #[arg(help_heading("Item Options"), long)]
+    #[arg(help("Metadata key=value to set (or key to delete in --update)"))]
+    pub meta: Vec<String>,
+
+    #[arg(help_heading("Item Options"), long, env("KEEP_FILTERS"))]
+    #[arg(help("Filter string to apply to content when getting items"))]
+    pub filters: Option<String>,
+
+    #[arg(help_heading("Export Options"), long, default_value = "{name}_{ts}")]
+    #[arg(help("Template for export tar filename (appends .keep.tar). Variables: {name} {ts}"))]
+    pub export_filename_format: String,
+
+    #[arg(help_heading("Export Options"), long, value_name("NAME"))]
+    #[arg(help("Export name used for {name} variable (default: export_<common-tags>)"))]
+    pub export_name: Option<String>,
+
+    #[arg(help_heading("Import Options"), long, value_name("DATA_FILE"))]
+    #[arg(help("Data file for import (reads from stdin if omitted)"))]
+    pub import_data_file: Option<PathBuf>,
+}
+
+/// Struct for general options, including verbosity, paths, and output settings.
+#[derive(Parser, Debug, Default, Clone)]
+pub struct OptionsArgs {
+    #[arg(long, env("KEEP_CONFIG"))]
+    #[arg(help("Specify the configuration file to use"))]
+    pub config: Option<PathBuf>,
+
+    #[arg(long, env("KEEP_DIR"))]
+    #[arg(help("Specify the directory to use for storage"))]
+    pub dir: Option<PathBuf>,
+
+    #[arg(
+        long,
+        env("KEEP_LIST_FORMAT"),
+        default_value("id,time,size,meta:text_line_count,tags,meta:hostname_short,meta:command")
+    )]
+    #[arg(help("A comma separated list of columns to display with --list"))]
+    pub list_format: String,
+
+    #[arg(short('H'), long)]
+    #[arg(help("Display file sizes with units"))]
+    pub human_readable: bool,
+
+    #[arg(long)]
+    #[arg(help("Only output item IDs (for scripting)"))]
+    pub ids_only: bool,
+
+    #[arg(short, long, action = clap::ArgAction::Count, conflicts_with("quiet"))]
+    #[arg(help("Increase message verbosity, can be given more than once"))]
+    pub verbose: u8,
+
+    #[arg(short, long)]
+    #[arg(help("Do not show any messages"))]
+    pub quiet: bool,
+
+    #[arg(long, value_enum, default_value("table"))]
+    #[arg(help("Output format (only works with --info, --status, --list)"))]
+    pub output_format: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PASSWORD"))]
+    #[arg(help("Password for server authentication (requires --server)"))]
+    pub server_password: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PASSWORD_HASH"))]
+    #[arg(help("Password hash for server authentication (requires --server)"))]
+    pub server_password_hash: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_USERNAME"))]
+    #[arg(help(
+        "Username for server Basic authentication (requires --server, defaults to 'keep')"
+    ))]
+    pub server_username: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_JWT_SECRET"))]
+    #[arg(help("JWT secret for token-based authentication (requires --server)"))]
+    pub server_jwt_secret: Option<String>,
+
+    #[cfg(feature = "server")]
+    #[arg(
+        help_heading("Server Options"),
+        long,
+        env("KEEP_SERVER_JWT_SECRET_FILE")
+    )]
+    #[arg(help("Path to file containing JWT secret (requires --server)"))]
+    pub server_jwt_secret_file: Option<PathBuf>,
+
+    #[cfg(feature = "server")]
+    #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_MAX_BODY_SIZE"))]
+    #[arg(help("Maximum request body size in bytes (requires --server, default: unlimited)"))]
+    pub server_max_body_size: Option<u64>,
+
+    #[cfg(feature = "client")]
+    #[arg(long, env("KEEP_CLIENT_URL"), help_heading("Client Options"))]
+    #[arg(help("Remote keep server URL for client mode"))]
+    pub client_url: Option<String>,
+
+    #[cfg(feature = "client")]
+    #[arg(long, env("KEEP_CLIENT_PASSWORD"), help_heading("Client Options"))]
+    #[arg(help("Password for remote keep server authentication"))]
+    pub client_password: Option<String>,
+
+    #[cfg(feature = "client")]
+    #[arg(long, env("KEEP_CLIENT_USERNAME"), help_heading("Client Options"))]
+    #[arg(help("Username for remote keep server authentication (defaults to 'keep')"))]
+    pub client_username: Option<String>,
+
+    #[cfg(feature = "client")]
+    #[arg(long, env("KEEP_CLIENT_JWT"), help_heading("Client Options"))]
+    #[arg(help("JWT token for remote keep server authentication"))]
+    pub client_jwt: Option<String>,
+
+    #[arg(
+        long,
+        help("Force output even when binary data would be sent to a TTY")
+    )]
+    pub force: bool,
+}
+
+/// Enum for representing either a number (item ID) or a string (tag).
+#[derive(Debug, Clone)]
+pub enum NumberOrString {
+    Number(i64),
+    Str(String),
+}
+
+impl FromStr for NumberOrString {
+    type Err = Error;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(s.parse::<i64>()
+            .map(NumberOrString::Number)
+            .unwrap_or_else(|_| NumberOrString::Str(s.to_string())))
+    }
+}
+
+/// Validates the parsed arguments based on mode constraints.
+///
+/// # Returns
+///
+/// `Result<(), String>` - Ok if valid, or an error message string.
+impl Args {
+    /// Validate the arguments based on the selected mode
+    pub fn validate(&self) -> Result<(), String> {
+        // Check if --delete is used and ids_or_tags is empty
+        if self.mode.delete && self.ids_or_tags.is_empty() {
+            return Err("At least one ID is required when using --delete".to_string());
+        }
+
+        // Check if --delete is used and any of the ids_or_tags are tags (strings)
+        if self.mode.delete {
+            for item in &self.ids_or_tags {
+                if let NumberOrString::Str(_) = item {
+                    return Err("Tags are not supported for --delete, only IDs".to_string());
+                }
+            }
+        }
+
+        Ok(())
+    }
+}

src/client.rs

@@ -0,0 +1,514 @@
+use crate::services::{ItemInfo, error::CoreError};
+use base64::Engine;
+use serde::de::DeserializeOwned;
+use std::collections::HashMap;
+use std::io::Read;
+
+/// Percent-encode a value for use in a URL query string.
+fn url_encode(s: &str) -> String {
+    let mut result = String::with_capacity(s.len() * 3);
+    for byte in s.bytes() {
+        match byte {
+            b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
+                result.push(byte as char);
+            }
+            _ => {
+                result.push('%');
+                result.push(char::from_digit((byte >> 4) as u32, 16).unwrap());
+                result.push(char::from_digit((byte & 0xF) as u32, 16).unwrap());
+            }
+        }
+    }
+    result
+}
+
+fn append_query_params(url: &mut String, params: &[(&str, &str)]) {
+    if !params.is_empty() {
+        url.push('?');
+        for (i, (key, value)) in params.iter().enumerate() {
+            if i > 0 {
+                url.push('&');
+            }
+            url.push_str(&format!("{}={}", url_encode(key), url_encode(value)));
+        }
+    }
+}
+
+pub struct KeepClient {
+    base_url: String,
+    agent: ureq::Agent,
+    username: Option<String>,
+    password: Option<String>,
+    jwt: Option<String>,
+}
+
+impl KeepClient {
+    pub fn new(
+        base_url: &str,
+        username: Option<String>,
+        password: Option<String>,
+        jwt: Option<String>,
+    ) -> Result<Self, CoreError> {
+        let base_url = base_url.trim_end_matches('/').to_string();
+        let agent = ureq::Agent::new_with_defaults();
+        Ok(Self {
+            base_url,
+            agent,
+            username,
+            password,
+            jwt,
+        })
+    }
+
+    pub fn base_url(&self) -> &str {
+        &self.base_url
+    }
+
+    pub fn username(&self) -> Option<&String> {
+        self.username.as_ref()
+    }
+
+    pub fn password(&self) -> Option<&String> {
+        self.password.as_ref()
+    }
+
+    pub fn jwt(&self) -> Option<&String> {
+        self.jwt.as_ref()
+    }
+
+    fn url(&self, path: &str) -> String {
+        format!("{}{}", self.base_url, path)
+    }
+
+    /// Get the Authorization header value for the current credentials.
+    ///
+    /// JWT token is sent as `Bearer <token>`.
+    /// Password is sent as `Basic base64(username:password)`
+    /// where username defaults to "keep".
+    fn auth_header(&self) -> Option<String> {
+        if let Some(ref jwt) = self.jwt {
+            Some(format!("Bearer {jwt}"))
+        } else if let Some(ref password) = self.password {
+            let username = self.username.as_deref().unwrap_or("keep");
+            let credentials = format!("{username}:{password}");
+            let encoded = base64::engine::general_purpose::STANDARD.encode(&credentials);
+            Some(format!("Basic {encoded}"))
+        } else {
+            None
+        }
+    }
+
+    fn handle_error<T>(&self, result: Result<T, ureq::Error>) -> Result<T, CoreError> {
+        match result {
+            Ok(v) => Ok(v),
+            Err(ureq::Error::StatusCode(code)) => Err(CoreError::Other(anyhow::anyhow!(
+                "Server returned error: HTTP {}",
+                code
+            ))),
+            Err(e) => Err(CoreError::Other(anyhow::anyhow!("Request failed: {}", e))),
+        }
+    }
+
+    pub fn get_json<T: DeserializeOwned>(&self, path: &str) -> Result<T, CoreError> {
+        let url = self.url(path);
+        let mut req = self.agent.get(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        let response = self.handle_error(req.call())?;
+        let body: T = self.handle_error(response.into_body().read_json())?;
+        Ok(body)
+    }
+
+    pub fn get_json_with_query<T: DeserializeOwned>(
+        &self,
+        path: &str,
+        params: &[(&str, &str)],
+    ) -> Result<T, CoreError> {
+        let mut url = self.url(path);
+        append_query_params(&mut url, params);
+        let mut req = self.agent.get(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        let response = self.handle_error(req.call())?;
+        let body: T = self.handle_error(response.into_body().read_json())?;
+        Ok(body)
+    }
+
+    pub fn get_bytes(&self, path: &str) -> Result<Vec<u8>, CoreError> {
+        let url = self.url(path);
+        let mut req = self.agent.get(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        let response = self.handle_error(req.call())?;
+        let mut body = response.into_body();
+        let bytes = body
+            .read_to_vec()
+            .map_err(|e| CoreError::Other(anyhow::anyhow!("{}", e)))?;
+        Ok(bytes)
+    }
+
+    pub fn post_bytes(
+        &self,
+        path: &str,
+        body_bytes: &[u8],
+        params: &[(&str, &str)],
+    ) -> Result<ItemInfo, CoreError> {
+        let mut cursor = std::io::Cursor::new(body_bytes);
+        self.post_stream(path, &mut cursor, params)
+    }
+
+    /// Stream data from a reader to the server using chunked transfer encoding.
+    ///
+    /// The reader is consumed in chunks and sent to the server without buffering
+    /// the entire body in memory. This enables true streaming for large payloads.
+    pub fn post_stream(
+        &self,
+        path: &str,
+        body_reader: &mut dyn Read,
+        params: &[(&str, &str)],
+    ) -> Result<ItemInfo, CoreError> {
+        let mut url = self.url(path);
+        append_query_params(&mut url, params);
+
+        let mut req = self.agent.post(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        req = req.header("Content-Type", "application/octet-stream");
+
+        let response = self.handle_error(req.send(ureq::SendBody::from_reader(body_reader)))?;
+
+        #[derive(serde::Deserialize)]
+        struct ApiResponse {
+            data: Option<ItemInfo>,
+            error: Option<String>,
+        }
+
+        let api_response: ApiResponse = self.handle_error(response.into_body().read_json())?;
+
+        if let Some(error) = api_response.error {
+            return Err(CoreError::Other(anyhow::anyhow!("Server error: {}", error)));
+        }
+
+        api_response
+            .data
+            .ok_or_else(|| CoreError::Other(anyhow::anyhow!("No data in response")))
+    }
+
+    pub fn delete(&self, path: &str) -> Result<(), CoreError> {
+        let url = self.url(path);
+        let mut req = self.agent.delete(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        self.handle_error(req.call())?;
+        Ok(())
+    }
+
+    pub fn get_status(&self) -> Result<crate::common::status::StatusInfo, CoreError> {
+        #[derive(serde::Deserialize)]
+        struct ApiResponse {
+            data: Option<crate::common::status::StatusInfo>,
+            error: Option<String>,
+        }
+        let response: ApiResponse = self.get_json("/api/status")?;
+        response.data.ok_or_else(|| {
+            CoreError::Other(anyhow::anyhow!(
+                "{}",
+                response
+                    .error
+                    .unwrap_or_else(|| "No status data returned".to_string())
+            ))
+        })
+    }
+
+    pub fn get_item_info(&self, id: i64) -> Result<ItemInfo, CoreError> {
+        #[derive(serde::Deserialize)]
+        struct ApiResponse {
+            data: Option<ItemInfo>,
+            error: Option<String>,
+        }
+        let response: ApiResponse = self.get_json(&format!("/api/item/{id}/info"))?;
+        response.data.ok_or_else(|| {
+            CoreError::Other(anyhow::anyhow!(
+                "{}",
+                response
+                    .error
+                    .unwrap_or_else(|| "Item not found".to_string())
+            ))
+        })
+    }
+
+    pub fn list_items(
+        &self,
+        ids: &[i64],
+        tags: &[String],
+        order: &str,
+        start: u64,
+        count: u64,
+        meta: &HashMap<String, Option<String>>,
+    ) -> Result<Vec<ItemInfo>, CoreError> {
+        #[derive(serde::Deserialize)]
+        struct ApiResponse {
+            data: Option<Vec<ItemInfo>>,
+            error: Option<String>,
+        }
+
+        let mut params: Vec<(String, String)> = Vec::new();
+        params.push(("order".to_string(), order.to_string()));
+        params.push(("start".to_string(), start.to_string()));
+        params.push(("count".to_string(), count.to_string()));
+        if !ids.is_empty() {
+            params.push((
+                "ids".to_string(),
+                ids.iter()
+                    .map(|i| i.to_string())
+                    .collect::<Vec<_>>()
+                    .join(","),
+            ));
+        }
+        if !tags.is_empty() {
+            params.push(("tags".to_string(), tags.join(",")));
+        }
+        if !meta.is_empty() {
+            let meta_json = serde_json::to_string(meta).map_err(|e| {
+                CoreError::Other(anyhow::anyhow!("Failed to serialize meta filter: {}", e))
+            })?;
+            params.push(("meta".to_string(), meta_json));
+        }
+
+        let param_refs: Vec<(&str, &str)> = params
+            .iter()
+            .map(|(k, v)| (k.as_str(), v.as_str()))
+            .collect();
+
+        let response: ApiResponse = self.get_json_with_query("/api/item/", &param_refs)?;
+        if let Some(data) = response.data {
+            return Ok(data);
+        }
+        if let Some(err) = response.error {
+            return Err(CoreError::Other(anyhow::anyhow!("Server error: {err}")));
+        }
+        Ok(Vec::new())
+    }
+
+    pub fn save_item(
+        &self,
+        content: &[u8],
+        tags: &[String],
+        metadata: &HashMap<String, String>,
+        compress: bool,
+        meta: bool,
+    ) -> Result<ItemInfo, CoreError> {
+        let mut params: Vec<(String, String)> = Vec::new();
+        if !tags.is_empty() {
+            params.push(("tags".to_string(), tags.join(",")));
+        }
+        if !metadata.is_empty() {
+            let meta_json = serde_json::to_string(metadata).map_err(|e| {
+                CoreError::Other(anyhow::anyhow!("Failed to serialize metadata: {}", e))
+            })?;
+            params.push(("metadata".to_string(), meta_json));
+        }
+        params.push(("compress".to_string(), compress.to_string()));
+        params.push(("meta".to_string(), meta.to_string()));
+
+        let param_refs: Vec<(&str, &str)> = params
+            .iter()
+            .map(|(k, v)| (k.as_str(), v.as_str()))
+            .collect();
+
+        self.post_bytes("/api/item/", content, &param_refs)
+    }
+
+    pub fn delete_item(&self, id: i64) -> Result<(), CoreError> {
+        self.delete(&format!("/api/item/{id}"))
+    }
+
+    /// Add metadata to an existing item.
+    pub fn post_metadata(
+        &self,
+        id: i64,
+        metadata: &HashMap<String, String>,
+    ) -> Result<(), CoreError> {
+        let url = self.url(&format!("/api/item/{id}/meta"));
+        let mut req = self.agent.post(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        req = req.header("Content-Type", "application/json");
+
+        let body = serde_json::to_vec(metadata)
+            .map_err(|e| CoreError::Other(anyhow::anyhow!("Failed to serialize metadata: {e}")))?;
+
+        let mut cursor = std::io::Cursor::new(body);
+        self.handle_error(req.send(ureq::SendBody::from_reader(&mut cursor)))?;
+        Ok(())
+    }
+
+    /// Set the uncompressed size for an item.
+    pub fn set_item_size(&self, id: i64, size: u64) -> Result<(), CoreError> {
+        let url = format!(
+            "{}?uncompressed_size={}",
+            self.url(&format!("/api/item/{id}/update")),
+            url_encode(&size.to_string())
+        );
+        let mut req = self.agent.post(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        self.handle_error(req.send(ureq::SendBody::from_reader(&mut std::io::empty())))?;
+        Ok(())
+    }
+
+    pub fn get_item_content_raw(&self, id: i64) -> Result<(Vec<u8>, String), CoreError> {
+        let (mut reader, compression) = self.get_item_content_stream(id)?;
+        let mut bytes = Vec::new();
+        reader
+            .read_to_end(&mut bytes)
+            .map_err(|e| CoreError::Other(anyhow::anyhow!("{}", e)))?;
+        Ok((bytes, compression))
+    }
+
+    /// Get a streaming reader for item content without decompression.
+    ///
+    /// Returns a reader over the HTTP response body and the compression type
+    /// from the X-Keep-Compression header. The caller can stream through
+    /// decompression readers without buffering the entire file in memory.
+    pub fn get_item_content_stream(&self, id: i64) -> Result<(Box<dyn Read>, String), CoreError> {
+        let url = format!(
+            "{}?decompress=false",
+            self.url(&format!("/api/item/{id}/content"))
+        );
+
+        let mut req = self.agent.get(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+
+        let response = self.handle_error(req.call())?;
+
+        let compression = response
+            .headers()
+            .get("X-Keep-Compression")
+            .and_then(|v| v.to_str().ok())
+            .unwrap_or("raw")
+            .to_string();
+
+        let reader = response.into_body().into_reader();
+        Ok((Box::new(reader), compression))
+    }
+
+    pub fn diff_items(&self, id_a: i64, id_b: i64) -> Result<Vec<String>, CoreError> {
+        #[derive(serde::Deserialize)]
+        struct ApiResponse {
+            data: Option<Vec<String>>,
+        }
+
+        let params = [("id_a", id_a.to_string()), ("id_b", id_b.to_string())];
+        let param_refs: Vec<(&str, &str)> = params.iter().map(|(k, v)| (*k, v.as_str())).collect();
+
+        let response: ApiResponse = self.get_json_with_query("/api/diff", &param_refs)?;
+        Ok(response.data.unwrap_or_default())
+    }
+
+    /// Export items to a tar archive, streaming the response to a file.
+    ///
+    /// # Arguments
+    ///
+    /// * `ids` - Item IDs to export (mutually exclusive with tags).
+    /// * `tags` - Tags to search for items (mutually exclusive with ids).
+    /// * `dest` - Destination file path.
+    pub fn export_items_to_file(
+        &self,
+        ids: &[i64],
+        tags: &[String],
+        dest: &std::path::Path,
+    ) -> Result<(), CoreError> {
+        let mut params: Vec<(String, String)> = Vec::new();
+        if !ids.is_empty() {
+            let id_strs: Vec<String> = ids.iter().map(|id| id.to_string()).collect();
+            params.push(("ids".to_string(), id_strs.join(",")));
+        }
+        if !tags.is_empty() {
+            params.push(("tags".to_string(), tags.join(",")));
+        }
+        let param_refs: Vec<(&str, &str)> = params
+            .iter()
+            .map(|(k, v)| (k.as_str(), v.as_str()))
+            .collect();
+
+        let mut url = self.url("/api/export");
+        append_query_params(&mut url, &param_refs);
+
+        let mut req = self.agent.get(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+
+        let response = self.handle_error(req.call())?;
+        let mut reader = response.into_body().into_reader();
+
+        let mut file = std::fs::File::create(dest).map_err(CoreError::Io)?;
+        let mut buf = [0u8; crate::common::PIPESIZE];
+        loop {
+            let n = reader.read(&mut buf).map_err(CoreError::Io)?;
+            if n == 0 {
+                break;
+            }
+            std::io::Write::write_all(&mut file, &buf[..n]).map_err(CoreError::Io)?;
+        }
+
+        Ok(())
+    }
+
+    /// Import items from a tar archive, streaming the file to the server.
+    ///
+    /// # Arguments
+    ///
+    /// * `tar_path` - Path to the `.keep.tar` file.
+    ///
+    /// # Returns
+    ///
+    /// A list of newly assigned item IDs.
+    pub fn import_tar_file(&self, tar_path: &std::path::Path) -> Result<Vec<i64>, CoreError> {
+        #[derive(serde::Deserialize)]
+        struct ApiResponse {
+            data: Option<ImportResponse>,
+            error: Option<String>,
+        }
+
+        #[derive(serde::Deserialize)]
+        struct ImportResponse {
+            ids: Vec<i64>,
+        }
+
+        let mut file = std::fs::File::open(tar_path).map_err(CoreError::Io)?;
+
+        let url = self.url("/api/import");
+        let mut req = self.agent.post(&url);
+        if let Some(ref auth) = self.auth_header() {
+            req = req.header("Authorization", auth);
+        }
+        req = req.header("Content-Type", "application/x-tar");
+
+        let response = self.handle_error(req.send(ureq::SendBody::from_reader(&mut file)))?;
+
+        let body = response
+            .into_body()
+            .read_to_string()
+            .map_err(|e| CoreError::InvalidInput(format!("Cannot read response: {e}")))?;
+
+        let api_response: ApiResponse = serde_json::from_str(&body)
+            .map_err(|e| CoreError::InvalidInput(format!("Cannot parse response: {e}")))?;
+
+        if let Some(error) = api_response.error {
+            return Err(CoreError::InvalidInput(error));
+        }
+
+        Ok(api_response.data.map(|d| d.ids).unwrap_or_default())
+    }
+}

src/common.rs

@@ -1,217 +0,0 @@
-use std::io::Read;
-
-/// Detect if data is binary or text
-/// Returns true if data is likely binary, false if likely text
-pub fn is_binary(data: &[u8]) -> bool {
-    if data.is_empty() {
-        return false;
-    }
-    
-    // First check for known binary file signatures
-    if has_binary_signature(data) {
-        return true;
-    }
-    
-    // Check for UTF-16 BOM (text)
-    if data.len() >= 2 {
-        if (data[0] == 0xFF && data[1] == 0xFE) || (data[0] == 0xFE && data[1] == 0xFF) {
-            return false; // UTF-16 with BOM is text
-        }
-    }
-    
-    // Check for UTF-8 BOM (text)
-    if data.len() >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF {
-        return false; // UTF-8 with BOM is text
-    }
-    
-    // Check if it's valid UTF-8
-    if std::str::from_utf8(data).is_ok() {
-        // Valid UTF-8, check printable character ratio
-        return calculate_printable_ratio(data) < 0.7;
-    }
-    
-    // Not valid UTF-8, check if it might be UTF-16 without BOM
-    if looks_like_utf16(data) {
-        return false; // Likely UTF-16 text
-    }
-    
-    // Check for TAR format (special case with no magic number)
-    if looks_like_tar(data) {
-        return true;
-    }
-    
-    // Final fallback: check printable character ratio
-    // For 1KB of random data, we expect very few printable characters
-    calculate_printable_ratio(data) < 0.7
-}
-
-/// Check for known binary file signatures
-fn has_binary_signature(data: &[u8]) -> bool {
-    // Define binary file signatures with their minimum required lengths
-    let signatures: &[(&[u8], usize)] = &[
-        // Image formats
-        (&[0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A], 8), // PNG
-        (&[0xFF, 0xD8, 0xFF], 3), // JPEG (various subtypes)
-        (&[0x47, 0x49, 0x46, 0x38, 0x37, 0x61], 6), // GIF87a
-        (&[0x47, 0x49, 0x46, 0x38, 0x39, 0x61], 6), // GIF89a
-        (&[0x42, 0x4D], 2), // BMP
-        (&[0x00, 0x00, 0x01, 0x00], 4), // ICO
-        (&[0x49, 0x49, 0x2A, 0x00], 4), // TIFF (little endian)
-        (&[0x4D, 0x4D, 0x00, 0x2A], 4), // TIFF (big endian)
-        (&[0x52, 0x49, 0x46, 0x46], 4), // WebP (RIFF container)
-        (&[0x00, 0x00, 0x00, 0x0C, 0x6A, 0x50, 0x20, 0x20], 8), // JPEG 2000
-        
-        // Audio/Video formats
-        (&[0x49, 0x44, 0x33], 3), // MP3 with ID3v2
-        (&[0xFF, 0xFB], 2), // MP3
-        (&[0xFF, 0xF3], 2), // MP3
-        (&[0xFF, 0xF2], 2), // MP3
-        (&[0x4F, 0x67, 0x67, 0x53], 4), // OGG
-        (&[0x66, 0x74, 0x79, 0x70], 4), // MP4/M4A/MOV (at offset 4)
-        (&[0x52, 0x49, 0x46, 0x46], 4), // WAV/AVI (RIFF)
-        (&[0x46, 0x4C, 0x56], 3), // FLV
-        (&[0x1A, 0x45, 0xDF, 0xA3], 4), // MKV/WebM
-        
-        // Archive formats
-        (&[0x50, 0x4B, 0x03, 0x04], 4), // ZIP
-        (&[0x50, 0x4B, 0x05, 0x06], 4), // ZIP (empty)
-        (&[0x50, 0x4B, 0x07, 0x08], 4), // ZIP (spanned)
-        (&[0x52, 0x61, 0x72, 0x21, 0x1A, 0x07, 0x00], 7), // RAR v1.5+
-        (&[0x52, 0x61, 0x72, 0x21, 0x1A, 0x07, 0x01, 0x00], 8), // RAR v5.0+
-        (&[0x1F, 0x8B], 2), // GZIP
-        (&[0x42, 0x5A, 0x68], 3), // BZIP2
-        (&[0xFD, 0x37, 0x7A, 0x58, 0x5A, 0x00], 6), // XZ
-        (&[0x28, 0xB5, 0x2F, 0xFD], 4), // Zstandard
-        (&[0x04, 0x22, 0x4D, 0x18], 4), // LZ4
-        (&[0x1F, 0x9D], 2), // LZW compressed
-        (&[0x1F, 0xA0], 2), // LZH compressed
-        (&[0x37, 0x7A, 0xBC, 0xAF, 0x27, 0x1C], 6), // 7-Zip
-        
-        // Document formats
-        (&[0x25, 0x50, 0x44, 0x46], 4), // PDF
-        (&[0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1], 8), // MS Office (OLE)
-        (&[0x50, 0x4B, 0x03, 0x04], 4), // Office Open XML (also ZIP)
-        (&[0x7B, 0x5C, 0x72, 0x74, 0x66], 5), // RTF
-        
-        // Executables and object files
-        (&[0x7F, 0x45, 0x4C, 0x46], 4), // ELF
-        (&[0x4D, 0x5A], 2), // Windows PE/DOS
-        (&[0xCA, 0xFE, 0xBA, 0xBE], 4), // Mach-O (big endian)
-        (&[0xFE, 0xED, 0xFA, 0xCE], 4), // Mach-O 32-bit (little endian)
-        (&[0xFE, 0xED, 0xFA, 0xCF], 4), // Mach-O 64-bit (little endian)
-        (&[0xCE, 0xFA, 0xED, 0xFE], 4), // Mach-O 32-bit (big endian)
-        (&[0xCF, 0xFA, 0xED, 0xFE], 4), // Mach-O 64-bit (big endian)
-        (&[0xCA, 0xFE, 0xBA, 0xBE], 4), // Java class file
-        (&[0xDE, 0xC0, 0x17, 0x0B], 4), // Dalvik executable
-        
-        // Database formats
-        (&[0x53, 0x51, 0x4C, 0x69, 0x74, 0x65, 0x20, 0x66, 0x6F, 0x72, 0x6D, 0x61, 0x74, 0x20, 0x33, 0x00], 16), // SQLite
-        (&[0x00, 0x01, 0x00, 0x00], 4), // Palm Database
-        
-        // Font formats
-        (&[0x00, 0x01, 0x00, 0x00, 0x00], 5), // TrueType
-        (&[0x4F, 0x54, 0x54, 0x4F], 4), // OpenType
-        (&[0x77, 0x4F, 0x46, 0x46], 4), // WOFF
-        (&[0x77, 0x4F, 0x46, 0x32], 4), // WOFF2
-        
-        // Virtual machine formats
-        (&[0x76, 0x6D, 0x64, 0x6B], 4), // VMDK
-        (&[0x3C, 0x3C, 0x3C, 0x20, 0x4F, 0x72, 0x61, 0x63, 0x6C, 0x65, 0x20, 0x56, 0x4D, 0x20, 0x56, 0x69, 0x72, 0x74, 0x75, 0x61, 0x6C, 0x42, 0x6F, 0x78, 0x20, 0x44, 0x69, 0x73, 0x6B, 0x20, 0x49, 0x6D, 0x61, 0x67, 0x65, 0x20, 0x3E, 0x3E, 0x3E], 39), // VirtualBox VDI
-        
-        // Disk image formats
-        (&[0xEB, 0x3C, 0x90], 3), // FAT12/16/32
-        (&[0xEB, 0x58, 0x90], 3), // FAT32
-        (&[0x55, 0xAA], 2), // Boot sector (at offset 510)
-        
-        // Other binary formats
-        (&[0x21, 0x3C, 0x61, 0x72, 0x63, 0x68, 0x3E, 0x0A], 8), // AR archive
-        (&[0x78, 0x01], 2), // zlib (default compression)
-        (&[0x78, 0x9C], 2), // zlib (best compression)
-        (&[0x78, 0xDA], 2), // zlib (fast compression)
-        (&[0x62, 0x76, 0x78, 0x32], 4), // LZFSE
-    ];
-    
-    for (signature, min_len) in signatures {
-        if data.len() >= *min_len && data.starts_with(signature) {
-            return true;
-        }
-    }
-    
-    // Special case: check for ftyp box in MP4/MOV files (at offset 4)
-    if data.len() >= 8 && &data[4..8] == b"ftyp" {
-        return true;
-    }
-    
-    false
-}
-
-/// Check if data looks like UTF-16 without BOM
-fn looks_like_utf16(data: &[u8]) -> bool {
-    if data.len() < 4 || data.len() % 2 != 0 {
-        return false;
-    }
-    
-    let mut zero_count = 0;
-    let pairs = data.len() / 2;
-    
-    // Check if every other byte is zero (indicating UTF-16)
-    for i in 0..pairs {
-        if data[i * 2 + 1] == 0 {
-            zero_count += 1;
-        }
-    }
-    
-    // If more than 50% of odd positions are zero, might be UTF-16
-    zero_count as f64 / pairs as f64 > 0.5
-}
-
-/// Check if data looks like a TAR archive
-fn looks_like_tar(data: &[u8]) -> bool {
-    if data.len() < 512 {
-        return false;
-    }
-    
-    // TAR header structure validation
-    // Filename should not start with null
-    if data[0] == 0 {
-        return false;
-    }
-    
-    // Check file mode field (should be octal digits)
-    for i in 100..108 {
-        if data[i] != 0 && (data[i] < b'0' || data[i] > b'7') && data[i] != b' ' {
-            return false;
-        }
-    }
-    
-    // Check checksum field (should be octal digits or spaces)
-    for i in 148..156 {
-        if data[i] != 0 && (data[i] < b'0' || data[i] > b'7') && data[i] != b' ' {
-            return false;
-        }
-    }
-    
-    // Check magic field for POSIX TAR
-    if data.len() >= 265 {
-        let magic = &data[257..262];
-        if magic == b"ustar" {
-            return true;
-        }
-    }
-    
-    // Additional heuristic: check if the structure looks reasonable
-    let has_reasonable_structure = 
-        data[0] != 0 && // Filename starts
-        data[100..108].iter().all(|&b| b == 0 || (b >= b'0' && b <= b'7') || b == b' '); // Mode field
-    
-    has_reasonable_structure
-}
-
-/// Calculate the ratio of printable characters in the data
-fn calculate_printable_ratio(data: &[u8]) -> f64 {
-    let printable_count = data.iter().filter(|&&b| {
-        b.is_ascii_graphic() || b.is_ascii_whitespace()
-    }).count();
-    
-    printable_count as f64 / data.len() as f64
-}

src/common/is_binary.rs

@@ -0,0 +1,253 @@
+/// Detect if data is binary or text
+/// Returns true if data is likely binary, false if likely text
+pub fn is_binary(data: &[u8]) -> bool {
+    if data.is_empty() {
+        return false;
+    }
+
+    // First check for known binary file signatures
+    if has_binary_signature(data) {
+        return true;
+    }
+
+    // Check for UTF-16 BOM (text)
+    if data.len() >= 2
+        && ((data[0] == 0xFF && data[1] == 0xFE) || (data[0] == 0xFE && data[1] == 0xFF))
+    {
+        return false; // UTF-16 with BOM is text
+    }
+
+    // Check for UTF-8 BOM (text)
+    if data.len() >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF {
+        return false; // UTF-8 with BOM is text
+    }
+
+    // Check if it's valid UTF-8
+    if std::str::from_utf8(data).is_ok() {
+        // Valid UTF-8, check printable character ratio
+        return calculate_printable_ratio(data) < 0.7;
+    }
+
+    // Not valid UTF-8, check if it might be UTF-16 without BOM
+    if looks_like_utf16(data) {
+        return false; // Likely UTF-16 text
+    }
+
+    // Check for TAR format (special case with no magic number)
+    if looks_like_tar(data) {
+        return true;
+    }
+
+    // Final fallback: check printable character ratio
+    // For 1KB of random data, we expect very few printable characters
+    calculate_printable_ratio(data) < 0.7
+}
+
+/// Check for known binary file signatures
+fn has_binary_signature(data: &[u8]) -> bool {
+    // Define binary file signatures with their minimum required lengths
+    let signatures: &[(&[u8], usize)] = &[
+        // Image formats
+        (&[0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A], 8), // PNG
+        (&[0xFF, 0xD8, 0xFF], 3),                               // JPEG (various subtypes)
+        (&[0x47, 0x49, 0x46, 0x38, 0x37, 0x61], 6),             // GIF87a
+        (&[0x47, 0x49, 0x46, 0x38, 0x39, 0x61], 6),             // GIF89a
+        (&[0x42, 0x4D], 2),                                     // BMP
+        (&[0x00, 0x00, 0x01, 0x00], 4),                         // ICO
+        (&[0x49, 0x49, 0x2A, 0x00], 4),                         // TIFF (little endian)
+        (&[0x4D, 0x4D, 0x00, 0x2A], 4),                         // TIFF (big endian)
+        (&[0x52, 0x49, 0x46, 0x46], 4),                         // WebP (RIFF container)
+        (&[0x00, 0x00, 0x00, 0x0C, 0x6A, 0x50, 0x20, 0x20], 8), // JPEG 2000
+        // Audio/Video formats
+        (&[0x49, 0x44, 0x33], 3),       // MP3 with ID3v2
+        (&[0xFF, 0xFB], 2),             // MP3
+        (&[0xFF, 0xF3], 2),             // MP3
+        (&[0xFF, 0xF2], 2),             // MP3
+        (&[0x4F, 0x67, 0x67, 0x53], 4), // OGG
+        (&[0x66, 0x74, 0x79, 0x70], 4), // MP4/M4A/MOV (at offset 4)
+        (&[0x52, 0x49, 0x46, 0x46], 4), // WAV/AVI (RIFF)
+        (&[0x46, 0x4C, 0x56], 3),       // FLV
+        (&[0x1A, 0x45, 0xDF, 0xA3], 4), // MKV/WebM
+        // Archive formats
+        (&[0x50, 0x4B, 0x03, 0x04], 4),                         // ZIP
+        (&[0x50, 0x4B, 0x05, 0x06], 4),                         // ZIP (empty)
+        (&[0x50, 0x4B, 0x07, 0x08], 4),                         // ZIP (spanned)
+        (&[0x52, 0x61, 0x72, 0x21, 0x1A, 0x07, 0x00], 7),       // RAR v1.5+
+        (&[0x52, 0x61, 0x72, 0x21, 0x1A, 0x07, 0x01, 0x00], 8), // RAR v5.0+
+        (&[0x1F, 0x8B], 2),                                     // GZIP
+        (&[0x42, 0x5A, 0x68], 3),                               // BZIP2
+        (&[0xFD, 0x37, 0x7A, 0x58, 0x5A, 0x00], 6),             // XZ
+        (&[0x28, 0xB5, 0x2F, 0xFD], 4),                         // Zstandard
+        (&[0x04, 0x22, 0x4D, 0x18], 4),                         // LZ4
+        (&[0x1F, 0x9D], 2),                                     // LZW compressed
+        (&[0x1F, 0xA0], 2),                                     // LZH compressed
+        (&[0x37, 0x7A, 0xBC, 0xAF, 0x27, 0x1C], 6),             // 7-Zip
+        // Document formats
+        (&[0x25, 0x50, 0x44, 0x46], 4),                         // PDF
+        (&[0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1], 8), // MS Office (OLE)
+        (&[0x50, 0x4B, 0x03, 0x04], 4),                         // Office Open XML (also ZIP)
+        (&[0x7B, 0x5C, 0x72, 0x74, 0x66], 5),                   // RTF
+        // Executables and object files
+        (&[0x7F, 0x45, 0x4C, 0x46], 4), // ELF
+        (&[0x4D, 0x5A], 2),             // Windows PE/DOS
+        (&[0xCA, 0xFE, 0xBA, 0xBE], 4), // Mach-O (big endian)
+        (&[0xFE, 0xED, 0xFA, 0xCE], 4), // Mach-O 32-bit (little endian)
+        (&[0xFE, 0xED, 0xFA, 0xCF], 4), // Mach-O 64-bit (little endian)
+        (&[0xCE, 0xFA, 0xED, 0xFE], 4), // Mach-O 32-bit (big endian)
+        (&[0xCF, 0xFA, 0xED, 0xFE], 4), // Mach-O 64-bit (big endian)
+        (&[0xCA, 0xFE, 0xBA, 0xBE], 4), // Java class file
+        (&[0xDE, 0xC0, 0x17, 0x0B], 4), // Dalvik executable
+        // Database formats
+        (
+            &[
+                0x53, 0x51, 0x4C, 0x69, 0x74, 0x65, 0x20, 0x66, 0x6F, 0x72, 0x6D, 0x61, 0x74, 0x20,
+                0x33, 0x00,
+            ],
+            16,
+        ), // SQLite
+        (&[0x00, 0x01, 0x00, 0x00], 4), // Palm Database
+        // Font formats
+        (&[0x00, 0x01, 0x00, 0x00, 0x00], 5), // TrueType
+        (&[0x4F, 0x54, 0x54, 0x4F], 4),       // OpenType
+        (&[0x77, 0x4F, 0x46, 0x46], 4),       // WOFF
+        (&[0x77, 0x4F, 0x46, 0x32], 4),       // WOFF2
+        // Virtual machine formats
+        (&[0x76, 0x6D, 0x64, 0x6B], 4), // VMDK
+        (
+            &[
+                0x3C, 0x3C, 0x3C, 0x20, 0x4F, 0x72, 0x61, 0x63, 0x6C, 0x65, 0x20, 0x56, 0x4D, 0x20,
+                0x56, 0x69, 0x72, 0x74, 0x75, 0x61, 0x6C, 0x42, 0x6F, 0x78, 0x20, 0x44, 0x69, 0x73,
+                0x6B, 0x20, 0x49, 0x6D, 0x61, 0x67, 0x65, 0x20, 0x3E, 0x3E, 0x3E,
+            ],
+            39,
+        ), // VirtualBox VDI
+        // Disk image formats
+        (&[0xEB, 0x3C, 0x90], 3), // FAT12/16/32
+        (&[0xEB, 0x58, 0x90], 3), // FAT32
+        (&[0x55, 0xAA], 2),       // Boot sector (at offset 510)
+        // Other binary formats
+        (&[0x21, 0x3C, 0x61, 0x72, 0x63, 0x68, 0x3E, 0x0A], 8), // AR archive
+        (&[0x78, 0x01], 2),                                     // zlib (default compression)
+        (&[0x78, 0x9C], 2),                                     // zlib (best compression)
+        (&[0x78, 0xDA], 2),                                     // zlib (fast compression)
+        (&[0x62, 0x76, 0x78, 0x32], 4),                         // LZFSE
+    ];
+
+    for (signature, min_len) in signatures {
+        if data.len() >= *min_len && data.starts_with(signature) {
+            return true;
+        }
+    }
+
+    // Special case: check for ftyp box in MP4/MOV files (at offset 4)
+    if data.len() >= 8 && &data[4..8] == b"ftyp" {
+        return true;
+    }
+
+    false
+}
+
+/// Check if data looks like UTF-16 without BOM
+fn looks_like_utf16(data: &[u8]) -> bool {
+    if data.len() < 4 || data.len() % 2 != 0 {
+        return false;
+    }
+
+    // Check if it could be UTF-16 by looking at null patterns
+    let mut null_pairs = 0;
+    let max_checks = std::cmp::min(data.len() / 2, 50); // Check up to 50 character pairs
+
+    for i in 0..max_checks {
+        if data[i * 2 + 1] == 0 {
+            null_pairs += 1;
+        }
+    }
+
+    // If most high bytes are zero, it's likely UTF-16
+    if max_checks > 0 && null_pairs as f64 / max_checks as f64 > 0.7 {
+        return true;
+    }
+
+    // Also check the reverse pattern (little-endian UTF-16)
+    let mut null_pairs_reverse = 0;
+    for i in 0..max_checks {
+        if i * 2 + 1 < data.len() && data[i * 2] == 0 {
+            null_pairs_reverse += 1;
+        }
+    }
+
+    null_pairs_reverse as f64 / max_checks as f64 > 0.7
+}
+
+/// Check if data looks like a TAR archive
+fn looks_like_tar(data: &[u8]) -> bool {
+    if data.len() < 512 {
+        return false;
+    }
+
+    // TAR header structure validation
+    // Filename should not start with null
+    if data[0] == 0 {
+        return false;
+    }
+
+    // Check file mode field (should be octal digits)
+    for byte in data.iter().skip(100).take(8) {
+        if *byte != 0 && !(b'0'..=b'7').contains(byte) && *byte != b' ' {
+            return false;
+        }
+    }
+
+    // Check checksum field (should be octal digits or spaces)
+    for &b in &data[148..156] {
+        if b != 0 && !(b'0'..=b'7').contains(&b) && b != b' ' {
+            return false;
+        }
+    }
+
+    // Check magic field for POSIX TAR
+    if data.len() >= 265 {
+        let magic = &data[257..262];
+        if magic == b"ustar" {
+            return true;
+        }
+    }
+
+    // Additional heuristic: check if the structure looks reasonable
+    // Mode field
+
+    data[0] != 0 && // Filename starts
+        data[100..108].iter().all(|&b| b == 0 || (b'0'..=b'7').contains(&b) || b == b' ')
+}
+
+/// Calculate the ratio of printable characters in the data
+fn calculate_printable_ratio(data: &[u8]) -> f64 {
+    let printable_count = data
+        .iter()
+        .filter(|&&b| b.is_ascii_graphic() || b.is_ascii_whitespace())
+        .count();
+
+    printable_count as f64 / data.len() as f64
+}
+
+/// Check if content is binary, using metadata as a fast path.
+///
+/// First checks for a "text" metadata field:
+/// - "false" means binary
+/// - "true" means text
+/// - Absent or other values fall back to byte sampling
+///
+/// # Arguments
+///
+/// * `metadata` - Key-value metadata map (e.g., from `meta_as_map()`)
+/// * `data` - Byte sample to analyze if metadata is inconclusive
+pub fn is_content_binary_from_metadata(
+    metadata: &std::collections::HashMap<String, String>,
+    data: &[u8],
+) -> bool {
+    if let Some(text_val) = metadata.get("text") {
+        text_val == "false"
+    } else {
+        is_binary(data)
+    }
+}

src/common/mod.rs

@@ -0,0 +1,91 @@
+pub mod is_binary;
+
+/// Detects if data is binary or text based on signatures and printable ratios.
+pub mod status;
+
+/// Plugin schema types and discovery functions.
+pub mod schema;
+
+/// Standard buffer size for I/O operations (8KB)
+pub const PIPESIZE: usize = 8192;
+
+/// Reads chunks from `reader` until EOF, passing each chunk to `f`.
+///
+/// Uses a fixed PIPESIZE buffer to ensure bounded memory usage.
+pub fn stream_copy<R: std::io::Read + ?Sized>(
+    reader: &mut R,
+    mut f: impl FnMut(&[u8]) -> std::io::Result<()>,
+) -> std::io::Result<()> {
+    let mut buffer = [0u8; PIPESIZE];
+    loop {
+        let n = reader.read(&mut buffer)?;
+        if n == 0 {
+            break;
+        }
+        f(&buffer[..n])?;
+    }
+    Ok(())
+}
+
+/// Reads content from a reader with offset and length bounds.
+///
+/// Skips `offset` bytes from the reader, then reads up to `length` bytes
+/// (or all remaining if `length` is 0). Uses PIPESIZE buffers throughout.
+///
+/// # Arguments
+///
+/// * `reader` - The source reader positioned at the start.
+/// * `offset` - Number of bytes to skip before reading.
+/// * `length` - Maximum bytes to read (0 = read all remaining).
+/// * `content_len` - Total content size (used to cap skip/read amounts).
+///
+/// # Returns
+///
+/// A `Vec<u8>` containing the requested byte range.
+pub fn read_with_bounds<R: std::io::Read>(
+    reader: &mut R,
+    offset: u64,
+    length: u64,
+    content_len: u64,
+) -> std::io::Result<Vec<u8>> {
+    // Skip offset bytes
+    let skip = std::cmp::min(offset, content_len);
+    let mut remaining = skip;
+    let mut buf = [0u8; PIPESIZE];
+    while remaining > 0 {
+        let to_read = std::cmp::min(remaining, buf.len() as u64) as usize;
+        match reader.read(&mut buf[..to_read]) {
+            Ok(0) => break,
+            Ok(n) => remaining -= n as u64,
+            Err(e) => return Err(e),
+        }
+    }
+
+    // Read bounded content
+    let max_bytes = if length > 0 {
+        std::cmp::min(length, content_len.saturating_sub(offset))
+    } else {
+        content_len.saturating_sub(offset)
+    };
+    let mut result = Vec::with_capacity(std::cmp::min(max_bytes, 64 * 1024) as usize);
+    let mut bytes_read = 0u64;
+    while bytes_read < max_bytes {
+        let to_read = std::cmp::min(max_bytes - bytes_read, buf.len() as u64) as usize;
+        match reader.read(&mut buf[..to_read]) {
+            Ok(0) => break,
+            Ok(n) => {
+                result.extend_from_slice(&buf[..n]);
+                bytes_read += n as u64;
+            }
+            Err(e) => return Err(e),
+        }
+    }
+    Ok(result)
+}
+
+/// Sanitize a timestamp string for use in filenames.
+///
+/// Replaces colons with hyphens (e.g., `2026-03-17T12:00:00Z` → `2026-03-17T12-00-00Z`).
+pub fn sanitize_ts_string(ts: &str) -> String {
+    ts.replace(':', "-")
+}

src/common/schema.rs

@@ -0,0 +1,166 @@
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use strum::IntoEnumIterator;
+
+/// Value type for a plugin option.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum OptionType {
+    String,
+    Integer,
+    Boolean,
+    Any,
+}
+
+impl OptionType {
+    /// Infer the option type from a YAML value.
+    pub fn from_yaml_value(value: &serde_yaml::Value) -> Self {
+        match value {
+            serde_yaml::Value::Bool(_) => OptionType::Boolean,
+            serde_yaml::Value::Number(_) => OptionType::Integer,
+            serde_yaml::Value::String(_) => OptionType::String,
+            _ => OptionType::Any,
+        }
+    }
+}
+
+/// Schema for a single plugin option.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OptionSchema {
+    pub name: String,
+    pub option_type: OptionType,
+    pub default: Option<serde_yaml::Value>,
+    pub required: bool,
+}
+
+/// Schema for a single plugin output.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OutputSchema {
+    pub name: String,
+    pub description: String,
+}
+
+/// Schema describing a plugin's configuration requirements.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct PluginSchema {
+    pub name: String,
+    pub description: String,
+    pub options: Vec<OptionSchema>,
+    pub outputs: Vec<OutputSchema>,
+}
+
+/// Gathers schemas from all registered meta plugins.
+///
+/// Iterates all `MetaPluginType` variants, attempts to create a default instance,
+/// and collects their schemas. Plugins that fail to register (e.g., feature-gated)
+/// are silently skipped.
+pub fn gather_meta_plugin_schemas() -> Vec<PluginSchema> {
+    use crate::meta_plugin::{MetaPluginType, get_meta_plugin};
+
+    let mut schemas = Vec::new();
+    let mut sorted_types: Vec<MetaPluginType> = MetaPluginType::iter().collect();
+    sorted_types.sort_by_key(|t| t.to_string());
+
+    for plugin_type in sorted_types {
+        let plugin = match get_meta_plugin(plugin_type.clone(), None, None) {
+            Ok(p) => p,
+            Err(_) => continue,
+        };
+
+        let name = plugin.meta_type().to_string();
+
+        let options: Vec<OptionSchema> = plugin
+            .options()
+            .iter()
+            .map(|(key, value)| {
+                let option_type = OptionType::from_yaml_value(value);
+                let (default, required) = if value.is_null() {
+                    (None, true)
+                } else {
+                    (Some(value.clone()), false)
+                };
+                OptionSchema {
+                    name: key.clone(),
+                    option_type,
+                    default,
+                    required,
+                }
+            })
+            .collect();
+
+        let mut outputs: Vec<OutputSchema> = Vec::new();
+        for (key, value) in plugin.outputs() {
+            if !value.is_null() {
+                outputs.push(OutputSchema {
+                    name: key.clone(),
+                    description: key.clone(),
+                });
+            }
+        }
+        // Also include default outputs if outputs map is empty
+        if outputs.is_empty() {
+            for output_name in plugin.default_outputs() {
+                outputs.push(OutputSchema {
+                    name: output_name.clone(),
+                    description: output_name,
+                });
+            }
+        }
+
+        schemas.push(PluginSchema {
+            name,
+            description: plugin.description().to_string(),
+            options,
+            outputs,
+        });
+    }
+
+    schemas
+}
+
+/// Gathers schemas from all registered filter plugins.
+///
+/// Uses the global filter plugin registry to discover all registered filters,
+/// creates a default instance of each, and collects their option schemas.
+pub fn gather_filter_plugin_schemas() -> Vec<PluginSchema> {
+    use crate::services::filter_service::get_available_filter_plugins;
+
+    let plugins = get_available_filter_plugins().unwrap_or_default();
+    let mut schemas: Vec<PluginSchema> = plugins
+        .into_iter()
+        .map(|(name, creator)| {
+            let plugin = creator();
+            let options: Vec<OptionSchema> = plugin
+                .options()
+                .iter()
+                .map(|opt| {
+                    let option_type = match &opt.default {
+                        Some(serde_json::Value::Bool(_)) => OptionType::Boolean,
+                        Some(serde_json::Value::Number(_)) => OptionType::Integer,
+                        Some(serde_json::Value::String(_)) => OptionType::String,
+                        _ => OptionType::Any,
+                    };
+                    OptionSchema {
+                        name: opt.name.clone(),
+                        option_type,
+                        default: opt.default.as_ref().map(|v| {
+                            // Convert serde_json::Value to serde_yaml::Value
+                            serde_yaml::to_value(v).unwrap_or(serde_yaml::Value::Null)
+                        }),
+                        required: opt.required,
+                    }
+                })
+                .collect();
+
+            PluginSchema {
+                name: name.clone(),
+                description: plugin.description().to_string(),
+                options,
+                outputs: Vec::new(),
+            }
+        })
+        .collect();
+
+    schemas.sort_by(|a, b| a.name.cmp(&b.name));
+    schemas
+}

src/common/status.rs

@@ -0,0 +1,224 @@
+use std::path::PathBuf;
+use strum::IntoEnumIterator;
+#[cfg(feature = "server")]
+use utoipa::ToSchema;
+
+use crate::compression_engine::{CompressionType, get_compression_engine};
+use crate::meta_plugin::MetaPluginType;
+
+use crate::filter_plugin::FilterOption;
+
+#[derive(serde::Serialize, serde::Deserialize, Clone, Debug)]
+#[cfg_attr(feature = "server", derive(ToSchema))]
+pub struct FilterPluginInfo {
+    pub name: String,
+    pub options: Vec<FilterOption>,
+    pub description: String,
+}
+
+#[derive(serde::Serialize, serde::Deserialize)]
+#[cfg_attr(feature = "server", derive(ToSchema))]
+pub struct StatusInfo {
+    pub paths: PathInfo,
+    pub compression: Vec<CompressionInfo>,
+    pub meta_plugins: std::collections::HashMap<String, MetaPluginInfo>,
+    pub enabled_meta_plugins: Vec<String>,
+    pub filter_plugins: Vec<FilterPluginInfo>,
+    pub configured_meta_plugins: Option<Vec<crate::config::MetaPluginConfig>>,
+}
+
+impl Default for StatusInfo {
+    fn default() -> Self {
+        Self {
+            paths: PathInfo {
+                data: String::new(),
+                database: String::new(),
+            },
+            compression: Vec::new(),
+            meta_plugins: std::collections::HashMap::new(),
+            enabled_meta_plugins: Vec::new(),
+            filter_plugins: Vec::new(),
+            configured_meta_plugins: None,
+        }
+    }
+}
+
+#[derive(serde::Serialize, serde::Deserialize)]
+#[cfg_attr(feature = "server", derive(ToSchema))]
+pub struct PathInfo {
+    pub data: String,
+    pub database: String,
+}
+
+#[derive(serde::Serialize, serde::Deserialize, Debug)]
+#[cfg_attr(feature = "server", derive(ToSchema))]
+pub struct CompressionInfo {
+    #[serde(rename = "type")]
+    pub compression_type: String,
+    pub found: bool,
+    pub default: bool,
+    pub binary: String,
+    pub compress: String,
+    pub decompress: String,
+}
+
+#[derive(serde::Serialize, serde::Deserialize, Clone, Debug)]
+#[cfg_attr(feature = "server", derive(ToSchema))]
+pub struct MetaPluginInfo {
+    pub meta_name: String,
+    pub outputs: std::collections::HashMap<String, serde_yaml::Value>,
+    pub options: std::collections::HashMap<String, serde_yaml::Value>,
+}
+
+pub fn generate_status_info(
+    data_path: PathBuf,
+    db_path: PathBuf,
+    enabled_meta_plugins: &[MetaPluginType],
+    enabled_compression_type: Option<CompressionType>,
+) -> anyhow::Result<StatusInfo> {
+    log::debug!("STATUS: Starting status info generation");
+    let path_info = PathInfo {
+        data: data_path
+            .into_os_string()
+            .into_string()
+            .map_err(|_| anyhow::anyhow!("Unable to convert data path to string"))?,
+        database: db_path
+            .into_os_string()
+            .into_string()
+            .map_err(|_| anyhow::anyhow!("Unable to convert DB path to string"))?,
+    };
+
+    let _default_type = crate::compression_engine::default_compression_type();
+    let mut compression_info = Vec::with_capacity(CompressionType::iter().count());
+
+    // Sort compression types by their string representation
+    let mut sorted_compression_types: Vec<CompressionType> = CompressionType::iter().collect();
+    sorted_compression_types.sort_by_key(|ct| ct.to_string());
+
+    for compression_type in sorted_compression_types {
+        let (binary, compress, decompress, supported) =
+            match get_compression_engine(compression_type.clone()) {
+                Ok(engine) => {
+                    let supp = engine.is_supported();
+                    if supp && engine.is_internal() {
+                        (
+                            "<INTERNAL>".to_string(),
+                            "".to_string(),
+                            "".to_string(),
+                            supp,
+                        )
+                    } else if supp {
+                        let (b, c, d) = engine.get_status_info();
+                        (b, c, d, supp)
+                    } else {
+                        (
+                            "<UNSUPPORTED>".to_string(),
+                            "".to_string(),
+                            "".to_string(),
+                            supp,
+                        )
+                    }
+                }
+                Err(_) => (
+                    "<UNSUPPORTED>".to_string(),
+                    "".to_string(),
+                    "".to_string(),
+                    false,
+                ),
+            };
+
+        let is_enabled = enabled_compression_type
+            .as_ref()
+            .is_some_and(|ct| *ct == compression_type);
+
+        compression_info.push(CompressionInfo {
+            compression_type: compression_type.to_string(),
+            found: supported,
+            default: is_enabled,
+            binary,
+            compress,
+            decompress,
+        });
+    }
+
+    let mut meta_plugins_map =
+        std::collections::HashMap::with_capacity(MetaPluginType::iter().count());
+    let mut enabled_meta_plugins_vec = Vec::new();
+
+    // Sort meta plugin types by their string representation to avoid creating plugins just for sorting
+    let mut sorted_meta_plugins: Vec<MetaPluginType> = MetaPluginType::iter().collect();
+    sorted_meta_plugins.sort_by_key(|meta_plugin_type| meta_plugin_type.to_string());
+
+    for meta_plugin_type in sorted_meta_plugins {
+        log::debug!("STATUS: Processing meta plugin type: {meta_plugin_type:?}");
+        let meta_plugin =
+            match crate::meta_plugin::get_meta_plugin(meta_plugin_type.clone(), None, None) {
+                Ok(p) => p,
+                Err(e) => {
+                    log::warn!(
+                        "STATUS: Skipping unregistered meta plugin {meta_plugin_type:?}: {e}"
+                    );
+                    continue;
+                }
+            };
+
+        // Get meta name first to avoid borrowing issues
+        log::debug!("STATUS: Getting meta name...");
+        let meta_name = meta_plugin.meta_type().to_string();
+        log::debug!("STATUS: Got meta name: {meta_name}");
+
+        // Check if this plugin is enabled
+        let is_enabled = enabled_meta_plugins.contains(&meta_plugin_type);
+        if is_enabled {
+            enabled_meta_plugins_vec.push(meta_name.clone());
+        }
+
+        // Create a display of outputs for status - use configured outputs if available, otherwise defaults
+        let outputs_display = if meta_plugin.outputs().is_empty() {
+            // No configured outputs, use defaults
+            let mut default_outputs = std::collections::HashMap::new();
+            for output_name in meta_plugin.default_outputs() {
+                default_outputs.insert(output_name.clone(), serde_yaml::Value::String(output_name));
+            }
+            default_outputs
+        } else {
+            // Use configured outputs
+            meta_plugin.outputs().clone()
+        };
+
+        // Get options
+        let options = meta_plugin.options().clone();
+
+        meta_plugins_map.insert(
+            meta_name.clone(),
+            MetaPluginInfo {
+                meta_name,
+                outputs: outputs_display,
+                options,
+            },
+        );
+    }
+
+    // Populate filter plugin info from the global registry
+    let filter_plugins_map = crate::services::filter_service::get_available_filter_plugins()?;
+    let filter_plugins_info: Vec<FilterPluginInfo> = filter_plugins_map
+        .into_iter()
+        .map(|(name, creator)| {
+            let plugin = creator();
+            FilterPluginInfo {
+                name: name.clone(),
+                options: plugin.options(),
+                description: format!("{name} filter plugin"),
+            }
+        })
+        .collect();
+
+    Ok(StatusInfo {
+        paths: path_info,
+        compression: compression_info,
+        meta_plugins: meta_plugins_map,
+        enabled_meta_plugins: enabled_meta_plugins_vec,
+        filter_plugins: filter_plugins_info,
+        configured_meta_plugins: None,
+    })
+}

src/compression_engine.rs

@@ -1,121 +0,0 @@
-use anyhow::Result;
-use std::io;
-use std::io::{Read, Write};
-use std::path::PathBuf;
-use strum::IntoEnumIterator;
-
-use log::*;
-
-use lazy_static::lazy_static;
-
-extern crate enum_map;
-use enum_map::enum_map;
-use enum_map::{Enum, EnumMap};
-
-pub mod gzip;
-pub mod lz4;
-pub mod none;
-pub mod program;
-
-use crate::compression_engine::gzip::CompressionEngineGZip;
-use crate::compression_engine::lz4::CompressionEngineLZ4;
-use crate::compression_engine::none::CompressionEngineNone;
-use crate::compression_engine::program::CompressionEngineProgram;
-
-#[derive(Debug, Eq, PartialEq, Clone, strum::EnumIter, strum::Display, strum::EnumString, Enum)]
-#[strum(ascii_case_insensitive)]
-pub enum CompressionType {
-    LZ4,
-    GZip,
-    BZip2,
-    XZ,
-    ZStd,
-    None,
-}
-
-pub trait CompressionEngine {
-    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read>>;
-    fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>>;
-
-    fn is_supported(&self) -> bool {
-        true
-    }
-
-    fn copy(&self, file_path: PathBuf, writer: &mut dyn Write) -> Result<()> {
-        let mut reader = self.open(file_path)?;
-        io::copy(&mut reader, writer)?;
-        writer.flush()?;
-        Ok(())
-    }
-
-    fn cat(&self, file_path: PathBuf) -> Result<()> {
-        let mut stdout = io::stdout().lock();
-        self.copy(file_path, &mut stdout)
-    }
-
-    fn size(&self, file_path: PathBuf) -> Result<usize> {
-        let mut reader = self.open(file_path)?;
-        let mut buffer = [0; libc::BUFSIZ as usize];
-        let mut size: usize = 0;
-
-        loop {
-            let n = reader.read(&mut buffer[..libc::BUFSIZ as usize])?;
-            if n == 0 {
-                debug!("COMPRESSION: EOF");
-                break;
-            }
-
-            size += n;
-        }
-
-        Ok(size)
-    }
-}
-
-lazy_static! {
-    pub static ref COMPRESSION_PROGRAMS: EnumMap<CompressionType, Option<CompressionEngineProgram>> = enum_map! {
-        CompressionType::LZ4 => None,
-        CompressionType::GZip => None,
-        CompressionType::BZip2 => {
-            let program = CompressionEngineProgram::new("bzip2", vec!["-qcf"], vec!["-dcf"]);
-            if program.supported { Some(program) } else { None }
-        },
-        CompressionType::XZ => {
-            let program = CompressionEngineProgram::new("xz", vec!["-qcf"], vec!["-dcf"]);
-            if program.supported { Some(program) } else { None }
-        },
-        CompressionType::ZStd => {
-            let program = CompressionEngineProgram::new("zstd", vec!["-qcf"], vec!["-dcf"]);
-            if program.supported { Some(program) } else { None }
-        },
-        CompressionType::None => None
-    };
-}
-
-pub fn get_compression_engine(
-    compression_type: CompressionType,
-) -> Result<Box<dyn CompressionEngine>> {
-    match compression_type {
-        CompressionType::LZ4 => Ok(Box::new(CompressionEngineLZ4::new())),
-        CompressionType::GZip => Ok(Box::new(CompressionEngineGZip::new())),
-        CompressionType::None => Ok(Box::new(CompressionEngineNone::new())),
-        compression_type => Ok(Box::new(
-            COMPRESSION_PROGRAMS[compression_type.clone()]
-                .clone()
-                .unwrap(),
-        )),
-    }
-}
-
-pub fn default_compression_type() -> CompressionType {
-    let mut default = CompressionType::None;
-    for compression_type in CompressionType::iter() {
-        let compression_engine =
-            get_compression_engine(compression_type.clone()).expect("Missing engine");
-        if compression_engine.is_supported() {
-            default = compression_type;
-            break;
-        }
-    }
-    default
-}

src/compression_engine/gzip.rs

@@ -1,31 +1,48 @@
+#[cfg(feature = "gzip")]
 use anyhow::Result;
+#[cfg(feature = "gzip")]
 use log::*;
+#[cfg(feature = "gzip")]
 use std::fs::File;
+#[cfg(feature = "gzip")]
 use std::io;
+#[cfg(feature = "gzip")]
 use std::io::{Read, Write};
+#[cfg(feature = "gzip")]
 use std::path::PathBuf;
 
+#[cfg(feature = "gzip")]
 use flate2::Compression;
+#[cfg(feature = "gzip")]
 use flate2::read::GzDecoder;
+#[cfg(feature = "gzip")]
 use flate2::write::GzEncoder;
 
+#[cfg(feature = "gzip")]
 use crate::compression_engine::CompressionEngine;
 
+#[cfg(feature = "gzip")]
 #[derive(Debug, Eq, PartialEq, Clone, Default)]
 pub struct CompressionEngineGZip {}
 
+#[cfg(feature = "gzip")]
 impl CompressionEngineGZip {
     pub fn new() -> CompressionEngineGZip {
         CompressionEngineGZip {}
     }
 }
 
+#[cfg(feature = "gzip")]
 impl CompressionEngine for CompressionEngineGZip {
     fn is_supported(&self) -> bool {
         true
     }
 
-    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read>> {
+    fn get_status_info(&self) -> (String, String, String) {
+        ("<INTERNAL>".to_string(), "".to_string(), "".to_string())
+    }
+
+    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read + Send>> {
         debug!("COMPRESSION: Opening {:?} using {:?}", file_path, *self);
 
         let file = File::open(file_path)?;
@@ -33,19 +50,26 @@ impl CompressionEngine for CompressionEngineGZip {
     }
 
     fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>> {
-        debug!("COMPRESSION: Writting to {:?} using {:?}", file_path, *self);
+        debug!("COMPRESSION: Writing to {:?} using {:?}", file_path, *self);
 
         let file = File::create(file_path)?;
         let gzip_write = GzEncoder::new(file, Compression::default());
 
         Ok(Box::new(AutoFinishGzEncoder::new(gzip_write)))
     }
+
+    fn clone_box(&self) -> Box<dyn CompressionEngine> {
+        Box::new(self.clone())
+    }
 }
 
+#[cfg(feature = "gzip")]
+#[derive(Debug)]
 pub struct AutoFinishGzEncoder<W: Write> {
     encoder: Option<GzEncoder<W>>,
 }
 
+#[cfg(feature = "gzip")]
 impl<W: Write> AutoFinishGzEncoder<W> {
     fn new(gz_encoder: GzEncoder<W>) -> AutoFinishGzEncoder<W> {
         AutoFinishGzEncoder {
@@ -54,21 +78,37 @@ impl<W: Write> AutoFinishGzEncoder<W> {
     }
 }
 
+#[cfg(feature = "gzip")]
 impl<W: Write> Drop for AutoFinishGzEncoder<W> {
     fn drop(&mut self) {
         if let Some(encoder) = self.encoder.take() {
             debug!("COMPRESSION: Finishing");
-            let _ = encoder.finish();
+            if let Err(e) = encoder.finish() {
+                warn!("Failed to finish GZip encoder: {e}");
+            }
         }
     }
 }
 
+#[cfg(feature = "gzip")]
 impl<W: Write> Write for AutoFinishGzEncoder<W> {
     fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
-        self.encoder.as_mut().unwrap().write(buf)
+        match self.encoder.as_mut() {
+            Some(encoder) => encoder.write(buf),
+            None => Err(io::Error::new(
+                io::ErrorKind::BrokenPipe,
+                "encoder already finished",
+            )),
+        }
     }
 
     fn flush(&mut self) -> io::Result<()> {
-        self.encoder.as_mut().unwrap().flush()
+        match self.encoder.as_mut() {
+            Some(encoder) => encoder.flush(),
+            None => Err(io::Error::new(
+                io::ErrorKind::BrokenPipe,
+                "encoder already finished",
+            )),
+        }
     }
 }

src/compression_engine/lz4.rs

@@ -1,25 +1,36 @@
+#[cfg(feature = "lz4")]
 use anyhow::Result;
+#[cfg(feature = "lz4")]
 use log::*;
+#[cfg(feature = "lz4")]
 use std::io::Write;
 
+#[cfg(feature = "lz4")]
 use lz4_flex::frame::{FrameDecoder, FrameEncoder};
+#[cfg(feature = "lz4")]
 use std::fs::File;
+#[cfg(feature = "lz4")]
 use std::io::Read;
+#[cfg(feature = "lz4")]
 use std::path::PathBuf;
 
+#[cfg(feature = "lz4")]
 use crate::compression_engine::CompressionEngine;
 
+#[cfg(feature = "lz4")]
 #[derive(Debug, Eq, PartialEq, Clone, Default)]
 pub struct CompressionEngineLZ4 {}
 
+#[cfg(feature = "lz4")]
 impl CompressionEngineLZ4 {
     pub fn new() -> CompressionEngineLZ4 {
         CompressionEngineLZ4 {}
     }
 }
 
+#[cfg(feature = "lz4")]
 impl CompressionEngine for CompressionEngineLZ4 {
-    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read>> {
+    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read + Send>> {
         debug!("COMPRESSION: Opening {:?} using {:?}", file_path, *self);
 
         let file = File::open(file_path)?;
@@ -27,11 +38,15 @@ impl CompressionEngine for CompressionEngineLZ4 {
     }
 
     fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>> {
-        debug!("COMPRESSION: Writting to {:?} using {:?}", file_path, *self);
+        debug!("COMPRESSION: Writing to {:?} using {:?}", file_path, *self);
 
         let file = File::create(file_path)?;
         let lz4_write = FrameEncoder::new(file).auto_finish();
 
         Ok(Box::new(lz4_write))
     }
+
+    fn clone_box(&self) -> Box<dyn CompressionEngine> {
+        Box::new(self.clone())
+    }
 }

src/compression_engine/mod.rs

@@ -0,0 +1,251 @@
+use anyhow::{Result, anyhow};
+use std::io;
+use std::io::{Read, Write};
+use std::path::PathBuf;
+use strum::IntoEnumIterator;
+use strum::{Display, EnumIter, EnumString};
+
+use log::*;
+
+extern crate enum_map;
+use enum_map::enum_map;
+use enum_map::{Enum, EnumMap};
+
+pub mod gzip;
+pub mod lz4;
+pub mod program;
+pub mod raw;
+pub mod zstd;
+
+use crate::compression_engine::program::CompressionEngineProgram;
+
+/// Enum representing different compression types supported by the system.
+///
+/// This enum defines all supported compression formats that can be used for
+/// storing and retrieving compressed items. Each variant corresponds to a
+/// specific compression algorithm or no compression.
+///
+/// # Examples
+///
+/// ```ignore
+/// assert_eq!(CompressionType::GZip.to_string(), "gzip");
+/// ```
+#[derive(Debug, Eq, PartialEq, Clone, EnumIter, Display, EnumString, enum_map::Enum)]
+#[strum(ascii_case_insensitive)]
+pub enum CompressionType {
+    #[strum(serialize = "lz4")]
+    LZ4,
+    #[strum(serialize = "gzip")]
+    GZip,
+    #[strum(serialize = "bzip2")]
+    BZip2,
+    #[strum(serialize = "xz")]
+    XZ,
+    #[strum(serialize = "zstd")]
+    ZStd,
+    #[strum(to_string = "raw", serialize = "raw", serialize = "none")]
+    Raw,
+}
+
+/// Trait defining the interface for compression engines.
+///
+/// This trait provides a unified API for different compression implementations.
+/// Implementors handle reading from and writing to compressed files, as well as
+/// utility operations like copying decompressed content or calculating sizes.
+///
+/// # Errors
+///
+/// Methods may return `anyhow::Error` for I/O failures, unsupported formats,
+/// or invalid file paths.
+///
+/// # Examples
+///
+/// ```ignore
+/// // Example usage would depend on a concrete implementation
+/// use keep::compression_engine::CompressionEngine;
+/// let engine = /* some engine */;
+/// let reader = engine.open("file.gz".into()).unwrap();
+/// ```
+pub trait CompressionEngine: Send + Sync {
+    /// Opens a compressed file for reading.
+    ///
+    /// Creates a reader that transparently decompresses the file contents as they are read.
+    ///
+    /// # Arguments
+    ///
+    /// * `file_path` - Path to the compressed file.
+    ///
+    /// # Returns
+    ///
+    /// * `Result<Box<dyn Read + Send>>` - A boxed reader that decompresses the file on read,
+    ///   or an error if the file cannot be opened or is invalid.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error if the file does not exist, is not a valid compressed file,
+    /// or if decompression fails.
+    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read + Send>>;
+
+    /// Creates a new compressed file for writing.
+    ///
+    /// Creates a writer that transparently compresses data as it is written.
+    ///
+    /// # Arguments
+    ///
+    /// * `file_path` - Path where the compressed file will be created.
+    ///
+    /// # Returns
+    ///
+    /// * `Result<Box<dyn Write>>` - A boxed writer that compresses data on write,
+    ///   or an error if the file cannot be created.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error if the path is invalid or if there are permission issues.
+    fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>>;
+
+    /// Checks if this compression engine is supported on the current system.
+    ///
+    /// Some compression types may require external programs or features to be enabled.
+    ///
+    /// # Returns
+    ///
+    /// * `bool` - True if supported, false otherwise.
+    fn is_supported(&self) -> bool {
+        true
+    }
+
+    /// Checks if this compression engine is internal (built-in) or external (program-based).
+    ///
+    /// Internal engines use Rust implementations without external dependencies.
+    /// External engines rely on system programs.
+    ///
+    /// # Returns
+    ///
+    /// * `bool` - True if internal, false if external.
+    fn is_internal(&self) -> bool {
+        true
+    }
+
+    /// Returns status information for this compression engine.
+    ///
+    /// For internal engines, returns ("<INTERNAL>", "", "").
+    /// For external program engines, returns (program_binary, compress_args, decompress_args).
+    ///
+    /// # Returns
+    ///
+    /// A tuple of (binary, compress_command, decompress_command).
+    fn get_status_info(&self) -> (String, String, String) {
+        ("<INTERNAL>".to_string(), "".to_string(), "".to_string())
+    }
+
+    /// Copies decompressed content from a file to a writer.
+    ///
+    /// Reads the compressed file and writes the decompressed content to the provided writer.
+    ///
+    /// # Arguments
+    ///
+    /// * `file_path` - Path to the compressed file.
+    /// * `writer` - Writer to receive decompressed content.
+    ///
+    /// # Returns
+    ///
+    /// * `Result<()>` - Success if the copy completes, or an error.
+    ///
+    /// # Errors
+    ///
+    /// Propagates errors from opening the file or copying data.
+    fn copy(&self, file_path: PathBuf, writer: &mut dyn Write) -> Result<()> {
+        let mut reader = self.open(file_path)?;
+        io::copy(&mut reader, writer)?;
+        writer.flush()?;
+        Ok(())
+    }
+
+    /// Clones this compression engine into a new boxed instance.
+    ///
+    /// Required for dynamic trait object cloning.
+    ///
+    /// # Returns
+    ///
+    /// A new `Box<dyn CompressionEngine>` clone of this engine.
+    fn clone_box(&self) -> Box<dyn CompressionEngine>;
+}
+
+impl Clone for Box<dyn CompressionEngine> {
+    fn clone(&self) -> Self {
+        self.as_ref().clone_box()
+    }
+}
+
+fn init_compression_engines() -> EnumMap<CompressionType, Box<dyn CompressionEngine>> {
+    #[allow(unused_mut)]
+    let mut em: EnumMap<CompressionType, Box<dyn CompressionEngine>> = enum_map! {
+        CompressionType::LZ4 => Box::new(crate::compression_engine::program::CompressionEngineProgram::new(
+            "lz4",
+            vec!["-c"],
+            vec!["-d", "-c"]
+        )) as Box<dyn CompressionEngine>,
+        CompressionType::GZip => Box::new(crate::compression_engine::program::CompressionEngineProgram::new(
+            "gzip",
+            vec!["-c"],
+            vec!["-d", "-c"]
+        )) as Box<dyn CompressionEngine>,
+        CompressionType::BZip2 => Box::new(crate::compression_engine::program::CompressionEngineProgram::new(
+            "bzip2",
+            vec!["-c"],
+            vec!["-d", "-c"]
+        )) as Box<dyn CompressionEngine>,
+        CompressionType::XZ => Box::new(crate::compression_engine::program::CompressionEngineProgram::new(
+            "xz",
+            vec!["-c"],
+            vec!["-d", "-c"]
+        )) as Box<dyn CompressionEngine>,
+        CompressionType::ZStd => Box::new(crate::compression_engine::program::CompressionEngineProgram::new(
+            "zstd",
+            vec!["-c"],
+            vec!["-d", "-c"]
+        )) as Box<dyn CompressionEngine>,
+        CompressionType::Raw => Box::new(crate::compression_engine::raw::CompressionEngineRaw::new()) as Box<dyn CompressionEngine>
+    };
+
+    #[cfg(feature = "gzip")]
+    {
+        em[CompressionType::GZip] =
+            Box::new(crate::compression_engine::gzip::CompressionEngineGZip::new())
+                as Box<dyn CompressionEngine>;
+    }
+
+    #[cfg(feature = "lz4")]
+    {
+        em[CompressionType::LZ4] =
+            Box::new(crate::compression_engine::lz4::CompressionEngineLZ4::new())
+                as Box<dyn CompressionEngine>;
+    }
+
+    #[cfg(feature = "zstd")]
+    {
+        em[CompressionType::ZStd] =
+            Box::new(crate::compression_engine::zstd::CompressionEngineZstd::new())
+                as Box<dyn CompressionEngine>;
+    }
+
+    em
+}
+
+static COMPRESSION_ENGINES: std::sync::LazyLock<
+    EnumMap<CompressionType, Box<dyn CompressionEngine>>,
+> = std::sync::LazyLock::new(init_compression_engines);
+
+pub fn default_compression_type() -> CompressionType {
+    CompressionType::LZ4
+}
+
+pub fn get_compression_engine(ct: CompressionType) -> Result<Box<dyn CompressionEngine>> {
+    let engine = &COMPRESSION_ENGINES[ct.clone()];
+    if engine.is_supported() {
+        Ok(engine.clone())
+    } else {
+        Err(anyhow!("Compression engine for {ct} is not supported",))
+    }
+}

src/compression_engine/none.rs

@@ -1,33 +0,0 @@
-use anyhow::Result;
-use log::*;
-use std::fs::File;
-use std::io::{Read, Write};
-use std::path::PathBuf;
-
-use crate::compression_engine::CompressionEngine;
-
-#[derive(Debug, Eq, PartialEq, Clone, Default)]
-pub struct CompressionEngineNone {}
-
-impl CompressionEngineNone {
-    pub fn new() -> CompressionEngineNone {
-        CompressionEngineNone {}
-    }
-}
-
-impl CompressionEngine for CompressionEngineNone {
-    fn size(&self, file_path: PathBuf) -> Result<usize> {
-        let item_file_metadata = file_path.metadata()?;
-        Ok(item_file_metadata.len() as usize)
-    }
-
-    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read>> {
-        debug!("COMPRESSION: Opening {:?} using {:?}", file_path, *self);
-        Ok(Box::new(File::open(file_path)?))
-    }
-
-    fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>> {
-        debug!("COMPRESSION: Writting to {:?} using {:?}", file_path, *self);
-        Ok(Box::new(File::create(file_path)?))
-    }
-}

src/compression_engine/program.rs

@@ -1,12 +1,10 @@
 use anyhow::{Context, Result, anyhow};
 use log::*;
-use std::env;
-use std::fs;
 use std::fs::File;
 use std::io::{Read, Write};
-use std::os::unix::fs::PermissionsExt;
 use std::path::PathBuf;
 use std::process::{Child, Command, Stdio};
+use which::which;
 
 use crate::compression_engine::CompressionEngine;
 
@@ -17,7 +15,13 @@ pub struct ProgramReader {
 
 impl Read for ProgramReader {
     fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
-        self.stdout.as_mut().unwrap().read(buf)
+        match self.stdout.as_mut() {
+            Some(stdout) => stdout.read(buf),
+            None => Err(std::io::Error::new(
+                std::io::ErrorKind::BrokenPipe,
+                "stdout already taken",
+            )),
+        }
     }
 }
 
@@ -35,11 +39,23 @@ pub struct ProgramWriter {
 
 impl Write for ProgramWriter {
     fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
-        self.stdin.as_mut().unwrap().write(buf)
+        match self.stdin.as_mut() {
+            Some(stdin) => stdin.write(buf),
+            None => Err(std::io::Error::new(
+                std::io::ErrorKind::BrokenPipe,
+                "stdin already taken",
+            )),
+        }
     }
 
     fn flush(&mut self) -> std::io::Result<()> {
-        self.stdin.as_mut().unwrap().flush()
+        match self.stdin.as_mut() {
+            Some(stdin) => stdin.flush(),
+            None => Err(std::io::Error::new(
+                std::io::ErrorKind::BrokenPipe,
+                "stdin already taken",
+            )),
+        }
     }
 }
 
@@ -66,11 +82,12 @@ impl CompressionEngineProgram {
         compress: Vec<&str>,
         decompress: Vec<&str>,
     ) -> CompressionEngineProgram {
-        let program_path = get_program_path(program);
+        let program_path = which(program);
         let supported = program_path.is_ok();
 
         CompressionEngineProgram {
-            program: program_path.unwrap_or(program.to_string()),
+            program: program_path
+                .map_or_else(|_| program.to_string(), |p| p.to_string_lossy().to_string()),
             compress: compress.iter().map(|s| s.to_string()).collect(),
             decompress: decompress.iter().map(|s| s.to_string()).collect(),
             supported,
@@ -78,39 +95,30 @@ impl CompressionEngineProgram {
     }
 }
 
-fn get_program_path(program: &str) -> Result<String> {
-    debug!("COMPRESSION: Looking for executable: {}", program);
-    if let Ok(path) = env::var("PATH") {
-        for p in path.split(':') {
-            let p_str = format!("{}/{}", p, program);
-            let stat = fs::metadata(p_str.clone());
-            if let Ok(stat) = stat {
-                let md = stat;
-                let permissions = md.permissions();
-                if md.is_file() && permissions.mode() & 0o111 != 0 {
-                    return Ok(p_str);
-                }
-            }
-        }
-    }
-    Err(anyhow!("Unable to find binary {} in PATH", program))
-}
-
 impl CompressionEngine for CompressionEngineProgram {
     fn is_supported(&self) -> bool {
         self.supported
     }
 
-    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read>> {
-        debug!("COMPRESSION: Opening {:?} using {:?}", file_path, *self);
+    fn is_internal(&self) -> bool {
+        false
+    }
+
+    fn get_status_info(&self) -> (String, String, String) {
+        (
+            self.program.clone(),
+            self.compress.join(" "),
+            self.decompress.join(" "),
+        )
+    }
+
+    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read + Send>> {
+        debug!("COMPRESSION: Opening {file_path:?} using {self:?}");
 
         let program = self.program.clone();
         let args = self.decompress.clone();
 
-        debug!(
-            "COMPRESSION: Executing command: {:?} {:?} reading from {:?}",
-            program, args, file_path
-        );
+        debug!("COMPRESSION: Executing command: {program:?} {args:?} reading from {file_path:?}");
 
         let file = File::open(file_path).context("Unable to open file for reading")?;
 
@@ -125,9 +133,10 @@ impl CompressionEngine for CompressionEngineProgram {
                 args
             ))?;
 
-        let stdout = process.stdout.take().ok_or_else(|| {
-            anyhow!("Failed to capture stdout from child process")
-        })?;
+        let stdout = process
+            .stdout
+            .take()
+            .ok_or_else(|| anyhow!("Failed to capture stdout from child process"))?;
 
         Ok(Box::new(ProgramReader {
             process,
@@ -136,15 +145,12 @@ impl CompressionEngine for CompressionEngineProgram {
     }
 
     fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>> {
-        debug!("COMPRESSION: Writing to {:?} using {:?}", file_path, *self);
+        debug!("COMPRESSION: Writing to {file_path:?} using {self:?}");
 
         let program = self.program.clone();
         let args = self.compress.clone();
 
-        debug!(
-            "COMPRESSION: Executing command: {:?} {:?} writing to {:?}",
-            program, args, file_path
-        );
+        debug!("COMPRESSION: Executing command: {program:?} {args:?} writing to {file_path:?}");
 
         let file = File::create(file_path).context("Unable to open file for writing")?;
 
@@ -159,13 +165,18 @@ impl CompressionEngine for CompressionEngineProgram {
                 args
             ))?;
 
-        let stdin = process.stdin.take().ok_or_else(|| {
-            anyhow!("Failed to capture stdin from child process")
-        })?;
+        let stdin = process
+            .stdin
+            .take()
+            .ok_or_else(|| anyhow!("Failed to capture stdin from child process"))?;
 
         Ok(Box::new(ProgramWriter {
             process,
             stdin: Some(stdin),
         }))
     }
+
+    fn clone_box(&self) -> Box<dyn CompressionEngine> {
+        Box::new(self.clone())
+    }
 }

src/compression_engine/raw.rs

@@ -0,0 +1,40 @@
+use anyhow::Result;
+use log::*;
+use std::fs::File;
+use std::io::{Read, Write};
+use std::path::PathBuf;
+
+use crate::compression_engine::CompressionEngine;
+
+#[derive(Debug, Eq, PartialEq, Clone, Default)]
+pub struct CompressionEngineRaw {}
+
+impl CompressionEngineRaw {
+    pub fn new() -> CompressionEngineRaw {
+        CompressionEngineRaw {}
+    }
+}
+
+impl CompressionEngine for CompressionEngineRaw {
+    fn is_supported(&self) -> bool {
+        true
+    }
+
+    fn get_status_info(&self) -> (String, String, String) {
+        ("<INTERNAL>".to_string(), "".to_string(), "".to_string())
+    }
+
+    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read + Send>> {
+        debug!("COMPRESSION: Opening {:?} using {:?}", file_path, *self);
+        Ok(Box::new(File::open(file_path)?))
+    }
+
+    fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>> {
+        debug!("COMPRESSION: Writing to {:?} using {:?}", file_path, *self);
+        Ok(Box::new(File::create(file_path)?))
+    }
+
+    fn clone_box(&self) -> Box<dyn CompressionEngine> {
+        Box::new(self.clone())
+    }
+}

src/compression_engine/zstd.rs

@@ -0,0 +1,54 @@
+#[cfg(feature = "zstd")]
+use anyhow::Result;
+#[cfg(feature = "zstd")]
+use log::*;
+#[cfg(feature = "zstd")]
+use std::io::Write;
+
+#[cfg(feature = "zstd")]
+use std::fs::File;
+#[cfg(feature = "zstd")]
+use std::io::Read;
+#[cfg(feature = "zstd")]
+use std::path::PathBuf;
+#[cfg(feature = "zstd")]
+use zstd::stream::read::Decoder;
+#[cfg(feature = "zstd")]
+use zstd::stream::write::Encoder;
+
+#[cfg(feature = "zstd")]
+use crate::compression_engine::CompressionEngine;
+
+#[cfg(feature = "zstd")]
+#[derive(Debug, Eq, PartialEq, Clone, Default)]
+pub struct CompressionEngineZstd {}
+
+#[cfg(feature = "zstd")]
+impl CompressionEngineZstd {
+    pub fn new() -> CompressionEngineZstd {
+        CompressionEngineZstd {}
+    }
+}
+
+#[cfg(feature = "zstd")]
+impl CompressionEngine for CompressionEngineZstd {
+    fn open(&self, file_path: PathBuf) -> Result<Box<dyn Read + Send>> {
+        debug!("COMPRESSION: Opening {:?} using {:?}", file_path, *self);
+
+        let file = File::open(file_path)?;
+        Ok(Box::new(Decoder::new(file)?))
+    }
+
+    fn create(&self, file_path: PathBuf) -> Result<Box<dyn Write>> {
+        debug!("COMPRESSION: Writing to {:?} using {:?}", file_path, *self);
+
+        let file = File::create(file_path)?;
+        let zstd_write = Encoder::new(file, 3)?.auto_finish();
+
+        Ok(Box::new(zstd_write))
+    }
+
+    fn clone_box(&self) -> Box<dyn CompressionEngine> {
+        Box::new(self.clone())
+    }
+}

src/config.rs

@@ -0,0 +1,842 @@
+use crate::args::Args;
+use anyhow::{Context, Result};
+use dirs;
+use log::{debug, error};
+use serde::{Deserialize, Serialize};
+use std::fs;
+use std::path::{Path, PathBuf};
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+#[serde(rename_all = "lowercase")]
+pub enum ColumnAlignment {
+    #[default]
+    Left,
+    Right,
+    Center,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+#[serde(rename_all = "lowercase")]
+pub enum ContentArrangement {
+    #[default]
+    Dynamic,
+    DynamicFullWidth,
+    Disabled,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+#[serde(rename_all = "lowercase")]
+pub enum TableStyle {
+    Ascii,
+    Utf8,
+    Utf8Full,
+    #[default]
+    Nothing,
+    Custom(String),
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+pub enum TableColor {
+    Black,
+    Red,
+    Green,
+    Yellow,
+    Blue,
+    Magenta,
+    Cyan,
+    White,
+    Gray,
+    DarkRed,
+    DarkGreen,
+    DarkYellow,
+    DarkBlue,
+    DarkMagenta,
+    DarkCyan,
+    Rgb(u8, u8, u8),
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+pub enum TableAttribute {
+    Bold,
+    Dim,
+    Italic,
+    Underlined,
+    SlowBlink,
+    RapidBlink,
+    Reverse,
+    Hidden,
+    CrossedOut,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct TableConfig {
+    #[serde(default)]
+    pub style: TableStyle,
+    #[serde(default)]
+    pub modifiers: Vec<String>,
+    #[serde(default)]
+    pub content_arrangement: ContentArrangement,
+    #[serde(default)]
+    pub truncation_indicator: String,
+}
+
+#[derive(Debug, Clone, Serialize, Default)]
+pub struct ColumnConfig {
+    pub name: String,
+    pub label: String,
+    #[serde(default)]
+    pub align: ColumnAlignment,
+    #[serde(default)]
+    pub max_len: Option<String>,
+    #[serde(default)]
+    pub fg_color: Option<TableColor>,
+    #[serde(default)]
+    pub bg_color: Option<TableColor>,
+    #[serde(default)]
+    pub attributes: Vec<TableAttribute>,
+    #[serde(default)]
+    pub padding: Option<(u16, u16)>,
+}
+
+impl<'de> serde::Deserialize<'de> for ColumnConfig {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        #[derive(Deserialize)]
+        struct Helper {
+            name: String,
+            label: Option<String>,
+            #[serde(default)]
+            align: ColumnAlignment,
+            #[serde(default)]
+            max_len: Option<String>,
+            #[serde(default)]
+            fg_color: Option<TableColor>,
+            #[serde(default)]
+            bg_color: Option<TableColor>,
+            #[serde(default)]
+            attributes: Vec<TableAttribute>,
+            #[serde(default)]
+            padding: Option<(u16, u16)>,
+        }
+
+        let helper = Helper::deserialize(deserializer)?;
+        let label = helper.label.unwrap_or_else(|| helper.name.clone());
+
+        Ok(ColumnConfig {
+            name: helper.name,
+            label,
+            align: helper.align,
+            max_len: helper.max_len,
+            fg_color: helper.fg_color,
+            bg_color: helper.bg_color,
+            attributes: helper.attributes,
+            padding: helper.padding,
+        })
+    }
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct ServerConfig {
+    pub address: Option<String>,
+    pub port: Option<u16>,
+    pub username: Option<String>,
+    pub password_file: Option<PathBuf>,
+    pub password: Option<String>,
+    pub password_hash: Option<String>,
+    pub jwt_secret: Option<String>,
+    pub jwt_secret_file: Option<PathBuf>,
+    pub cert_file: Option<PathBuf>,
+    pub key_file: Option<PathBuf>,
+    pub cors_origin: Option<String>,
+    pub max_body_size: Option<u64>,
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct CompressionPluginConfig {
+    pub name: String,
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct ClientConfig {
+    pub url: Option<String>,
+    pub username: Option<String>,
+    pub password: Option<String>,
+    pub jwt: Option<String>,
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[cfg_attr(feature = "server", derive(utoipa::ToSchema))]
+pub struct MetaPluginConfig {
+    pub name: String,
+    #[serde(default)]
+    #[cfg_attr(feature = "server", schema(value_type = Object))]
+    pub options: std::collections::HashMap<String, serde_yaml::Value>,
+    #[serde(default)]
+    #[cfg_attr(feature = "server", schema(value_type = Object))]
+    pub outputs: std::collections::HashMap<String, String>,
+}
+
+/// Unified settings that merges config file and CLI arguments
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct Settings {
+    #[serde(default)]
+    pub dir: PathBuf,
+    #[serde(default)]
+    pub list_format: Vec<ColumnConfig>,
+    #[serde(default)]
+    pub table_config: TableConfig,
+    #[serde(default)]
+    pub human_readable: bool,
+    #[serde(default)]
+    pub ids_only: bool,
+    pub output_format: Option<String>,
+    #[serde(default)]
+    pub quiet: bool,
+    #[serde(default)]
+    pub force: bool,
+    pub server: Option<ServerConfig>,
+    pub compression_plugin: Option<CompressionPluginConfig>,
+    pub meta_plugins: Option<Vec<MetaPluginConfig>>,
+    pub client: Option<ClientConfig>,
+    // Non-serializable fields populated from CLI args
+    #[serde(skip)]
+    pub client_url: Option<String>,
+    #[serde(skip)]
+    pub client_username: Option<String>,
+    #[serde(skip)]
+    pub client_password: Option<String>,
+    #[serde(skip)]
+    pub client_jwt: Option<String>,
+    // Metadata key-value pairs from --meta CLI flag
+    #[serde(skip)]
+    pub meta: Vec<(String, Option<String>)>,
+    // Export filename format template (--export-filename-format)
+    #[serde(skip)]
+    pub export_filename_format: String,
+    // Export name for {name} variable (--export-name)
+    #[serde(skip)]
+    pub export_name: Option<String>,
+    // Import data file path (--import-data-file)
+    #[serde(skip)]
+    pub import_data_file: Option<std::path::PathBuf>,
+}
+
+impl Settings {
+    /// Create unified settings from config and args with proper priority
+    pub fn new(args: &Args, default_dir: PathBuf) -> Result<Self> {
+        debug!("CONFIG: Creating settings with default dir: {default_dir:?}");
+
+        let config_path = if let Some(config_path) = &args.options.config {
+            config_path.clone()
+        } else if let Ok(env_config) = std::env::var("KEEP_CONFIG") {
+            PathBuf::from(env_config)
+        } else {
+            let default_path = dirs::config_dir()
+                .map(|mut p| {
+                    p.push("keep");
+                    p.push("config.yml");
+                    p
+                })
+                .unwrap_or_else(|| PathBuf::from("~/.config/keep/config.yml"));
+            debug!("CONFIG: Using default config path: {default_path:?}");
+            default_path
+        };
+
+        debug!("CONFIG: Using config path: {config_path:?}");
+
+        let mut config_builder = config::Config::builder();
+
+        // Load config file if it exists
+        if config_path.exists() {
+            debug!("CONFIG: Loading config file: {config_path:?}");
+            config_builder =
+                config_builder.add_source(config::File::from(config_path.clone()).required(false));
+        } else {
+            debug!("CONFIG: Config file does not exist: {config_path:?}");
+        }
+
+        // Add environment variables
+        debug!("CONFIG: Adding environment variables");
+        let env_source = config::Environment::with_prefix("KEEP")
+            .separator("__")
+            .ignore_empty(true);
+        config_builder = config_builder.add_source(env_source);
+
+        // Override with CLI args
+        if let Some(dir) = &args.options.dir {
+            debug!("CONFIG: Overriding dir with CLI arg: {dir:?}");
+            config_builder = config_builder.set_override(
+                "dir",
+                dir.to_str()
+                    .ok_or_else(|| anyhow::anyhow!("non-UTF-8 directory path"))?,
+            )?;
+        }
+
+        if args.options.human_readable {
+            config_builder = config_builder.set_override("human_readable", true)?;
+        }
+
+        if args.options.ids_only {
+            config_builder = config_builder.set_override("ids_only", true)?;
+        }
+
+        if let Some(output_format) = &args.options.output_format {
+            config_builder =
+                config_builder.set_override("output_format", output_format.as_str())?;
+        }
+
+        if args.options.verbose > 0 {
+            config_builder = config_builder.set_override("verbose", args.options.verbose)?;
+        }
+
+        if args.options.quiet {
+            config_builder = config_builder.set_override("quiet", true)?;
+        }
+
+        if args.options.force {
+            config_builder = config_builder.set_override("force", true)?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_password) = &args.options.server_password {
+            config_builder =
+                config_builder.set_override("server.password", server_password.as_str())?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_password_hash) = &args.options.server_password_hash {
+            config_builder = config_builder
+                .set_override("server.password_hash", server_password_hash.as_str())?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_username) = &args.options.server_username {
+            config_builder =
+                config_builder.set_override("server.username", server_username.as_str())?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_address) = &args.mode.server_address {
+            config_builder =
+                config_builder.set_override("server.address", server_address.as_str())?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_port) = args.mode.server_port {
+            config_builder = config_builder.set_override("server.port", server_port)?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_cert) = &args.mode.server_cert {
+            config_builder = config_builder
+                .set_override("server.cert_file", server_cert.to_string_lossy().as_ref())?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(server_key) = &args.mode.server_key {
+            config_builder = config_builder
+                .set_override("server.key_file", server_key.to_string_lossy().as_ref())?;
+        }
+
+        #[cfg(feature = "server")]
+        if let Some(max_body_size) = args.options.server_max_body_size {
+            config_builder = config_builder.set_override("server.max_body_size", max_body_size)?;
+        }
+
+        if let Some(compression) = &args.item.compression {
+            config_builder =
+                config_builder.set_override("compression_plugin.name", compression.as_str())?;
+        }
+
+        // Build MetaPluginConfig entries from --meta-plugin args (name[:json])
+        // These are handled after config deserialization (see below).
+
+        let config = config_builder.build()?;
+        debug!("CONFIG: Built config, attempting to deserialize");
+
+        match config.try_deserialize::<Settings>() {
+            Ok(mut settings) => {
+                debug!("CONFIG: Successfully deserialized settings: {settings:?}");
+
+                // Set defaults for list_format if not provided
+                if settings.list_format.is_empty() {
+                    debug!("CONFIG: Setting default list_format");
+                    settings.list_format = vec![
+                        ColumnConfig {
+                            name: "id".to_string(),
+                            label: "Item".to_string(),
+                            align: ColumnAlignment::Right,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                        ColumnConfig {
+                            name: "time".to_string(),
+                            label: "Time".to_string(),
+                            align: ColumnAlignment::Right,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                        ColumnConfig {
+                            name: "size".to_string(),
+                            label: "Size".to_string(),
+                            align: ColumnAlignment::Right,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                        ColumnConfig {
+                            name: "meta:text_line_count".to_string(),
+                            label: "Lines".to_string(),
+                            align: ColumnAlignment::Right,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                        ColumnConfig {
+                            name: "tags".to_string(),
+                            label: "Tags".to_string(),
+                            align: ColumnAlignment::Left,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                        ColumnConfig {
+                            name: "meta:hostname_short".to_string(),
+                            label: "Host".to_string(),
+                            align: ColumnAlignment::Left,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                        ColumnConfig {
+                            name: "meta:command".to_string(),
+                            label: "Command".to_string(),
+                            align: ColumnAlignment::Left,
+                            max_len: None,
+                            fg_color: None,
+                            bg_color: None,
+                            attributes: Vec::new(),
+                            padding: None,
+                        },
+                    ];
+                }
+
+                // Set default meta_plugins to include 'env' if not provided
+                if settings.meta_plugins.is_none() {
+                    debug!("CONFIG: Setting default meta_plugins to include 'env'");
+                    settings.meta_plugins = Some(vec![MetaPluginConfig {
+                        name: "env".to_string(),
+                        options: std::collections::HashMap::new(),
+                        outputs: std::collections::HashMap::new(),
+                    }]);
+                }
+
+                // Override meta_plugins from --meta-plugin CLI args
+                if !args.item.meta_plugins.is_empty() {
+                    debug!("CONFIG: Overriding meta_plugins from --meta-plugin CLI args");
+                    let cli_plugins: Vec<MetaPluginConfig> = args
+                        .item
+                        .meta_plugins
+                        .iter()
+                        .map(|arg| {
+                            let mut options = std::collections::HashMap::new();
+                            let mut outputs = std::collections::HashMap::new();
+                            if let Some(serde_json::Value::Object(obj)) = &arg.options {
+                                // Extract options and outputs from JSON value
+                                if let Some(serde_json::Value::Object(opts_obj)) =
+                                    obj.get("options")
+                                {
+                                    for (k, v) in opts_obj {
+                                        let yaml_str = serde_json::to_string(v).unwrap_or_default();
+                                        let yaml_val: serde_yaml::Value =
+                                            serde_yaml::from_str(&yaml_str)
+                                                .unwrap_or(serde_yaml::Value::Null);
+                                        options.insert(k.clone(), yaml_val);
+                                    }
+                                }
+                                if let Some(serde_json::Value::Object(outs_obj)) =
+                                    obj.get("outputs")
+                                {
+                                    for (k, v) in outs_obj {
+                                        let val_str = match v {
+                                            serde_json::Value::String(s) => s.clone(),
+                                            _ => v.to_string(),
+                                        };
+                                        outputs.insert(k.clone(), val_str);
+                                    }
+                                }
+                            }
+                            MetaPluginConfig {
+                                name: arg.name.clone(),
+                                options,
+                                outputs,
+                            }
+                        })
+                        .collect();
+                    settings.meta_plugins = Some(cli_plugins);
+                }
+
+                // Override list_format from --list-format CLI arg
+                if args.options.list_format
+                    != "id,time,size,meta:text_line_count,tags,meta:hostname_short,meta:command"
+                {
+                    debug!("CONFIG: Overriding list_format from --list-format CLI arg");
+                    settings.list_format = Settings::parse_list_format(&args.options.list_format);
+                }
+
+                // Set dir to default if not provided or is empty
+                if settings.dir == PathBuf::new() {
+                    debug!("CONFIG: Setting default dir: {default_dir:?}");
+                    settings.dir = default_dir;
+                }
+
+                // Populate client settings from CLI args and config
+                #[cfg(feature = "client")]
+                {
+                    settings.client_url = args
+                        .options
+                        .client_url
+                        .clone()
+                        .or_else(|| settings.client.as_ref().and_then(|c| c.url.clone()));
+                    settings.client_username = args
+                        .options
+                        .client_username
+                        .clone()
+                        .or_else(|| settings.client.as_ref().and_then(|c| c.username.clone()));
+                    settings.client_password = args
+                        .options
+                        .client_password
+                        .clone()
+                        .or_else(|| settings.client.as_ref().and_then(|c| c.password.clone()));
+                    settings.client_jwt = args
+                        .options
+                        .client_jwt
+                        .clone()
+                        .or_else(|| settings.client.as_ref().and_then(|c| c.jwt.clone()));
+                }
+
+                // Parse --meta key=value and bare key arguments
+                settings.meta = args
+                    .item
+                    .meta
+                    .iter()
+                    .map(|s| {
+                        if let Some((key, value)) = s.split_once('=') {
+                            (key.to_string(), Some(value.to_string()))
+                        } else {
+                            (s.to_string(), None)
+                        }
+                    })
+                    .collect();
+
+                // Set export filename format from CLI args
+                settings.export_filename_format = args.item.export_filename_format.clone();
+                settings.export_name = args.item.export_name.clone();
+                settings.import_data_file = args.item.import_data_file.clone();
+
+                // Expand ~ in all path fields
+                settings.dir = Settings::expand_tilde(&settings.dir);
+                settings.import_data_file = settings
+                    .import_data_file
+                    .as_ref()
+                    .map(|p| Settings::expand_tilde(p));
+                if let Some(ref mut server) = settings.server {
+                    server.password_file = server
+                        .password_file
+                        .as_ref()
+                        .map(|p| Settings::expand_tilde(p));
+                    server.jwt_secret_file = server
+                        .jwt_secret_file
+                        .as_ref()
+                        .map(|p| Settings::expand_tilde(p));
+                    server.cert_file = server.cert_file.as_ref().map(|p| Settings::expand_tilde(p));
+                    server.key_file = server.key_file.as_ref().map(|p| Settings::expand_tilde(p));
+                }
+
+                debug!("CONFIG: Final settings: {settings:?}");
+                Ok(settings)
+            }
+            Err(e) => {
+                error!("CONFIG: Failed to deserialize settings: {e}");
+                Err(e.into())
+            }
+        }
+    }
+
+    pub fn default_dir() -> anyhow::Result<PathBuf> {
+        let mut path =
+            dirs::data_dir().ok_or_else(|| anyhow::anyhow!("No data directory found"))?;
+        path.push("keep");
+        if !path.exists() {
+            std::fs::create_dir_all(&path)?;
+        }
+        Ok(path)
+    }
+
+    /// Expand a leading `~` in a path to the user's home directory.
+    ///
+    /// Returns the path unchanged if it doesn't start with `~` or if the
+    /// home directory cannot be determined.
+    fn expand_tilde(path: &Path) -> PathBuf {
+        let path_str = path.to_string_lossy();
+        if let Some(rest) = path_str.strip_prefix("~/") {
+            if let Some(home) = dirs::home_dir() {
+                return home.join(rest);
+            }
+        } else if path_str == "~" {
+            if let Some(home) = dirs::home_dir() {
+                return home;
+            }
+        }
+        path.to_path_buf()
+    }
+
+    /// Get server password from password_file or directly from config if configured
+    pub fn get_server_password(&self) -> Result<Option<String>> {
+        if let Some(server) = &self.server {
+            // First check for password_file
+            if let Some(password_file) = &server.password_file {
+                debug!("CONFIG: Reading password from file: {password_file:?}");
+                let password = fs::read(password_file)
+                    .with_context(|| format!("Failed to read password file: {password_file:?}"))?;
+                let end = password.len().min(4096);
+                let password = String::from_utf8_lossy(&password[..end]).trim().to_string();
+                return Ok(Some(password));
+            }
+
+            // Fall back to direct password field
+            if let Some(password) = &server.password {
+                debug!("CONFIG: Using password from config");
+                return Ok(Some(password.clone()));
+            }
+        }
+        Ok(None)
+    }
+
+    // Helper methods to access configuration values
+    pub fn server_password(&self) -> Option<String> {
+        self.get_server_password().ok().flatten()
+    }
+
+    pub fn server_password_hash(&self) -> Option<String> {
+        self.server.as_ref().and_then(|s| s.password_hash.clone())
+    }
+
+    pub fn server_username(&self) -> Option<String> {
+        self.server.as_ref().and_then(|s| s.username.clone())
+    }
+
+    /// Get JWT secret from jwt_secret_file or directly from config if configured
+    pub fn get_server_jwt_secret(&self) -> Result<Option<String>> {
+        if let Some(server) = &self.server {
+            // First check for jwt_secret_file
+            if let Some(jwt_secret_file) = &server.jwt_secret_file {
+                debug!("CONFIG: Reading JWT secret from file: {jwt_secret_file:?}");
+                let secret = fs::read(jwt_secret_file).with_context(|| {
+                    format!("Failed to read JWT secret file: {jwt_secret_file:?}")
+                })?;
+                let end = secret.len().min(4096);
+                let secret = String::from_utf8_lossy(&secret[..end]).trim().to_string();
+                return Ok(Some(secret));
+            }
+
+            // Fall back to direct jwt_secret field
+            if let Some(secret) = &server.jwt_secret {
+                debug!("CONFIG: Using JWT secret from config");
+                return Ok(Some(secret.clone()));
+            }
+        }
+        Ok(None)
+    }
+
+    pub fn server_jwt_secret(&self) -> Option<String> {
+        self.get_server_jwt_secret().ok().flatten()
+    }
+
+    pub fn server_address(&self) -> Option<String> {
+        self.server.as_ref().and_then(|s| s.address.clone())
+    }
+
+    pub fn server_port(&self) -> Option<u16> {
+        self.server.as_ref().and_then(|s| s.port)
+    }
+
+    pub fn server_cert_file(&self) -> Option<PathBuf> {
+        self.server.as_ref().and_then(|s| s.cert_file.clone())
+    }
+
+    pub fn server_key_file(&self) -> Option<PathBuf> {
+        self.server.as_ref().and_then(|s| s.key_file.clone())
+    }
+
+    pub fn server_cors_origin(&self) -> Option<String> {
+        self.server.as_ref().and_then(|s| s.cors_origin.clone())
+    }
+
+    pub fn compression(&self) -> Option<String> {
+        self.compression_plugin.as_ref().map(|c| c.name.clone())
+    }
+
+    pub fn meta_plugins_names(&self) -> Vec<String> {
+        self.meta_plugins
+            .as_ref()
+            .map(|plugins| plugins.iter().map(|p| p.name.clone()).collect())
+            .unwrap_or_default()
+    }
+
+    /// Returns the metadata filter as a HashMap.
+    ///
+    /// Converts the `meta` field (list of key-value pairs from CLI --meta flags)
+    /// into a `HashMap<String, Option<String>>` suitable for filtering.
+    pub fn meta_filter(&self) -> std::collections::HashMap<String, Option<String>> {
+        self.meta.iter().cloned().collect()
+    }
+
+    /// Validates the configuration against plugin schemas.
+    ///
+    /// Checks that:
+    /// - All configured meta plugin names are valid and registered
+    /// - Required options are present for each meta plugin
+    /// - Compression plugin name (if set) is a valid compression type
+    ///
+    /// Returns a list of warning strings. An empty list means the config is valid.
+    pub fn validate_config(&self) -> Vec<String> {
+        use crate::common::schema::gather_meta_plugin_schemas;
+        use crate::compression_engine::CompressionType;
+        use strum::IntoEnumIterator;
+
+        let mut warnings = Vec::new();
+
+        // Validate compression plugin
+        if let Some(ref comp) = self.compression_plugin {
+            let valid_types: Vec<String> =
+                CompressionType::iter().map(|ct| ct.to_string()).collect();
+            if !valid_types.contains(&comp.name) {
+                warnings.push(format!(
+                    "Unknown compression_plugin.name: '{}'. Valid types: {}",
+                    comp.name,
+                    valid_types.join(", ")
+                ));
+            }
+        }
+
+        // Validate meta plugins
+        if let Some(ref plugins) = self.meta_plugins {
+            let schemas = gather_meta_plugin_schemas();
+            let schema_map: std::collections::HashMap<&str, &crate::common::schema::PluginSchema> =
+                schemas.iter().map(|s| (s.name.as_str(), s)).collect();
+
+            for plugin in plugins {
+                match schema_map.get(plugin.name.as_str()) {
+                    Some(schema) => {
+                        // Check required options
+                        for opt in &schema.options {
+                            if opt.required && !plugin.options.contains_key(&opt.name) {
+                                warnings.push(format!(
+                                    "Meta plugin '{}': missing required option '{}'",
+                                    plugin.name, opt.name
+                                ));
+                            }
+                        }
+                    }
+                    None => {
+                        warnings.push(format!(
+                            "Unknown meta plugin: '{}'. Available: {}",
+                            plugin.name,
+                            schema_map.keys().copied().collect::<Vec<_>>().join(", ")
+                        ));
+                    }
+                }
+            }
+        }
+
+        warnings
+    }
+
+    /// Parse a comma-separated column list string into Vec<ColumnConfig>.
+    ///
+    /// Maps known column names to their default labels and alignment.
+    /// For unknown names (including meta:* columns), uses the name as its own label.
+    fn parse_list_format(input: &str) -> Vec<ColumnConfig> {
+        input
+            .split(',')
+            .map(|s| s.trim())
+            .filter(|s| !s.is_empty())
+            .map(|name| {
+                let (label, align) = match name {
+                    "id" => ("Item", ColumnAlignment::Right),
+                    "time" => ("Time", ColumnAlignment::Right),
+                    "size" => ("Size", ColumnAlignment::Right),
+                    "meta:text_line_count" => ("Lines", ColumnAlignment::Right),
+                    "meta:token_count" => ("Tokens", ColumnAlignment::Right),
+                    "tags" => ("Tags", ColumnAlignment::Left),
+                    "meta:hostname_short" => ("Host", ColumnAlignment::Left),
+                    "meta:hostname" => ("Host", ColumnAlignment::Left),
+                    "meta:command" => ("Command", ColumnAlignment::Left),
+                    "compression" => ("Compression", ColumnAlignment::Left),
+                    other if other.starts_with("meta:") => {
+                        let sub = other.strip_prefix("meta:").unwrap_or(other);
+                        (sub, ColumnAlignment::Left)
+                    }
+                    other => (other, ColumnAlignment::Left),
+                };
+                ColumnConfig {
+                    name: name.to_string(),
+                    label: label.to_string(),
+                    align,
+                    ..Default::default()
+                }
+            })
+            .collect()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::Path;
+
+    #[test]
+    fn test_expand_tilde_with_slash() {
+        let home = dirs::home_dir().unwrap();
+        let result = Settings::expand_tilde(Path::new("~/foo/bar"));
+        assert_eq!(result, home.join("foo/bar"));
+    }
+
+    #[test]
+    fn test_expand_tilde_bare() {
+        let home = dirs::home_dir().unwrap();
+        let result = Settings::expand_tilde(Path::new("~"));
+        assert_eq!(result, home);
+    }
+
+    #[test]
+    fn test_expand_tilde_absolute() {
+        let result = Settings::expand_tilde(Path::new("/etc/keep"));
+        assert_eq!(result, PathBuf::from("/etc/keep"));
+    }
+
+    #[test]
+    fn test_expand_tilde_relative() {
+        let result = Settings::expand_tilde(Path::new("foo/bar"));
+        assert_eq!(result, PathBuf::from("foo/bar"));
+    }
+}

src/export_tar.rs

@@ -0,0 +1,167 @@
+use anyhow::{Context, Result, anyhow};
+use log::debug;
+use std::collections::HashSet;
+use std::fs;
+use std::io::{Read, Seek, Write};
+use std::path::Path;
+
+use tar::{Builder, Header};
+
+use crate::filter_plugin::FilterChain;
+use crate::modes::common::ExportMeta;
+use crate::services::item_service::ItemService;
+use crate::services::types::ItemWithMeta;
+
+/// Compute the intersection of all items' tag sets.
+///
+/// Returns sorted tags that are present on ALL items.
+pub fn common_tags(items: &[ItemWithMeta]) -> Vec<String> {
+    if items.is_empty() {
+        return Vec::new();
+    }
+
+    let mut common: HashSet<String> = items[0].tag_names().into_iter().collect();
+
+    for item in items.iter().skip(1) {
+        let item_tags: HashSet<String> = item.tag_names().into_iter().collect();
+        common = common.intersection(&item_tags).cloned().collect();
+    }
+
+    let mut result: Vec<String> = common.into_iter().collect();
+    result.sort();
+    result
+}
+
+/// Resolve the export name from the CLI arg or compute default from common tags.
+///
+/// If `arg` is Some, uses that value directly.
+/// Otherwise, computes `export_<common-tags>` or just `export` if no common tags.
+pub fn export_name(arg: &Option<String>, items: &[ItemWithMeta]) -> String {
+    if let Some(name) = arg {
+        return name.clone();
+    }
+
+    let tags = common_tags(items);
+    if tags.is_empty() {
+        "export".to_string()
+    } else {
+        format!("export_{}", tags.join("_"))
+    }
+}
+
+/// Write items to a tar archive, streaming data without loading files into memory.
+///
+/// The archive contains `<dir_name>/<id>.data.<compression>` and
+/// `<dir_name>/<id>.meta.yml` for each item.
+///
+/// # Arguments
+///
+/// * `writer` - The output writer (e.g., a File).
+/// * `dir_name` - Top-level directory name inside the tar.
+/// * `items` - Items to export.
+/// * `data_path` - Path to the data storage directory.
+/// * `filter_chain` - Optional filter chain for transforming content on export.
+/// * `item_service` - Item service for streaming content.
+/// * `conn` - Database connection for filter chain operations.
+pub fn write_export_tar<W: Write>(
+    writer: W,
+    dir_name: &str,
+    items: &[ItemWithMeta],
+    data_path: &Path,
+    filter_chain: Option<&FilterChain>,
+    item_service: &ItemService,
+    conn: &rusqlite::Connection,
+) -> Result<()> {
+    let mut builder = Builder::new(writer);
+
+    for item_with_meta in items {
+        let item_id = item_with_meta.item.id.context("Item missing ID")?;
+
+        let compression = &item_with_meta.item.compression;
+        let item_tags = item_with_meta.tag_names();
+        let meta_map = item_with_meta.meta_as_map();
+
+        let data_path_entry = format!("{dir_name}/{item_id}.data.{compression}");
+        let meta_path_entry = format!("{dir_name}/{item_id}.meta.yml");
+
+        // Meta entry (small, in-memory is fine)
+        let export_meta = ExportMeta {
+            ts: item_with_meta.item.ts,
+            compression: compression.clone(),
+            uncompressed_size: item_with_meta.item.uncompressed_size,
+            tags: item_tags,
+            metadata: meta_map,
+        };
+        let meta_yaml = serde_yaml::to_string(&export_meta)?;
+        let meta_bytes = meta_yaml.into_bytes();
+        let meta_len = meta_bytes.len() as u64;
+
+        let mut meta_header = Header::new_gnu();
+        meta_header.set_size(meta_len);
+        meta_header.set_mode(0o644);
+        meta_header.set_path(&meta_path_entry)?;
+        meta_header.set_cksum();
+        builder
+            .append(&meta_header, meta_bytes.as_slice())
+            .with_context(|| format!("Cannot write meta entry for item {item_id}"))?;
+        debug!("EXPORT_TAR: Wrote meta entry {meta_path_entry}");
+
+        // Data entry
+        let mut item_file_path = data_path.to_path_buf();
+        item_file_path.push(item_id.to_string());
+
+        if let Some(chain) = filter_chain {
+            // Filtered export: spool through filter chain to a temp file,
+            // then stream the temp file into the tar with known size.
+            let (mut reader, _, _) = item_service.get_item_content_info_streaming_with_chain(
+                conn,
+                item_id,
+                Some(chain),
+            )?;
+
+            let mut tmp = tempfile::NamedTempFile::new()
+                .context("Cannot create temp file for filtered export")?;
+            let mut buf = [0u8; crate::common::PIPESIZE];
+            loop {
+                let n = reader.read(&mut buf)?;
+                if n == 0 {
+                    break;
+                }
+                tmp.write_all(&buf[..n])?;
+            }
+            tmp.flush()?;
+            let total_size = tmp.as_file().metadata()?.len();
+            tmp.rewind()?;
+
+            let mut data_header = Header::new_gnu();
+            data_header.set_size(total_size);
+            data_header.set_mode(0o644);
+            data_header.set_path(&data_path_entry)?;
+            data_header.set_cksum();
+            builder
+                .append(&data_header, &mut tmp)
+                .with_context(|| format!("Cannot write data entry for item {item_id}"))?;
+            debug!("EXPORT_TAR: Wrote filtered data entry {data_path_entry} ({total_size} bytes)");
+        } else {
+            // Unfiltered export: stream raw compressed file
+            let file = fs::File::open(&item_file_path)
+                .with_context(|| format!("Cannot open data file: {}", item_file_path.display()))?;
+            let file_size = file.metadata()?.len();
+
+            let mut data_header = Header::new_gnu();
+            data_header.set_size(file_size);
+            data_header.set_mode(0o644);
+            data_header.set_path(&data_path_entry)?;
+            data_header.set_cksum();
+            builder
+                .append(&data_header, file)
+                .with_context(|| format!("Cannot write data entry for item {item_id}"))?;
+            debug!("EXPORT_TAR: Wrote data entry {data_path_entry} ({file_size} bytes)");
+        }
+    }
+
+    builder.finish().context("Cannot finalize tar archive")?;
+    debug!("EXPORT_TAR: Archive finalized");
+
+    Ok(())
+}

src/filter_plugin/exec.rs

@@ -0,0 +1,222 @@
+use super::{FilterOption, FilterPlugin};
+use log::*;
+use std::io::{Read, Result, Write};
+use std::process::{Child, Command, Stdio};
+use which::which;
+
+/// A filter that executes an external program and pipes input through it.
+///
+/// This filter spawns an external command, pipes the input stream to its stdin,
+/// and writes the stdout to the output stream. Supports async-like behavior via
+/// threads for concurrent I/O. Requires the program to be available on PATH.
+#[derive(Debug, Clone)]
+pub struct ExecFilter {
+    program: String,
+    args: Vec<String>,
+    supported: bool,
+    split_whitespace: bool,
+    child_process: Option<Child>,
+    stdin_writer: Option<std::process::ChildStdin>,
+    stdout_reader: Option<std::process::ChildStdout>,
+}
+
+impl ExecFilter {
+    /// Creates a new `ExecFilter` for the specified program and arguments.
+    ///
+    /// Checks if the program is available using `which` and stores the resolved path.
+    ///
+    /// # Arguments
+    ///
+    /// * `program` - The name or path of the program to execute.
+    /// * `args` - A slice of string slices representing the arguments to pass to the program.
+    /// * `split_whitespace` - Whether to split arguments on whitespace when parsing (unused in this context).
+    ///
+    /// # Returns
+    ///
+    /// A new `ExecFilter` instance.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use keep::filter_plugin::exec::ExecFilter;
+    ///
+    /// let filter = ExecFilter::new("grep", vec!["-i", "error"], false);
+    /// assert!(filter.supported);
+    /// ```
+    pub fn new(program: &str, args: Vec<&str>, split_whitespace: bool) -> ExecFilter {
+        let program_path = which(program);
+        let supported = program_path.is_ok();
+
+        ExecFilter {
+            program: program_path
+                .map_or_else(|| program.to_string(), |p| p.to_string_lossy().to_string()),
+            args: args.iter().map(|s| s.to_string()).collect(),
+            supported,
+            split_whitespace,
+            child_process: None,
+            stdin_writer: None,
+            stdout_reader: None,
+        }
+    }
+}
+
+impl FilterPlugin for ExecFilter {
+    /// Filters the input by piping it through the external program and writing the output.
+    ///
+    /// Spawns the process with piped I/O, uses threads for concurrent input/output
+    /// copying, and waits for completion. Errors if the program isn't found or fails.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - A boxed mutable reference to the input reader providing the data stream to pipe to the program.
+    /// * `writer` - A boxed mutable reference to the output writer where the program's output is sent.
+    ///
+    /// # Returns
+    ///
+    /// Returns `Ok(())` on success, or an `io::Error` if process spawning, piping, or execution fails.
+    ///
+    /// # Errors
+    ///
+    /// * NotFound - Program not available.
+    /// * Other - Spawn, I/O, or wait failures.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use keep::filter_plugin::exec::ExecFilter;
+    /// use std::io::{Read, Write};
+    ///
+    /// let mut filter = ExecFilter::new("cat", vec![], false);
+    /// // In filter context:
+    /// filter.filter(Box::new(&mut input), Box::new(&mut output)).unwrap();
+    /// ```
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if !self.supported {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::NotFound,
+                format!("Program '{}' not found", self.program),
+            ));
+        }
+
+        debug!(
+            "FILTER_EXEC: Executing command: {} {:?}",
+            self.program, self.args
+        );
+
+        // Read all input first
+        let mut input_data = Vec::new();
+        std::io::copy(reader, &mut input_data)?;
+
+        let mut child = Command::new(&self.program)
+            .args(&self.args)
+            .stdin(Stdio::piped())
+            .stdout(Stdio::piped())
+            .stderr(Stdio::piped())
+            .spawn()
+            .map_err(|e| {
+                std::io::Error::new(
+                    std::io::ErrorKind::Other,
+                    format!("Failed to spawn process '{}': {}", self.program, e),
+                )
+            })?;
+
+        let mut stdin = child.stdin.take().ok_or_else(|| {
+            std::io::Error::new(
+                std::io::ErrorKind::Other,
+                "Failed to capture stdin from child process",
+            )
+        })?;
+
+        // Write input to child stdin
+        stdin.write_all(&input_data)?;
+        drop(stdin); // Close stdin to signal EOF
+
+        let mut stdout = child.stdout.take().ok_or_else(|| {
+            std::io::Error::new(
+                std::io::ErrorKind::Other,
+                "Failed to capture stdout from child process",
+            )
+        })?;
+
+        // Copy stdout to writer
+        std::io::copy(&mut stdout, writer)?;
+
+        // Wait for the child process to finish
+        let output = child.wait_with_output().map_err(|e| {
+            std::io::Error::new(
+                std::io::ErrorKind::Other,
+                format!("Failed to wait on child process: {}", e),
+            )
+        })?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            if !stderr.is_empty() {
+                warn!("FILTER_EXEC: Process stderr: {}", stderr);
+            }
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::Other,
+                format!("Process exited with error: {:?}", output.status),
+            ));
+        }
+
+        debug!("FILTER_EXEC: Process completed successfully");
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(ExecFilter {
+            program: self.program.clone(),
+            args: self.args.clone(),
+            supported: self.supported,
+            split_whitespace: self.split_whitespace,
+            child_process: None,
+            stdin_writer: None,
+            stdout_reader: None,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    ///
+    /// Defines "command" as required and "split_whitespace" as optional.
+    ///
+    /// # Returns
+    ///
+    /// A vector of `FilterOption` describing the filter's configurable parameters.
+    fn options(&self) -> Vec<FilterOption> {
+        vec![
+            FilterOption {
+                name: "command".to_string(),
+                default: None,
+                required: true,
+            },
+            FilterOption {
+                name: "split_whitespace".to_string(),
+                default: Some(serde_json::Value::Bool(true)),
+                required: false,
+            },
+        ]
+    }
+
+    fn description(&self) -> &str {
+        "Pipe input through an external command"
+    }
+}
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_exec_filter() {
+    crate::services::filter_service::register_filter_plugin("exec", || {
+        // Create a dummy instance - actual creation happens in create method
+        Box::new(ExecFilter {
+            program: String::new(),
+            args: Vec::new(),
+            supported: false,
+            split_whitespace: true,
+            child_process: None,
+            stdin_writer: None,
+            stdout_reader: None,
+        })
+    })
+    .expect("Failed to register exec filter");
+}

src/filter_plugin/grep.rs

@@ -0,0 +1,120 @@
+use super::{FilterOption, FilterPlugin};
+use regex::Regex;
+use std::io::{BufRead, Read, Result, Write};
+
+/// A filter that matches lines against a regular expression pattern.
+///
+/// Outputs only lines that match the given regex. Uses BufRead for line-by-line processing
+/// and preserves original line endings.
+///
+/// # Fields
+///
+/// * `regex` - Compiled regex for matching.
+#[derive(Debug, Clone)]
+pub struct GrepFilter {
+    regex: Regex,
+}
+
+/// Creates a new `GrepFilter` with the specified regex pattern.
+///
+/// Compiles the pattern using regex crate.
+///
+/// # Arguments
+///
+/// * `pattern` - The regular expression pattern (string) used to match lines.
+///
+/// # Returns
+///
+/// `Ok(Self)` on success.
+///
+/// # Errors
+///
+/// Returns `Err(io::Error::InvalidInput)` if pattern compilation fails (invalid regex).
+///
+/// # Examples
+///
+/// ```
+/// # use keep::filter_plugin::GrepFilter;
+/// let filter = GrepFilter::new("error|warn".to_string())?;
+/// # Ok::<(), std::io::Error>(())
+/// ```
+impl GrepFilter {
+    pub fn new(pattern: String) -> Result<Self> {
+        let regex = Regex::new(&pattern)
+            .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidInput, e))?;
+        Ok(Self { regex })
+    }
+}
+
+/// Filters the input by writing only lines that match the regex pattern.
+///
+/// Reads lines from the input and writes matching lines to the output, preserving newlines.
+/// Uses BufReader for efficient line iteration.
+///
+/// # Arguments
+///
+/// * `reader` - Mutable reference to the input data stream.
+/// * `writer` - Mutable reference to the output writer where matching lines are sent.
+///
+/// # Returns
+///
+/// `Ok(())` on success.
+///
+/// # Errors
+///
+/// Propagates `io::Error` from BufRead lines() or writeln! (e.g., read/write failures, UTF-8 issues).
+///
+/// # Examples
+///
+/// ```
+/// # use std::io::{Read, Write, Cursor};
+/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
+/// # let mut filter = GrepFilter::new("error".to_string())?;
+/// let mut input: &mut dyn Read = &mut Cursor::new(b"error: something failed\nok: all good\n");
+/// let mut output = Vec::new();
+/// filter.filter(&mut input, &mut output)?;
+/// # Ok::<(), std::io::Error>(())
+/// ```
+impl FilterPlugin for GrepFilter {
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        let mut buf_reader = std::io::BufReader::new(reader);
+        for line in buf_reader.by_ref().lines() {
+            let line = line?;
+            if self.regex.is_match(&line) {
+                writeln!(writer, "{line}")?;
+            }
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            regex: self.regex.clone(),
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    ///
+    /// The only option is the required "pattern" for the regex.
+    ///
+    /// # Returns
+    ///
+    /// A vector containing one `FilterOption` for "pattern" (required, no default).
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
+    /// let filter = GrepFilter::new("test".to_string()).unwrap();
+    /// let opts = filter.options();
+    /// assert_eq!(opts.len(), 1);
+    /// assert!(opts[0].required);
+    /// ```
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::pattern_option()
+    }
+
+    fn description(&self) -> &str {
+        "Filter lines matching a regex pattern"
+    }
+}

src/filter_plugin/head.rs

@@ -0,0 +1,228 @@
+use super::{FilterOption, FilterPlugin};
+use crate::common::PIPESIZE;
+use crate::services::filter_service::register_filter_plugin;
+use std::io::{BufRead, Read, Result, Write};
+
+#[derive(Clone)]
+pub struct HeadBytesFilter {
+    remaining: usize,
+}
+
+/// A filter that reads the first N bytes from the input stream.
+///
+/// Limits the output to the initial bytes specified in the configuration.
+/// Useful for previewing file contents without reading everything.
+///
+/// # Fields
+///
+/// * `remaining` - Number of bytes left to read before stopping.
+impl HeadBytesFilter {
+    /// Creates a new `HeadBytesFilter` that will read up to the specified number of bytes.
+    ///
+    /// # Arguments
+    ///
+    /// * `count` - The maximum number of bytes to read from the input.
+    ///
+    /// # Returns
+    ///
+    /// A new instance configured to read at most `count` bytes.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::HeadBytesFilter;
+    /// let filter = HeadBytesFilter::new(1024);
+    /// ```
+    pub fn new(count: usize) -> Self {
+        Self { remaining: count }
+    }
+}
+
+/// Filters input by reading only the first N bytes and writing them to the output.
+///
+/// Reads from the input in chunks until the byte limit is reached or EOF, then writes
+/// the collected bytes to the output. Stops early if the limit is zero.
+///
+/// # Arguments
+///
+/// * `reader` - Mutable reference to the input data stream.
+/// * `writer` - Mutable reference to the output stream.
+///
+/// # Returns
+///
+/// * `Result<()>` - Success if filtering completes, or I/O error.
+///
+/// # Errors
+///
+/// * `io::Error` from reading or writing operations.
+///
+/// # Examples
+///
+/// ```
+/// # use std::io::{Read, Write, Cursor};
+/// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
+/// # let mut filter = HeadBytesFilter::new(5);
+/// let mut input: &mut dyn Read = &mut Cursor::new(b"Hello World");
+/// let mut output = Vec::new();
+/// filter.filter(&mut input, &mut output)?;
+/// assert_eq!(output, b"Hello");
+/// # Ok::<(), std::io::Error>(())
+/// ```
+impl FilterPlugin for HeadBytesFilter {
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if self.remaining == 0 {
+            return Ok(());
+        }
+
+        let mut buffer = vec![0; PIPESIZE];
+        while self.remaining > 0 {
+            let to_read = std::cmp::min(self.remaining, PIPESIZE);
+            let bytes_read = reader.read(&mut buffer[..to_read])?;
+            if bytes_read == 0 {
+                break;
+            }
+            writer.write_all(&buffer[..bytes_read])?;
+            self.remaining -= bytes_read;
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            remaining: self.remaining,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    ///
+    /// Defines the "count" parameter as required with no default.
+    ///
+    /// # Returns
+    ///
+    /// Vector of `FilterOption` describing parameters.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
+    /// let filter = HeadBytesFilter::new(100);
+    /// let opts = filter.options();
+    /// assert_eq!(opts.len(), 1);
+    /// assert_eq!(opts[0].name, "count");
+    /// assert!(opts[0].required);
+    /// ```
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::count_option()
+    }
+
+    fn description(&self) -> &str {
+        "Read the first N bytes"
+    }
+}
+
+#[derive(Clone)]
+pub struct HeadLinesFilter {
+    remaining: usize,
+}
+
+/// A filter that reads the first N lines from the input stream.
+///
+/// Limits output to the initial lines specified, writing each full line to output.
+/// Handles line endings properly using buffered reading.
+///
+/// # Fields
+///
+/// * `remaining` - Number of lines left to read before stopping.
+impl HeadLinesFilter {
+    /// Creates a new `HeadLinesFilter` that will read up to the specified number of lines.
+    ///
+    /// # Arguments
+    ///
+    /// * `count` - The maximum number of lines to read from the input.
+    ///
+    /// # Returns
+    ///
+    /// A new instance configured to read at most `count` lines.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::HeadLinesFilter;
+    /// let filter = HeadLinesFilter::new(3);
+    /// ```
+    pub fn new(count: usize) -> Self {
+        Self { remaining: count }
+    }
+}
+
+/// Filters input by reading only the first N lines and writing them to the output.
+///
+/// Uses buffered line reading to process input line-by-line until the limit or EOF.
+///
+/// # Arguments
+///
+/// * `reader` - Mutable reference to the input data stream.
+/// * `writer` - Mutable reference to the output stream.
+///
+/// # Returns
+///
+/// * `Result<()>` - Success if filtering completes, or I/O error.
+///
+/// # Errors
+///
+/// * `io::Error` from line reading or writing operations.
+///
+/// # Examples
+///
+/// ```
+/// # use std::io::{Read, Write, Cursor};
+/// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
+/// # let mut filter = HeadLinesFilter::new(2);
+/// let mut input: &mut dyn Read = &mut Cursor::new(b"Line1\nLine2\nLine3\n");
+/// let mut output = Vec::new();
+/// filter.filter(&mut input, &mut output)?;
+/// assert_eq!(output, b"Line1\nLine2\n");
+/// # Ok::<(), std::io::Error>(())
+/// ```
+impl FilterPlugin for HeadLinesFilter {
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if self.remaining == 0 {
+            return Ok(());
+        }
+
+        let mut buf_reader = std::io::BufReader::new(reader);
+        for line in buf_reader.by_ref().lines() {
+            let line = line?;
+            writeln!(writer, "{line}")?;
+            self.remaining -= 1;
+            if self.remaining == 0 {
+                break;
+            }
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            remaining: self.remaining,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::count_option()
+    }
+
+    fn description(&self) -> &str {
+        "Read the first N lines"
+    }
+}
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_head_filters() {
+    register_filter_plugin("head_bytes", || Box::new(HeadBytesFilter::new(0)))
+        .expect("Failed to register head_bytes filter");
+    register_filter_plugin("head_lines", || Box::new(HeadLinesFilter::new(0)))
+        .expect("Failed to register head_lines filter");
+}

src/filter_plugin/mod.rs

@@ -0,0 +1,827 @@
+use std::io::{Read, Result, Write};
+use std::str::FromStr;
+use strum::EnumString;
+
+#[cfg(feature = "filter_grep")]
+pub mod grep;
+/// Filter plugin module for processing input streams.
+///
+/// This module defines the `FilterPlugin` trait and `FilterChain` for chaining filters,
+/// along with parsing utilities for filter strings. Filters can process data like head/tail,
+/// grep, etc.
+///
+/// # Usage
+///
+/// Parse a filter string and apply to a reader:
+///
+/// ```
+/// # use std::io::{Read, Write};
+/// # use keep::filter_plugin::parse_filter_string;
+/// let mut chain = parse_filter_string("head_lines(10)|tail_lines(5)")?;
+/// # let mut reader: &mut dyn Read = &mut std::io::empty();
+/// # let mut writer: Vec<u8> = Vec::new();
+/// # chain.filter(&mut reader, &mut writer)?;
+/// # Ok::<(), std::io::Error>(())
+/// ```
+pub mod head;
+pub mod skip;
+pub mod strip_ansi;
+pub mod tail;
+#[cfg(feature = "meta_tokens")]
+pub mod tokens;
+pub mod utils;
+
+use std::collections::HashMap;
+
+#[cfg(feature = "filter_grep")]
+pub use grep::GrepFilter;
+pub use head::{HeadBytesFilter, HeadLinesFilter};
+pub use skip::{SkipBytesFilter, SkipLinesFilter};
+pub use strip_ansi::StripAnsiFilter;
+pub use tail::{TailBytesFilter, TailLinesFilter};
+
+/// Represents an option for a filter plugin.
+///
+/// Defines a configurable parameter for filters, with name, default, and required flag.
+///
+/// # Fields
+///
+/// * `name` - Option name.
+/// * `default` - Optional default value.
+/// * `required` - If true, must be provided.
+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
+#[cfg_attr(feature = "server", derive(utoipa::ToSchema))]
+pub struct FilterOption {
+    pub name: String,
+    #[cfg_attr(feature = "server", schema(value_type = Option<Object>))]
+    pub default: Option<serde_json::Value>,
+    pub required: bool,
+}
+
+/// Trait for filter plugins that process input streams.
+///
+/// Implement this trait to create a filter that reads from an input stream and writes filtered output.
+///
+/// # Required Methods
+///
+/// * `filter` - Process the stream.
+/// * `clone_box` - For cloning dynamic instances.
+/// * `options` - Describe configurable options.
+///
+/// # Examples
+///
+/// ```
+/// # use std::io::{Read, Write, Result};
+/// # use keep::filter_plugin::{FilterPlugin, FilterOption};
+/// struct MyFilter;
+/// impl FilterPlugin for MyFilter {
+///     fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+///         // Implementation
+///         Ok(())
+///     }
+///     fn clone_box(&self) -> Box<dyn FilterPlugin> {
+///         Box::new(MyFilter)
+///     }
+///     fn options(&self) -> Vec<FilterOption> {
+///         vec![]
+///     }
+/// }
+/// ```
+pub trait FilterPlugin: Send {
+    /// Processes the input stream and writes the filtered output.
+    ///
+    /// This method reads from the input reader and applies filtering logic,
+    /// writing the processed data to the output writer.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - A mutable reference to the input reader providing the data to filter.
+    /// * `writer` - A mutable reference to the output writer where the processed data is written.
+    ///
+    /// # Returns
+    ///
+    /// A `Result` indicating success (`Ok(())`) or failure with an `io::Error`.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use std::io::{Read, Write, Result};
+    /// # use keep::filter_plugin::{FilterPlugin, FilterOption};
+    /// struct MyFilter;
+    /// impl FilterPlugin for MyFilter {
+    ///     fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+    ///         let mut buf = [0; 1024];
+    ///         loop {
+    ///             let n = reader.read(&mut buf)?;
+    ///             if n == 0 { break; }
+    ///             writer.write_all(&buf[0..n])?;
+    ///         }
+    ///         Ok(())
+    ///     }
+    ///     fn clone_box(&self) -> Box<dyn FilterPlugin> {
+    ///         Box::new(Self)
+    ///     }
+    ///     fn options(&self) -> Vec<FilterOption> {
+    ///         vec![]
+    ///     }
+    /// }
+    /// ```
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        let _ = std::io::copy(reader, writer)?;
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin>;
+
+    /// Returns the configuration options for this plugin.
+    ///
+    /// Describes the configurable parameters, including names, defaults, and required flags.
+    ///
+    /// # Returns
+    ///
+    /// A vector of `FilterOption` structs describing the plugin's options.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::FilterOption;
+    /// fn example_options() -> Vec<FilterOption> {
+    ///     vec![
+    ///         FilterOption {
+    ///             name: "pattern".to_string(),
+    ///             default: None,
+    ///             required: true,
+    ///         },
+    ///     ]
+    /// }
+    /// ```
+    fn options(&self) -> Vec<FilterOption>;
+
+    /// Returns a human-readable description of this filter.
+    ///
+    /// # Returns
+    ///
+    /// A description string (empty by default).
+    fn description(&self) -> &str {
+        ""
+    }
+}
+
+pub fn count_option() -> Vec<FilterOption> {
+    vec![FilterOption {
+        name: "count".to_string(),
+        default: None,
+        required: true,
+    }]
+}
+
+pub fn pattern_option() -> Vec<FilterOption> {
+    vec![FilterOption {
+        name: "pattern".to_string(),
+        default: None,
+        required: true,
+    }]
+}
+
+/// Enum representing the different types of filters.
+///
+/// Used for parsing and instantiating specific filter plugins.
+///
+/// # Variants
+///
+/// * `HeadBytes` - Head by bytes.
+/// * `HeadLines` - Head by lines.
+/// * ... etc.
+#[derive(Debug, EnumString, strum::VariantNames, strum::Display)]
+#[strum(serialize_all = "snake_case")]
+pub enum FilterType {
+    HeadBytes,
+    HeadLines,
+    TailBytes,
+    TailLines,
+    SkipBytes,
+    SkipLines,
+    #[cfg(feature = "filter_grep")]
+    Grep,
+    StripAnsi,
+    #[cfg(feature = "meta_tokens")]
+    HeadTokens,
+    #[cfg(feature = "meta_tokens")]
+    SkipTokens,
+    #[cfg(feature = "meta_tokens")]
+    TailTokens,
+}
+
+/// Maximum buffer size (256 MB) for filter chain intermediate results.
+/// Prevents OOM on large files by rejecting inputs that exceed this limit.
+const MAX_FILTER_BUFFER_SIZE: usize = 256 * 1024 * 1024;
+
+struct BoundedVecWriter {
+    data: Vec<u8>,
+    limit: usize,
+}
+
+impl BoundedVecWriter {
+    fn new(limit: usize) -> Self {
+        Self {
+            data: Vec::new(),
+            limit,
+        }
+    }
+
+    fn into_inner(self) -> Vec<u8> {
+        self.data
+    }
+}
+
+impl std::io::Write for BoundedVecWriter {
+    fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
+        if self.data.len() + buf.len() > self.limit {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidData,
+                format!(
+                    "Input size exceeds maximum filter buffer size ({} bytes)",
+                    MAX_FILTER_BUFFER_SIZE
+                ),
+            ));
+        }
+        self.data.write_all(buf)?;
+        Ok(buf.len())
+    }
+
+    fn flush(&mut self) -> std::io::Result<()> {
+        Ok(())
+    }
+}
+
+/// A chain of filter plugins applied sequentially.
+///
+/// Chains multiple filters, applying them in order to the input stream.
+///
+/// # Fields
+///
+/// * `plugins` - Vector of boxed filter plugins.
+pub struct FilterChain {
+    plugins: Vec<Box<dyn FilterPlugin>>,
+}
+
+/// A chain of filter plugins applied sequentially.
+///
+/// Chains multiple filters, applying them in order to the input stream.
+///
+/// # Fields
+///
+/// * `plugins` - Vector of boxed filter plugins.
+///
+/// # Examples
+///
+/// ```
+/// # use std::io::{Read, Write, Result};
+/// # use keep::filter_plugin::{FilterChain, HeadLinesFilter};
+/// let mut chain = FilterChain::new();
+/// chain.add_plugin(Box::new(HeadLinesFilter::new(10)));
+/// # let mut reader: &mut dyn Read = &mut std::io::empty();
+/// # let mut writer: Vec<u8> = Vec::new();
+/// # chain.filter(&mut reader, &mut writer)?;
+/// # Ok::<(), std::io::Error>(())
+/// ```
+impl Clone for FilterChain {
+    /// Clones this filter chain.
+    ///
+    /// # Returns
+    ///
+    /// A new `FilterChain` with cloned plugins.
+    fn clone(&self) -> Self {
+        let mut plugins = Vec::with_capacity(self.plugins.len());
+        for plugin in &self.plugins {
+            plugins.push(plugin.clone_box());
+        }
+        FilterChain { plugins }
+    }
+}
+
+impl Clone for Box<dyn FilterPlugin> {
+    fn clone(&self) -> Self {
+        self.clone_box()
+    }
+}
+
+#[macro_export]
+macro_rules! filter_clone_box {
+    ($self:expr) => {
+        Box::new($self.clone())
+    };
+    ($self:expr, $field:ident) => {
+        Box::new(Self { $field: $self.$field.clone() })
+    };
+    ($self:expr, $field:ident, $($rest:ident),+) => {
+        Box::new(Self {
+            $field: $self.$field.clone(),
+            $($rest: $self.$rest.clone()),+
+        })
+    };
+}
+
+impl Default for FilterChain {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl FilterChain {
+    /// Creates a new empty filter chain.
+    ///
+    /// # Returns
+    ///
+    /// A new `FilterChain` with no plugins.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::FilterChain;
+    /// let chain = FilterChain::new();
+    /// // Chain starts empty
+    /// ```
+    pub fn new() -> Self {
+        Self {
+            plugins: Vec::new(),
+        }
+    }
+
+    /// Adds a plugin to the chain.
+    ///
+    /// Plugins are applied in the order they are added.
+    ///
+    /// # Arguments
+    ///
+    /// * `plugin` - The boxed filter plugin to add to the chain.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::filter_plugin::FilterChain;
+    /// let mut chain = FilterChain::new();
+    /// ```
+    pub fn add_plugin(&mut self, plugin: Box<dyn FilterPlugin>) {
+        self.plugins.push(plugin);
+    }
+
+    /// Applies the filter chain to the input and writes to the output.
+    ///
+    /// If no plugins are present, data is copied directly from reader to writer.
+    /// For multiple plugins, intermediate results are buffered.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - A mutable reference to the input reader providing the data stream.
+    /// * `writer` - A mutable reference to the output writer where the fully filtered data is sent.
+    ///
+    /// # Returns
+    ///
+    /// A `Result` indicating success (`Ok(())`) or failure with an `io::Error` if any filter in the chain fails.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use std::io::{Read, Write, Result};
+    /// # use keep::filter_plugin::{FilterChain, HeadBytesFilter};
+    /// let mut chain = FilterChain::new();
+    /// chain.add_plugin(Box::new(HeadBytesFilter::new(100)));
+    /// # let mut input_reader: &mut dyn Read = &mut std::io::empty();
+    /// # let mut output_writer: Vec<u8> = Vec::new();
+    /// # chain.filter(&mut input_reader, &mut output_writer)?;
+    /// # Ok::<(), std::io::Error>(())
+    /// ```
+    pub fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if self.plugins.is_empty() {
+            // If no plugins, just copy the input to output
+            std::io::copy(reader, writer)?;
+            return Ok(());
+        }
+
+        // For multiple plugins, we need to chain them together
+        // We'll use a bounded buffer to hold intermediate results
+        let mut bounded_writer = BoundedVecWriter::new(MAX_FILTER_BUFFER_SIZE);
+        std::io::copy(reader, &mut bounded_writer)?;
+        let mut current_data = bounded_writer.into_inner();
+
+        // Store the plugins length to avoid borrowing issues
+        let plugins_len = self.plugins.len();
+
+        for i in 0..plugins_len {
+            // Create a cursor for the current data
+            let mut input = std::io::Cursor::new(std::mem::take(&mut current_data));
+
+            // For the last plugin, write directly to the output writer
+            if i == plugins_len - 1 {
+                self.plugins[i].filter(&mut input, writer)?;
+            } else {
+                // For intermediate plugins, write to a buffer
+                let mut output_vec = Vec::new();
+                self.plugins[i].filter(&mut input, &mut output_vec)?;
+
+                if output_vec.len() > MAX_FILTER_BUFFER_SIZE {
+                    return Err(std::io::Error::new(
+                        std::io::ErrorKind::InvalidData,
+                        format!(
+                            "Filter output size ({} bytes) exceeds maximum filter buffer size ({} bytes).",
+                            output_vec.len(),
+                            MAX_FILTER_BUFFER_SIZE
+                        ),
+                    ));
+                }
+
+                current_data = output_vec;
+            }
+        }
+        Ok(())
+    }
+}
+
+/// Parses a filter string into a `FilterChain`.
+///
+/// # Arguments
+///
+/// * `filter_str` - The filter string specifying the chain, e.g., "head_lines(10)|grep(pattern=error)".
+///
+/// # Returns
+///
+/// A `Result` containing the parsed `FilterChain` on success, or an `io::Error` if the string is invalid.
+pub fn parse_filter_string(filter_str: &str) -> Result<FilterChain> {
+    let mut chain = FilterChain::new();
+
+    for part in filter_str.split('|') {
+        let part = part.trim();
+        if part.is_empty() {
+            continue;
+        }
+
+        // Parse the filter name and parameters
+        if let Some((filter_name, params)) = part.split_once('(') {
+            if let Some(params) = params.strip_suffix(')') {
+                // Parse parameters
+                let mut options = HashMap::new();
+                let mut unnamed_params = Vec::new();
+
+                // Split parameters by commas
+                for param in params.split(',') {
+                    let param = param.trim();
+                    if param.is_empty() {
+                        continue;
+                    }
+
+                    // Check if it's a named parameter (key=value)
+                    if let Some((key, value)) = param.split_once('=') {
+                        let key = key.trim();
+                        let value = parse_option_value(value.trim())?;
+                        options.insert(key.to_string(), value);
+                    } else {
+                        // Unnamed parameter
+                        let value = parse_option_value(param)?;
+                        unnamed_params.push(value);
+                    }
+                }
+
+                // Create the appropriate filter plugin
+                if let Ok(filter_type) = FilterType::from_str(filter_name) {
+                    let plugin =
+                        create_filter_with_options(filter_type, &unnamed_params, &options)?;
+                    chain.add_plugin(plugin);
+                    continue;
+                }
+            }
+        } else {
+            // Handle filters without parameters
+            if let Ok(filter_type) = FilterType::from_str(part) {
+                match filter_type {
+                    FilterType::StripAnsi => {
+                        chain.add_plugin(Box::new(strip_ansi::StripAnsiFilter::new()));
+                        continue;
+                    }
+                    _ => {
+                        return Err(std::io::Error::new(
+                            std::io::ErrorKind::InvalidInput,
+                            format!("Filter '{part}' requires parameters"),
+                        ));
+                    }
+                }
+            }
+        }
+
+        // If we get here, the filter wasn't recognized
+        return Err(std::io::Error::new(
+            std::io::ErrorKind::InvalidInput,
+            format!("Unknown filter: {part}"),
+        ));
+    }
+
+    Ok(chain)
+}
+
+/// Creates a filter plugin with the given options.
+///
+/// # Arguments
+///
+/// * `filter_type` - The enum variant indicating the type of filter to instantiate.
+/// * `unnamed_params` - A slice of unnamed JSON parameters passed to the filter.
+/// * `named_options` - A hashmap of named options as key-value pairs.
+///
+/// # Returns
+///
+/// A `Result` containing a boxed `FilterPlugin` on success, or an `io::Error` if creation fails.
+fn create_filter_with_options(
+    filter_type: FilterType,
+    unnamed_params: &[serde_json::Value],
+    named_options: &HashMap<String, serde_json::Value>,
+) -> Result<Box<dyn FilterPlugin>> {
+    // Get the default options for this filter type by creating a temporary instance
+    // To do this, we need to create a default instance of the appropriate filter
+    let option_defs = match filter_type {
+        #[cfg(feature = "filter_grep")]
+        FilterType::Grep => grep::GrepFilter::new("".to_string())?.options(),
+        FilterType::HeadBytes => head::HeadBytesFilter::new(0).options(),
+        FilterType::HeadLines => head::HeadLinesFilter::new(0).options(),
+        FilterType::TailBytes => tail::TailBytesFilter::new(0).options(),
+        FilterType::TailLines => tail::TailLinesFilter::new(0).options(),
+        FilterType::SkipBytes => skip::SkipBytesFilter::new(0).options(),
+        FilterType::SkipLines => skip::SkipLinesFilter::new(0).options(),
+        FilterType::StripAnsi => strip_ansi::StripAnsiFilter::new().options(),
+        #[cfg(feature = "meta_tokens")]
+        FilterType::HeadTokens => tokens::HeadTokensFilter::new(0).options(),
+        #[cfg(feature = "meta_tokens")]
+        FilterType::SkipTokens => tokens::SkipTokensFilter::new(0).options(),
+        #[cfg(feature = "meta_tokens")]
+        FilterType::TailTokens => tokens::TailTokensFilter::new(0).options(),
+    };
+
+    let mut options = HashMap::new();
+
+    // Process unnamed parameters
+    if unnamed_params.len() > option_defs.len() {
+        return Err(std::io::Error::new(
+            std::io::ErrorKind::InvalidInput,
+            format!(
+                "Too many unnamed parameters (expected at most {})",
+                option_defs.len()
+            ),
+        ));
+    }
+
+    for (i, param) in unnamed_params.iter().enumerate() {
+        if i >= option_defs.len() {
+            break;
+        }
+        let option_name = &option_defs[i].name;
+        options.insert(option_name.clone(), param.clone());
+    }
+
+    // Process named options
+    for (key, value) in named_options {
+        // Check if the option exists
+        if !option_defs.iter().any(|opt| &opt.name == key) {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                format!("Unknown option '{key}'"),
+            ));
+        }
+        options.insert(key.clone(), value.clone());
+    }
+
+    // Fill in defaults and check required options
+    for opt_def in option_defs {
+        if !options.contains_key(&opt_def.name) {
+            if let Some(default) = &opt_def.default {
+                options.insert(opt_def.name.clone(), default.clone());
+            } else if opt_def.required {
+                return Err(std::io::Error::new(
+                    std::io::ErrorKind::InvalidInput,
+                    format!("Missing required option '{}'", opt_def.name),
+                ));
+            }
+        }
+    }
+
+    // Create the specific filter with the processed options
+    create_specific_filter(filter_type, &options)
+}
+
+/// Creates a specific filter instance based on type and options.
+///
+/// # Arguments
+///
+/// * `filter_type` - The enum variant indicating the type of filter to instantiate.
+/// * `options` - A reference to the hashmap of processed options for the filter.
+///
+/// # Returns
+///
+/// A `Result` containing a boxed `FilterPlugin` on success, or an `io::Error` if instantiation fails.
+fn create_specific_filter(
+    filter_type: FilterType,
+    options: &HashMap<String, serde_json::Value>,
+) -> Result<Box<dyn FilterPlugin>> {
+    match filter_type {
+        #[cfg(feature = "filter_grep")]
+        FilterType::Grep => {
+            let pattern = options
+                .get("pattern")
+                .and_then(|v| v.as_str())
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "grep filter requires 'pattern' parameter",
+                    )
+                })?;
+            grep::GrepFilter::new(pattern.to_string()).map(|f| Box::new(f) as Box<dyn FilterPlugin>)
+        }
+        FilterType::HeadBytes => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "head_bytes filter requires 'count' parameter",
+                    )
+                })?;
+            Ok(Box::new(head::HeadBytesFilter::new(count)))
+        }
+        FilterType::HeadLines => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "head_lines filter requires 'count' parameter",
+                    )
+                })?;
+            Ok(Box::new(head::HeadLinesFilter::new(count)))
+        }
+        FilterType::TailBytes => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "tail_bytes filter requires 'count' parameter",
+                    )
+                })?;
+            Ok(Box::new(tail::TailBytesFilter::new(count)))
+        }
+        FilterType::TailLines => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "tail_lines filter requires 'count' parameter",
+                    )
+                })?;
+            Ok(Box::new(tail::TailLinesFilter::new(count)))
+        }
+        FilterType::SkipBytes => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "skip_bytes filter requires 'count' parameter",
+                    )
+                })?;
+            Ok(Box::new(skip::SkipBytesFilter::new(count)))
+        }
+        FilterType::SkipLines => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "skip_lines filter requires 'count' parameter",
+                    )
+                })?;
+            Ok(Box::new(skip::SkipLinesFilter::new(count)))
+        }
+        FilterType::StripAnsi => {
+            // StripAnsi doesn't take any parameters
+            if !options.is_empty() {
+                return Err(std::io::Error::new(
+                    std::io::ErrorKind::InvalidInput,
+                    "strip_ansi filter doesn't take parameters",
+                ));
+            }
+            Ok(Box::new(strip_ansi::StripAnsiFilter::new()))
+        }
+        #[cfg(feature = "meta_tokens")]
+        FilterType::HeadTokens => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "head_tokens filter requires 'count' parameter",
+                    )
+                })?;
+            let (encoding, tokenizer) = parse_encoding_option(options);
+            let mut f = tokens::HeadTokensFilter::new(count);
+            f.tokenizer = tokenizer;
+            f.encoding = encoding;
+            Ok(Box::new(f))
+        }
+        #[cfg(feature = "meta_tokens")]
+        FilterType::SkipTokens => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "skip_tokens filter requires 'count' parameter",
+                    )
+                })?;
+            let (encoding, tokenizer) = parse_encoding_option(options);
+            let mut f = tokens::SkipTokensFilter::new(count);
+            f.tokenizer = tokenizer;
+            f.encoding = encoding;
+            Ok(Box::new(f))
+        }
+        #[cfg(feature = "meta_tokens")]
+        FilterType::TailTokens => {
+            let count = options
+                .get("count")
+                .and_then(|v| v.as_u64())
+                .map(|n| n as usize)
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidInput,
+                        "tail_tokens filter requires 'count' parameter",
+                    )
+                })?;
+            let (encoding, tokenizer) = parse_encoding_option(options);
+            let mut f = tokens::TailTokensFilter::new(count);
+            f.tokenizer = tokenizer;
+            f.encoding = encoding;
+            Ok(Box::new(f))
+        }
+    }
+}
+
+#[cfg(feature = "meta_tokens")]
+fn parse_encoding_option(
+    options: &std::collections::HashMap<String, serde_json::Value>,
+) -> (crate::tokenizer::TokenEncoding, crate::tokenizer::Tokenizer) {
+    let encoding = options
+        .get("encoding")
+        .and_then(|v| v.as_str())
+        .and_then(|s| s.parse::<crate::tokenizer::TokenEncoding>().ok())
+        .unwrap_or_default();
+    let tokenizer = crate::tokenizer::get_tokenizer(encoding).clone();
+    (encoding, tokenizer)
+}
+
+/// Parses an option value from a string into a JSON value.
+///
+/// # Arguments
+///
+/// * `input` - The input string.
+///
+/// # Returns
+///
+/// A `Result` containing the parsed JSON value.
+fn parse_option_value(input: &str) -> Result<serde_json::Value> {
+    // Remove quotes if present
+    let input = input.trim_matches(|c| c == '\'' || c == '"');
+
+    // Try to parse as number
+    if let Ok(num) = input.parse::<i64>() {
+        return Ok(serde_json::Value::Number(num.into()));
+    }
+    if let Ok(num) = input.parse::<f64>()
+        && let Some(number) = serde_json::Number::from_f64(num)
+    {
+        return Ok(serde_json::Value::Number(number));
+    }
+
+    // Try to parse as boolean
+    if input.eq_ignore_ascii_case("true") {
+        return Ok(serde_json::Value::Bool(true));
+    }
+    if input.eq_ignore_ascii_case("false") {
+        return Ok(serde_json::Value::Bool(false));
+    }
+
+    // Treat as string
+    Ok(serde_json::Value::String(input.to_string()))
+}

src/filter_plugin/skip.rs

@@ -0,0 +1,133 @@
+use super::{FilterOption, FilterPlugin};
+use crate::common::PIPESIZE;
+use crate::services::filter_service::register_filter_plugin;
+use std::io::{BufRead, Read, Result, Write};
+
+/// A filter that skips the first N bytes from the input stream.
+#[derive(Clone)]
+pub struct SkipBytesFilter {
+    remaining: usize,
+}
+
+impl SkipBytesFilter {
+    /// Creates a new `SkipBytesFilter` that will skip the specified number of bytes.
+    ///
+    /// # Arguments
+    ///
+    /// * `count` - The number of bytes to skip from the beginning of the input.
+    pub fn new(count: usize) -> Self {
+        Self { remaining: count }
+    }
+}
+
+impl FilterPlugin for SkipBytesFilter {
+    /// Filters the input by skipping the first N bytes and writing the rest to the output.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - Mutable reference to the input reader providing the data stream.
+    /// * `writer` - Mutable reference to the output writer where filtered data is sent.
+    ///
+    /// # Returns
+    ///
+    /// Returns `Ok(())` on success, or an `io::Error` if reading or writing fails.
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        // Skip bytes in chunks
+        if self.remaining > 0 {
+            let mut buffer = vec![0; PIPESIZE];
+            while self.remaining > 0 {
+                let to_read = std::cmp::min(self.remaining, PIPESIZE);
+                let bytes_read = reader.read(&mut buffer[..to_read])?;
+                if bytes_read == 0 {
+                    break;
+                }
+                self.remaining -= bytes_read;
+            }
+        }
+
+        // Copy the remaining data using io::copy for efficiency
+        std::io::copy(reader, writer)?;
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            remaining: self.remaining,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::count_option()
+    }
+
+    fn description(&self) -> &str {
+        "Skip the first N bytes"
+    }
+}
+
+/// A filter that skips the first N lines from the input stream.
+#[derive(Clone)]
+pub struct SkipLinesFilter {
+    remaining: usize,
+}
+
+impl SkipLinesFilter {
+    /// Creates a new `SkipLinesFilter` that will skip the specified number of lines.
+    ///
+    /// # Arguments
+    ///
+    /// * `count` - The number of lines to skip from the beginning of the input.
+    pub fn new(count: usize) -> Self {
+        Self { remaining: count }
+    }
+}
+
+impl FilterPlugin for SkipLinesFilter {
+    /// Filters the input by skipping the first N lines and writing the rest to the output.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - Mutable reference to the input reader providing the data stream.
+    /// * `writer` - Mutable reference to the output writer where filtered data is sent.
+    ///
+    /// # Returns
+    ///
+    /// Returns `Ok(())` on success, or an `io::Error` if reading or writing fails.
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        let mut buf_reader = std::io::BufReader::new(reader);
+        for line in buf_reader.by_ref().lines() {
+            let line = line?;
+            if self.remaining > 0 {
+                self.remaining -= 1;
+            } else {
+                writeln!(writer, "{line}")?;
+            }
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            remaining: self.remaining,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::count_option()
+    }
+
+    fn description(&self) -> &str {
+        "Skip the first N lines"
+    }
+}
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_skip_filters() {
+    register_filter_plugin("skip_bytes", || Box::new(SkipBytesFilter::new(0)))
+        .expect("Failed to register skip_bytes filter");
+    register_filter_plugin("skip_lines", || Box::new(SkipLinesFilter::new(0)))
+        .expect("Failed to register skip_lines filter");
+}

src/filter_plugin/strip_ansi.rs

@@ -0,0 +1,53 @@
+use super::{FilterOption, FilterPlugin};
+use std::io::{Read, Result, Write};
+use strip_ansi_escapes::Writer;
+
+/// A filter that removes ANSI escape sequences from the input.
+///
+/// # Fields
+///
+/// None, stateless filter.
+#[derive(Default, Clone)]
+pub struct StripAnsiFilter;
+
+impl StripAnsiFilter {
+    /// Creates a new `StripAnsiFilter`.
+    ///
+    /// # Returns
+    ///
+    /// A new instance of `StripAnsiFilter`.
+    pub fn new() -> Self {
+        Self
+    }
+}
+
+impl FilterPlugin for StripAnsiFilter {
+    /// Filters the input by stripping ANSI escape sequences and writing the plain text to the output.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - Mutable reference to the input reader providing the data stream with potential ANSI codes.
+    /// * `writer` - Mutable reference to the output writer where plain text is sent.
+    ///
+    /// # Returns
+    ///
+    /// Returns `Ok(())` on success, or an `io::Error` if reading or writing fails.
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        let mut ansi_writer = Writer::new(writer);
+        std::io::copy(reader, &mut ansi_writer)?;
+        ansi_writer.flush()?;
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self)
+    }
+
+    fn options(&self) -> Vec<FilterOption> {
+        Vec::new()
+    }
+
+    fn description(&self) -> &str {
+        "Strip ANSI escape sequences"
+    }
+}

src/filter_plugin/tail.rs

@@ -0,0 +1,151 @@
+use super::{FilterOption, FilterPlugin};
+use crate::common::PIPESIZE;
+use crate::services::filter_service::register_filter_plugin;
+use std::collections::VecDeque;
+use std::io::{BufRead, Read, Result, Write};
+
+#[derive(Clone)]
+pub struct TailBytesFilter {
+    buffer: VecDeque<u8>,
+    count: usize,
+}
+
+impl TailBytesFilter {
+    /// Creates a new `TailBytesFilter` that will keep the last specified number of bytes.
+    ///
+    /// # Arguments
+    ///
+    /// * `count` - The number of bytes to retain from the end of the input.
+    pub fn new(count: usize) -> Self {
+        Self {
+            buffer: VecDeque::with_capacity(count),
+            count,
+        }
+    }
+}
+
+impl FilterPlugin for TailBytesFilter {
+    /// Filters the input by keeping only the last N bytes and writing them to the output.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - Mutable reference to the input reader providing the data stream.
+    /// * `writer` - Mutable reference to the output writer where filtered data is sent.
+    ///
+    /// # Returns
+    ///
+    /// Returns `Ok(())` on success, or an `io::Error` if reading or writing fails.
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        let mut temp_buffer = vec![0; PIPESIZE];
+        loop {
+            let bytes_read = reader.read(&mut temp_buffer)?;
+            if bytes_read == 0 {
+                break;
+            }
+
+            // Add new data to the buffer
+            for &byte in &temp_buffer[..bytes_read] {
+                if self.buffer.len() == self.count {
+                    self.buffer.pop_front();
+                }
+                self.buffer.push_back(byte);
+            }
+        }
+
+        // Write the buffered data at the end
+        let result: Vec<u8> = self.buffer.iter().cloned().collect();
+        writer.write_all(&result)?;
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            buffer: self.buffer.clone(),
+            count: self.count,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::count_option()
+    }
+
+    fn description(&self) -> &str {
+        "Read the last N bytes"
+    }
+}
+
+/// A filter that reads the last N lines from the input stream.
+#[derive(Clone)]
+pub struct TailLinesFilter {
+    lines: VecDeque<String>,
+    count: usize,
+}
+
+impl TailLinesFilter {
+    /// Creates a new `TailLinesFilter` that will keep the last specified number of lines.
+    ///
+    /// # Arguments
+    ///
+    /// * `count` - The number of lines to retain from the end of the input.
+    pub fn new(count: usize) -> Self {
+        Self {
+            lines: VecDeque::with_capacity(count),
+            count,
+        }
+    }
+}
+
+impl FilterPlugin for TailLinesFilter {
+    /// Filters the input by keeping only the last N lines and writing them to the output.
+    ///
+    /// # Arguments
+    ///
+    /// * `reader` - Mutable reference to the input reader providing the data stream.
+    /// * `writer` - Mutable reference to the output writer where filtered data is sent.
+    ///
+    /// # Returns
+    ///
+    /// Returns `Ok(())` on success, or an `io::Error` if reading or writing fails.
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        let mut buf_reader = std::io::BufReader::new(reader);
+        for line in buf_reader.by_ref().lines() {
+            let line = line?;
+            if self.lines.len() == self.count {
+                self.lines.pop_front();
+            }
+            self.lines.push_back(line);
+        }
+
+        // Write the buffered lines
+        for line in &self.lines {
+            writeln!(writer, "{line}")?;
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            lines: self.lines.clone(),
+            count: self.count,
+        })
+    }
+
+    /// Returns the configuration options for this filter.
+    fn options(&self) -> Vec<FilterOption> {
+        crate::filter_plugin::count_option()
+    }
+
+    fn description(&self) -> &str {
+        "Read the last N lines"
+    }
+}
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_tail_filters() {
+    register_filter_plugin("tail_bytes", || Box::new(TailBytesFilter::new(0)))
+        .expect("Failed to register tail_bytes filter");
+    register_filter_plugin("tail_lines", || Box::new(TailLinesFilter::new(0)))
+        .expect("Failed to register tail_lines filter");
+}

src/filter_plugin/tokens.rs

@@ -0,0 +1,500 @@
+use super::{FilterOption, FilterPlugin};
+use crate::common::PIPESIZE;
+use crate::services::filter_service::register_filter_plugin;
+use crate::tokenizer::{TokenEncoding, Tokenizer, get_tokenizer};
+use std::io::{Read, Result, Write};
+
+// ---------------------------------------------------------------------------
+// head_tokens
+// ---------------------------------------------------------------------------
+
+#[derive(Clone)]
+pub struct HeadTokensFilter {
+    pub remaining: usize,
+    pub tokenizer: Tokenizer,
+    pub encoding: TokenEncoding,
+}
+
+impl HeadTokensFilter {
+    pub fn new(count: usize) -> Self {
+        let encoding = TokenEncoding::default();
+        Self {
+            remaining: count,
+            tokenizer: get_tokenizer(encoding).clone(),
+            encoding,
+        }
+    }
+}
+
+impl FilterPlugin for HeadTokensFilter {
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if self.remaining == 0 {
+            return Ok(());
+        }
+
+        let tokenizer = &self.tokenizer;
+        let mut buffer = vec![0u8; PIPESIZE];
+        let mut total_tokens = 0usize;
+
+        loop {
+            let n = reader.read(&mut buffer)?;
+            if n == 0 {
+                break;
+            }
+
+            let chunk = &buffer[..n];
+            let text = String::from_utf8_lossy(chunk);
+            let chunk_tokens = tokenizer.count(&text);
+
+            if total_tokens + chunk_tokens <= self.remaining {
+                // Entire chunk fits — write it directly
+                writer.write_all(chunk)?;
+                total_tokens += chunk_tokens;
+                if total_tokens >= self.remaining {
+                    break;
+                }
+            } else {
+                // Cutoff is within this chunk — use iterator to find exact
+                // boundary without allocating all token strings
+                let tokens_to_write = self.remaining - total_tokens;
+                let mut byte_pos = 0usize;
+                for token_str in tokenizer.split_by_token_iter(&text).take(tokens_to_write) {
+                    byte_pos += token_str
+                        .map_err(|e| std::io::Error::other(e.to_string()))?
+                        .len();
+                }
+                let write_len = map_lossy_pos_to_bytes(chunk, &text, byte_pos);
+                writer.write_all(&chunk[..write_len])?;
+                break;
+            }
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            remaining: self.remaining,
+            tokenizer: self.tokenizer.clone(),
+            encoding: self.encoding,
+        })
+    }
+
+    fn options(&self) -> Vec<FilterOption> {
+        vec![
+            FilterOption {
+                name: "count".to_string(),
+                default: None,
+                required: true,
+            },
+            FilterOption {
+                name: "encoding".to_string(),
+                default: Some(serde_json::Value::String("cl100k_base".to_string())),
+                required: false,
+            },
+        ]
+    }
+
+    fn description(&self) -> &str {
+        "Read the first N LLM tokens"
+    }
+}
+
+// ---------------------------------------------------------------------------
+// skip_tokens
+// ---------------------------------------------------------------------------
+
+#[derive(Clone)]
+pub struct SkipTokensFilter {
+    pub remaining: usize,
+    pub tokenizer: Tokenizer,
+    pub encoding: TokenEncoding,
+}
+
+impl SkipTokensFilter {
+    pub fn new(count: usize) -> Self {
+        let encoding = TokenEncoding::default();
+        Self {
+            remaining: count,
+            tokenizer: get_tokenizer(encoding).clone(),
+            encoding,
+        }
+    }
+}
+
+impl FilterPlugin for SkipTokensFilter {
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if self.remaining == 0 {
+            return std::io::copy(reader, writer).map(|_| ());
+        }
+
+        let tokenizer = &self.tokenizer;
+        let mut buffer = vec![0u8; PIPESIZE];
+        let mut total_tokens = 0usize;
+        let mut done_skipping = false;
+
+        loop {
+            let n = reader.read(&mut buffer)?;
+            if n == 0 {
+                break;
+            }
+
+            if done_skipping {
+                writer.write_all(&buffer[..n])?;
+                continue;
+            }
+
+            let chunk = &buffer[..n];
+            let text = String::from_utf8_lossy(chunk);
+            let chunk_tokens = tokenizer.count(&text);
+
+            if total_tokens + chunk_tokens <= self.remaining {
+                // Entire chunk is skipped
+                total_tokens += chunk_tokens;
+                if total_tokens >= self.remaining {
+                    done_skipping = true;
+                }
+            } else {
+                // Cutoff is within this chunk — use iterator to skip past
+                // the boundary without allocating all token strings
+                let tokens_to_skip = self.remaining - total_tokens;
+                let mut byte_pos = 0usize;
+                for token_str in tokenizer.split_by_token_iter(&text).take(tokens_to_skip) {
+                    byte_pos += token_str
+                        .map_err(|e| std::io::Error::other(e.to_string()))?
+                        .len();
+                }
+                let skip_len = map_lossy_pos_to_bytes(chunk, &text, byte_pos);
+                if skip_len < n {
+                    writer.write_all(&chunk[skip_len..])?;
+                }
+                done_skipping = true;
+            }
+        }
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            remaining: self.remaining,
+            tokenizer: self.tokenizer.clone(),
+            encoding: self.encoding,
+        })
+    }
+
+    fn options(&self) -> Vec<FilterOption> {
+        vec![
+            FilterOption {
+                name: "count".to_string(),
+                default: None,
+                required: true,
+            },
+            FilterOption {
+                name: "encoding".to_string(),
+                default: Some(serde_json::Value::String("cl100k_base".to_string())),
+                required: false,
+            },
+        ]
+    }
+
+    fn description(&self) -> &str {
+        "Skip the first N LLM tokens"
+    }
+}
+
+// ---------------------------------------------------------------------------
+// tail_tokens
+// ---------------------------------------------------------------------------
+
+/// A filter that outputs only the last N tokens of the input stream.
+///
+#[derive(Clone)]
+pub struct TailTokensFilter {
+    pub count: usize,
+    /// Buffer holding all bytes from the stream.
+    buffer: Vec<u8>,
+    pub tokenizer: Tokenizer,
+    pub encoding: TokenEncoding,
+}
+
+impl TailTokensFilter {
+    pub fn new(count: usize) -> Self {
+        let encoding = TokenEncoding::default();
+        Self {
+            count,
+            buffer: Vec::with_capacity(PIPESIZE),
+            tokenizer: get_tokenizer(encoding).clone(),
+            encoding,
+        }
+    }
+}
+
+impl FilterPlugin for TailTokensFilter {
+    fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
+        if self.count == 0 {
+            return Ok(());
+        }
+
+        let tokenizer = &self.tokenizer;
+
+        // Buffer all bytes from the stream
+        std::io::copy(reader, &mut self.buffer)?;
+
+        if self.buffer.is_empty() {
+            return Ok(());
+        }
+
+        let text = String::from_utf8_lossy(&self.buffer);
+        let token_strs = tokenizer
+            .split_by_token(&text)
+            .map_err(|e| std::io::Error::other(e.to_string()))?;
+
+        if token_strs.len() <= self.count {
+            // All tokens fit — write everything
+            writer.write_all(&self.buffer)?;
+        } else {
+            // Write only the last N tokens
+            let skip = token_strs.len() - self.count;
+            let mut byte_offset = 0usize;
+            for token_str in token_strs.iter().take(skip) {
+                byte_offset += token_str.len();
+            }
+            let write_len = map_lossy_pos_to_bytes(&self.buffer, &text, byte_offset);
+            if write_len < self.buffer.len() {
+                writer.write_all(&self.buffer[write_len..])?;
+            }
+        }
+
+        Ok(())
+    }
+
+    fn clone_box(&self) -> Box<dyn FilterPlugin> {
+        Box::new(Self {
+            count: self.count,
+            buffer: Vec::new(),
+            tokenizer: self.tokenizer.clone(),
+            encoding: self.encoding,
+        })
+    }
+
+    fn options(&self) -> Vec<FilterOption> {
+        vec![
+            FilterOption {
+                name: "count".to_string(),
+                default: None,
+                required: true,
+            },
+            FilterOption {
+                name: "encoding".to_string(),
+                default: Some(serde_json::Value::String("cl100k_base".to_string())),
+                required: false,
+            },
+        ]
+    }
+
+    fn description(&self) -> &str {
+        "Read the last N LLM tokens"
+    }
+}
+
+// ---------------------------------------------------------------------------
+// Helpers
+// ---------------------------------------------------------------------------
+
+/// Map a byte position in a lossy string back to a position in the original byte slice.
+///
+/// `String::from_utf8_lossy` replaces invalid UTF-8 bytes with the Unicode
+/// replacement character (U+FFFD), which encodes to 3 bytes in UTF-8. This
+/// function walks both the original bytes and the lossy string in lockstep,
+/// finding the original byte position that corresponds to `lossy_pos`.
+fn map_lossy_pos_to_bytes(original: &[u8], lossy: &str, lossy_pos: usize) -> usize {
+    if lossy_pos == 0 {
+        return 0;
+    }
+
+    let replacement = '\u{FFFD}';
+    let replacement_len = replacement.len_utf8(); // 3 bytes
+
+    let mut orig_idx = 0usize;
+    let mut lossy_idx = 0usize;
+    let lossy_bytes = lossy.as_bytes();
+
+    while lossy_idx < lossy_pos && orig_idx < original.len() {
+        // Try to decode the next character from the original bytes
+        match std::str::from_utf8(&original[orig_idx..]) {
+            Ok("") => break,
+            Ok(s) => {
+                let ch = s.chars().next().unwrap();
+                let ch_len = ch.len_utf8();
+                // Check if this is a replacement character in the lossy string
+                if ch == replacement
+                    && lossy_idx + replacement_len <= lossy_pos
+                    && lossy_bytes[lossy_idx..].starts_with(
+                        &replacement.encode_utf8(&mut [0; 4]).as_bytes()[..replacement_len],
+                    )
+                {
+                    // Could be a real U+FFFD or a replacement of invalid bytes.
+                    // If the original byte at this position is valid UTF-8 start, it's real.
+                    if original[orig_idx] < 0x80 || original[orig_idx] >= 0xC0 {
+                        // Real character
+                        orig_idx += ch_len;
+                        lossy_idx += ch_len;
+                    } else {
+                        // Invalid byte that was replaced — advance original by 1
+                        orig_idx += 1;
+                        lossy_idx += replacement_len;
+                    }
+                } else {
+                    orig_idx += ch_len;
+                    lossy_idx += ch_len;
+                }
+            }
+            Err(e) => {
+                let valid = e.valid_up_to();
+                if valid > 0 {
+                    // Some valid bytes, then invalid
+                    orig_idx += valid;
+                    lossy_idx += valid;
+                } else {
+                    // Invalid byte — in lossy it becomes 3-byte replacement char
+                    orig_idx += 1;
+                    lossy_idx += replacement_len;
+                }
+            }
+        }
+    }
+
+    orig_idx.min(original.len())
+}
+
+// ---------------------------------------------------------------------------
+// Registration
+// ---------------------------------------------------------------------------
+
+#[ctor::ctor]
+fn register_token_filters() {
+    register_filter_plugin("head_tokens", || Box::new(HeadTokensFilter::new(0)))
+        .expect("Failed to register head_tokens filter");
+    register_filter_plugin("skip_tokens", || Box::new(SkipTokensFilter::new(0)))
+        .expect("Failed to register skip_tokens filter");
+    register_filter_plugin("tail_tokens", || Box::new(TailTokensFilter::new(0)))
+        .expect("Failed to register tail_tokens filter");
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::io::Cursor;
+
+    fn make_tokenizer() -> Tokenizer {
+        get_tokenizer(TokenEncoding::Cl100kBase).clone()
+    }
+
+    #[test]
+    fn test_head_tokens_basic() {
+        let mut filter = HeadTokensFilter::new(3);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"The quick brown fox";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+
+        let result = String::from_utf8_lossy(&output);
+        // "The quick brown" is typically 3 tokens
+        assert!(!result.is_empty());
+        assert!(result.len() <= input.len());
+    }
+
+    #[test]
+    fn test_head_tokens_zero() {
+        let mut filter = HeadTokensFilter::new(0);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"The quick brown fox";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+        assert!(output.is_empty());
+    }
+
+    #[test]
+    fn test_head_tokens_more_than_available() {
+        let mut filter = HeadTokensFilter::new(1000);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"Hello world";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+        assert_eq!(output, input);
+    }
+
+    #[test]
+    fn test_skip_tokens_basic() {
+        let mut filter = SkipTokensFilter::new(2);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"The quick brown fox";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+
+        let result = String::from_utf8_lossy(&output);
+        // Should have skipped some tokens
+        assert!(result.len() < input.len());
+    }
+
+    #[test]
+    fn test_skip_tokens_zero() {
+        let mut filter = SkipTokensFilter::new(0);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"Hello world";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+        assert_eq!(output, input);
+    }
+
+    #[test]
+    fn test_tail_tokens_basic() {
+        let mut filter = TailTokensFilter::new(2);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"The quick brown fox jumps over the lazy dog";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+
+        let result = String::from_utf8_lossy(&output);
+        // Should only have last 2 tokens
+        assert!(!result.is_empty());
+        assert!(result.len() < input.len());
+    }
+
+    #[test]
+    fn test_tail_tokens_zero() {
+        let mut filter = TailTokensFilter::new(0);
+        filter.tokenizer = make_tokenizer();
+
+        let input = b"Hello world";
+        let mut output = Vec::new();
+        filter.filter(&mut Cursor::new(input), &mut output).unwrap();
+        assert!(output.is_empty());
+    }
+
+    #[test]
+    fn test_map_lossy_pos_ascii() {
+        let original = b"Hello world";
+        let lossy = String::from_utf8_lossy(original);
+        assert_eq!(map_lossy_pos_to_bytes(original, &lossy, 5), 5);
+        assert_eq!(map_lossy_pos_to_bytes(original, &lossy, 0), 0);
+        assert_eq!(map_lossy_pos_to_bytes(original, &lossy, 11), 11);
+    }
+
+    #[test]
+    fn test_map_lossy_pos_with_invalid_utf8() {
+        let original = b"Hello\x80world";
+        let lossy = String::from_utf8_lossy(original);
+        // lossy = "Hello\u{FFFD}world" (13 bytes)
+        // Position 5 in lossy = after "Hello" = position 5 in original
+        assert_eq!(map_lossy_pos_to_bytes(original, &lossy, 5), 5);
+        // Position 8 in lossy = "Hello\u{FFFD}" = position 6 in original
+        // (the invalid byte \x80 at position 5 was replaced)
+        assert_eq!(map_lossy_pos_to_bytes(original, &lossy, 8), 6);
+    }
+}

src/filter_plugin/utils.rs

@@ -0,0 +1,33 @@
+use std::io::Result;
+
+/// Creates a filter chain from a filter string specification.
+///
+/// # Arguments
+///
+/// * `filter_str` - The string describing the filter chain, such as "head_lines(10)|grep(pattern=error)"
+///
+/// # Returns
+///
+/// * `Result<Option<super::FilterChain>>` - A result containing:
+///     * `Ok(Some(FilterChain))` if parsing succeeds
+///     * `Ok(None)` if the filter string is empty
+///     * `Err(io::Error)` if the string is invalid
+pub fn create_filter_chain(filter_str: &str) -> Result<Option<super::FilterChain>> {
+    super::parse_filter_string(filter_str).map(Some)
+}
+
+/// Parses a string into a number of type T.
+///
+/// # Arguments
+///
+/// * `s` - The string to parse into a number
+///
+/// # Returns
+///
+/// * `Result<T>` - A result containing:
+///     * `Ok(T)` - The parsed number on success
+///     * `Err(io::Error)` - If the string is not a valid number
+pub fn parse_number<T: std::str::FromStr>(s: &str) -> Result<T> {
+    s.parse::<T>()
+        .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidInput, "Invalid number"))
+}

src/import_tar.rs

@@ -0,0 +1,225 @@
+use anyhow::{Context, Result, anyhow};
+use log::debug;
+use std::collections::HashMap;
+use std::fs;
+use std::io::{Read, Write};
+use std::path::Path;
+use std::str::FromStr;
+use tempfile::TempDir;
+
+use tar::Archive;
+
+use crate::common::PIPESIZE;
+use crate::compression_engine::CompressionType;
+use crate::db;
+use crate::modes::common::ImportMeta;
+
+/// Represents a parsed tar entry from an export archive.
+struct TarEntry {
+    /// Path to the extracted data file in the temp directory.
+    data_path: Option<std::path::PathBuf>,
+    /// Path to the extracted meta file in the temp directory.
+    meta_path: Option<std::path::PathBuf>,
+}
+
+/// Import all items from a `.keep.tar` archive.
+///
+/// Items are imported in ascending order of their original IDs,
+/// ensuring chronological ordering is preserved. Each imported item
+/// receives a new auto-incremented ID from the target database.
+///
+/// # Arguments
+///
+/// * `tar_path` - Path to the `.keep.tar` file.
+/// * `conn` - Mutable database connection.
+/// * `data_path` - Path to the data storage directory.
+///
+/// # Returns
+///
+/// A list of newly assigned item IDs.
+pub fn import_from_tar(
+    tar_path: &Path,
+    conn: &mut rusqlite::Connection,
+    data_path: &Path,
+) -> Result<Vec<i64>> {
+    let file = fs::File::open(tar_path)
+        .with_context(|| format!("Cannot open tar file: {}", tar_path.display()))?;
+    let mut archive = Archive::new(file);
+
+    let tmp_dir = TempDir::new().context("Cannot create temporary directory for import")?;
+    let tmp_path = tmp_dir.path();
+
+    // Extract entries to temp dir
+    let mut entries_map: HashMap<i64, TarEntry> = HashMap::new();
+
+    for entry_result in archive.entries().context("Cannot read tar entries")? {
+        let mut entry = entry_result.context("Cannot read tar entry")?;
+        let entry_path = entry.path().context("Cannot get entry path")?.to_path_buf();
+
+        let path_str = entry_path.to_string_lossy().replace('\\', "/");
+
+        // Reject path traversal attempts
+        if path_str.starts_with('/') || path_str.starts_with("..") || path_str.contains("/../") {
+            return Err(anyhow!("Rejected path traversal entry: {path_str}"));
+        }
+
+        // Skip directory entries
+        if entry.header().entry_type().is_dir() {
+            debug!("IMPORT_TAR: Skipping directory entry: {path_str}");
+            continue;
+        }
+
+        // Parse: <dir>/<id>.data.<compression> or <dir>/<id>.meta.yml
+        let filename = entry_path
+            .file_name()
+            .ok_or_else(|| anyhow!("Invalid entry path: {path_str}"))?
+            .to_string_lossy();
+
+        let (orig_id, is_data) = if let Some(id_str) = filename.strip_suffix(".meta.yml") {
+            let id: i64 = id_str
+                .parse()
+                .with_context(|| format!("Invalid ID in entry: {path_str}"))?;
+            (id, false)
+        } else if let Some(dot_pos) = filename.find(".data.") {
+            let id_str = &filename[..dot_pos];
+            let id: i64 = id_str
+                .parse()
+                .with_context(|| format!("Invalid ID in entry: {path_str}"))?;
+            (id, true)
+        } else {
+            debug!("IMPORT_TAR: Skipping unrecognized entry: {path_str}");
+            continue;
+        };
+
+        let entry_ref = entries_map.entry(orig_id).or_insert_with(|| TarEntry {
+            data_path: None,
+            meta_path: None,
+        });
+
+        if is_data {
+            let dest = tmp_path.join(format!("{orig_id}.data"));
+            let mut dest_file = fs::File::create(&dest).context("Cannot create temp data file")?;
+            let mut buf = [0u8; PIPESIZE];
+            loop {
+                let n = entry.read(&mut buf)?;
+                if n == 0 {
+                    break;
+                }
+                dest_file.write_all(&buf[..n])?;
+            }
+            entry_ref.data_path = Some(dest);
+            debug!("IMPORT_TAR: Extracted data for original ID {orig_id}");
+        } else {
+            let dest = tmp_path.join(format!("{orig_id}.meta.yml"));
+            let mut dest_file = fs::File::create(&dest).context("Cannot create temp meta file")?;
+            let mut buf = [0u8; PIPESIZE];
+            loop {
+                let n = entry.read(&mut buf)?;
+                if n == 0 {
+                    break;
+                }
+                dest_file.write_all(&buf[..n])?;
+            }
+            entry_ref.meta_path = Some(dest);
+            debug!("IMPORT_TAR: Extracted meta for original ID {orig_id}");
+        }
+    }
+
+    if entries_map.is_empty() {
+        return Err(anyhow!("No items found in archive"));
+    }
+
+    // Sort by original ID ascending
+    let mut sorted_ids: Vec<i64> = entries_map.keys().copied().collect();
+    sorted_ids.sort_unstable();
+
+    let mut imported_ids = Vec::new();
+
+    for orig_id in sorted_ids {
+        let entry = entries_map.get(&orig_id).expect("ID should exist in map");
+
+        let meta_path = entry
+            .meta_path
+            .as_ref()
+            .ok_or_else(|| anyhow!("Item {orig_id} missing .meta.yml entry"))?;
+        let data_path_entry = entry
+            .data_path
+            .as_ref()
+            .ok_or_else(|| anyhow!("Item {orig_id} missing .data entry"))?;
+
+        // Parse metadata
+        let meta_yaml = fs::read_to_string(meta_path)
+            .with_context(|| format!("Cannot read meta file for item {orig_id}"))?;
+        let import_meta: ImportMeta = serde_yaml::from_str(&meta_yaml)
+            .with_context(|| format!("Cannot parse meta file for item {orig_id}"))?;
+
+        // Validate compression type
+        CompressionType::from_str(&import_meta.compression).map_err(|_| {
+            anyhow!(
+                "Invalid compression type '{}' for item {}",
+                import_meta.compression,
+                orig_id
+            )
+        })?;
+
+        // Create item with original timestamp
+        let item = db::insert_item_with_ts(conn, import_meta.ts, &import_meta.compression)?;
+        let new_id = item.id.context("New item missing ID")?;
+
+        // Set tags
+        let tags = if !import_meta.tags.is_empty() {
+            db::set_item_tags(conn, item.clone(), &import_meta.tags)?;
+            import_meta.tags.clone()
+        } else {
+            Vec::new()
+        };
+
+        // Stream data to storage
+        let mut storage_path = data_path.to_path_buf();
+        storage_path.push(new_id.to_string());
+
+        let mut reader = fs::File::open(data_path_entry)
+            .with_context(|| format!("Cannot read data file for item {orig_id}"))?;
+        let mut writer = fs::File::create(&storage_path)
+            .with_context(|| format!("Cannot create storage file for item {new_id}"))?;
+        let mut buf = [0u8; PIPESIZE];
+        let mut total = 0i64;
+        loop {
+            let n = reader.read(&mut buf)?;
+            if n == 0 {
+                break;
+            }
+            writer.write_all(&buf[..n])?;
+            total += n as i64;
+        }
+
+        if total == 0 {
+            return Err(anyhow!("Item {orig_id} has empty data file"));
+        }
+
+        // Set metadata
+        for (key, value) in &import_meta.metadata {
+            db::query_upsert_meta(
+                conn,
+                db::Meta {
+                    id: new_id,
+                    name: key.clone(),
+                    value: value.clone(),
+                },
+            )?;
+        }
+
+        // Update item sizes
+        let size_to_record = import_meta.uncompressed_size.unwrap_or(total);
+        let mut updated_item = item;
+        updated_item.uncompressed_size = Some(size_to_record);
+        updated_item.compressed_size = Some(std::fs::metadata(&storage_path)?.len() as i64);
+        updated_item.closed = true;
+        db::update_item(conn, updated_item)?;
+
+        log::info!("KEEP: Imported item {new_id} (was {orig_id}) tags: {tags:?}");
+        imported_ids.push(new_id);
+    }
+
+    Ok(imported_ids)
+}

src/lib.rs

@@ -0,0 +1,109 @@
+#![deny(clippy::all)]
+#![deny(unsafe_code)]
+#![allow(unused_imports)]
+
+//! Keep library for managing temporary files with compression and metadata.
+//!
+//! This library provides core functionality for the Keep application, including
+//! database operations, compression engines, item services, and plugin systems
+//! for metadata and filtering. It supports CLI modes, server APIs, and plugin
+//! registration via ctors.
+//!
+//! # Usage
+//!
+//! Add to Cargo.toml and use re-exported types:
+//! ```toml
+//! [dependencies]
+//! keep = "0.1"
+//! ```
+//!
+//! ```rust
+//! # use keep::Args;
+//! # use clap::Parser;
+//! let args = Args::parse();
+//! ```
+//!
+//! # Features
+//!
+//! - `server`: Enables Axum-based HTTP server.
+//! - `gzip`, `lz4`: Built-in compression support.
+//! - `magic`: File type detection via libmagic.
+
+// Re-export modules for testing
+pub mod args;
+pub mod common;
+pub mod compression_engine;
+pub mod config;
+pub mod db;
+pub mod export_tar;
+pub mod filter_plugin;
+pub mod import_tar;
+pub mod meta_plugin;
+pub mod modes;
+pub mod services;
+
+#[cfg(feature = "client")]
+pub mod client;
+
+#[cfg(feature = "meta_tokens")]
+pub mod tokenizer;
+
+// Re-export Args struct for library usage
+pub use args::Args;
+// Re-export PIPESIZE constant
+pub use common::PIPESIZE;
+pub use services::CoreError;
+
+// Import all filter plugins to ensure they register themselves
+#[allow(unused_imports)]
+#[cfg(feature = "filter_grep")]
+use filter_plugin::grep;
+#[allow(unused_imports)]
+use filter_plugin::{head, skip, strip_ansi, tail};
+
+#[cfg(feature = "meta_tokens")]
+#[allow(unused_imports)]
+use filter_plugin::tokens as token_filters;
+
+use crate::meta_plugin::{
+    cwd, digest, env, exec, hostname, keep_pid, read_rate, read_time, shell, shell_pid, user,
+};
+
+#[cfg(feature = "meta_magic")]
+#[allow(unused_imports)]
+use crate::meta_plugin::magic_file;
+
+#[cfg(feature = "meta_tokens")]
+#[allow(unused_imports)]
+use crate::meta_plugin::tokens;
+
+#[cfg(feature = "meta_infer")]
+#[allow(unused_imports)]
+use crate::meta_plugin::infer_plugin;
+
+#[cfg(feature = "meta_tree_magic_mini")]
+#[allow(unused_imports)]
+use crate::meta_plugin::tree_magic_mini;
+
+/// Initializes plugins at library load time.
+///
+/// Plugin registration happens automatically via `#[ctor]` constructors
+/// when each plugin module is loaded. The explicit module imports in
+/// `lib.rs` guarantee this happens at library initialization time.
+///
+/// This function exists as a public API entry point for callers that
+/// want to explicitly ensure plugins are ready. It intentionally does
+/// no additional work.
+///
+/// # Examples
+///
+/// ```
+/// keep::init_plugins();
+/// ```
+pub fn init_plugins() {
+    // Plugins self-register via #[ctor] on module load.
+    // The use-statements in lib.rs guarantee module inclusion.
+}
+
+#[cfg(test)]
+mod tests;

src/main.rs

@@ -1,218 +1,17 @@
-use std::path::PathBuf;
+use std::io::Write;
+use std::time::Instant;
 
 use anyhow::{Context, Error, Result, anyhow};
-use clap::*;
 use clap::error::ErrorKind;
+use clap::*;
 use log::*;
-mod modes;
 
-extern crate directories;
 use directories::ProjectDirs;
 
-extern crate prettytable;
-
-use std::str::FromStr;
-
-extern crate lazy_static;
-
-pub mod compression_engine;
-pub mod db;
-pub mod plugins;
-pub mod meta_plugin;
-//pub mod item;
-
-extern crate term;
-extern crate serde_json;
-extern crate serde_yaml;
-extern crate serde;
-
-mod common;
-
-/**
- * Main struct for command-line arguments.
- */
-#[derive(Parser, Debug, Clone)]
-#[command(author, version, about, long_about = None)]
-pub struct Args {
-    #[command(flatten)]
-    mode: ModeArgs,
-    #[command(flatten)]
-    item: ItemArgs,
-    #[command(flatten)]
-    options: OptionsArgs,
-
-    #[arg(help("A list of either item IDs or tags"))]
-    ids_or_tags: Vec<NumberOrString>,
-}
-
-/**
- * Struct for mode-specific arguments.
- */
-#[derive(Parser, Debug, Clone)]
-struct ModeArgs {
-    #[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["get", "diff", "list", "update", "delete", "info", "status"]))]
-    #[arg(help("Save an item using any tags or metadata provided"))]
-    save: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "diff", "list", "update", "delete", "info", "status"]))]
-    #[arg(help(
-        "Get an item either by it's ID or by a combination of matching tags and metatdata"
-    ))]
-    get: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "list", "update", "delete", "info", "status"]))]
-    #[arg(help("Show a diff between two items by ID"))]
-    diff: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "update", "delete", "info", "status"]))]
-    #[arg(help("List items, filtering on tags or metadata if given"))]
-    list: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "status"]), requires("ids_or_tags"))]
-    #[arg(help("Update a specified item ID's tags and/or metadata"))]
-    update: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "list", "update", "info", "status"]), requires("ids_or_tags"))]
-    #[arg(help("Delete items either by ID or by matching tags"))]
-    delete: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "list", "update", "delete", "status"]), requires("ids_or_tags"))]
-    #[arg(help(
-        "Get an item either by it's ID or by a combination of matching tags and metatdata"
-    ))]
-    info: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), short('S'), long, conflicts_with_all(["save", "get", "diff", "list", "update", "delete", "info", "server"]))]
-    #[arg(help("Show status of directories and supported compression algorithms"))]
-    status: bool,
-
-    #[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "diff", "list", "update", "delete", "info", "status"]))]
-    #[arg(help("Start REST HTTP server on specified address:port or socket path"))]
-    server: Option<String>,
-}
-
-/**
- * Struct for item-specific arguments.
- */
-#[derive(Parser, Debug, Clone)]
-struct ItemArgs {
-    #[arg(help_heading("Item Options"), short, long, conflicts_with_all(["get", "delete", "status"]))]
-    #[arg(help(
-        "Set metadata for the item using the format KEY=[VALUE], the metadata will be removed if VALUE is not provided"
-    ))]
-    meta: Vec<KeyValue>,
-
-    #[arg(help_heading("Item Options"), long, env("KEEP_DIGEST"))]
-    #[arg(help("Digest algorithm to use when saving items"))]
-    digest: Option<String>,
-
-    #[arg(help_heading("Item Options"), short, long, env("KEEP_COMPRESSION"))]
-    #[arg(help("Compression algorithm to use when saving items"))]
-    compression: Option<String>,
-
-    #[arg(help_heading("Item Options"), short('M'), long, env("KEEP_META_PLUGINS"))]
-    #[arg(help("Meta plugins to use when saving items"))]
-    meta_plugins: Vec<String>,
-}
-
-
-/**
- * Struct for general options.
- */
-#[derive(Parser, Debug, Default, Clone)]
-struct OptionsArgs {
-    #[arg(long, env("KEEP_DIR"))]
-    #[arg(help("Specify the directory to use for storage"))]
-    dir: Option<PathBuf>,
-
-    #[arg(
-        long,
-        env("KEEP_LIST_FORMAT"),
-        default_value("id,time,size,tags,meta:hostname")
-    )]
-    #[arg(help("A comma separated list of columns to display with --list"))]
-    list_format: String,
-
-    #[arg(short('H'), long)]
-    #[arg(help("Display file sizes with units"))]
-    human_readable: bool,
-
-    #[arg(short, long, action = clap::ArgAction::Count, conflicts_with("quiet"))]
-    #[arg(help("Increase message verbosity, can be given more than once"))]
-    verbose: u8,
-
-    #[arg(short, long)]
-    #[arg(help("Do not show any messages"))]
-    quiet: bool,
-
-    #[arg(long, value_enum, default_value("table"))]
-    #[arg(help("Output format (only works with --info, --status, --list)"))]
-    output_format: Option<String>,
-
-    #[arg(long, env("KEEP_SERVER_PASSWORD"))]
-    #[arg(help("Password for server authentication (requires --server)"))]
-    server_password: Option<String>,
-
-    #[arg(long, help("Force output even when binary data would be sent to a TTY"))]
-    force: bool,
-}
-
-/**
- * Enum representing the different modes of operation.
- */
-#[derive(Debug, PartialEq)]
-enum KeepModes {
-    Unknown,
-    Save,
-    Get,
-    Diff,
-    List,
-    Update,
-    Delete,
-    Info,
-    Status,
-    Server,
-}
-
-/**
- * Struct for key-value pairs.
- */
-#[derive(Debug, Clone)]
-struct KeyValue {
-    key: String,
-    value: String,
-}
-
-impl FromStr for KeyValue {
-    type Err = Error;
-    fn from_str(s: &str) -> Result<Self, Error> {
-        match s.split_once('=') {
-            Some(kv) => Ok(KeyValue {
-                key: kv.0.to_string(),
-                value: kv.1.to_string(),
-            }),
-            None => Err(anyhow!("Unable to parse key=value pair")),
-        }
-    }
-}
-
-/**
- * Enum for representing either a number or a string.
- */
-#[derive(Debug, Clone)]
-enum NumberOrString {
-    Number(i64),
-    Str(String),
-}
-
-impl FromStr for NumberOrString {
-    type Err = Error;
-    fn from_str(s: &str) -> Result<Self, Self::Err> {
-        Ok(s.parse::<i64>()
-            .map(NumberOrString::Number)
-            .unwrap_or_else(|_| NumberOrString::Str(s.to_string())))
-    }
-}
+use keep::args::{Args, NumberOrString};
+use keep::config::Settings;
+use keep::db;
+use keep::modes;
 
 /**
  * Main function to handle command-line arguments and execute the appropriate mode.
@@ -222,31 +21,112 @@ fn main() -> Result<(), Error> {
     let proj_dirs = ProjectDirs::from("gt0.ca", "Andrew Phillips", "Keep");
 
     let mut cmd = Args::command();
-    let mut args = Args::parse();
+    let args = Args::parse();
 
-    stderrlog::new()
-        .module(module_path!())
-        .quiet(args.options.quiet)
-        .verbosity(usize::from(args.options.verbose + 2))
-        //.timestamp(stderrlog::Timestamp::Second)
-        .init()
-        .unwrap();
+    // Validate arguments based on mode
+    if let Err(e) = args.validate() {
+        cmd.error(ErrorKind::ValueValidation, e).exit();
+    }
+
+    // Handle --generate-completion early (prints to stdout and exits)
+    if let Some(shell) = args.mode.generate_completion {
+        clap_complete::generate(shell, &mut Args::command(), "keep", &mut std::io::stdout());
+        std::process::exit(0);
+    }
+
+    let start = Instant::now();
+    let mut builder = env_logger::Builder::new();
+    let show_module = args.options.verbose >= 2;
+    builder.format(move |buf, record| {
+        let elapsed = start.elapsed();
+        let ts = format!("[{:>6}.{:03}]", elapsed.as_secs(), elapsed.subsec_millis());
+        if show_module {
+            writeln!(
+                buf,
+                "{} {:<5} {}: {}",
+                ts,
+                record.level(),
+                record.module_path().unwrap_or("?"),
+                record.args()
+            )
+        } else {
+            writeln!(buf, "{} {:<5} {}", ts, record.level(), record.args())
+        }
+    });
+    let max_level = if args.options.quiet {
+        LevelFilter::Off
+    } else {
+        match args.options.verbose {
+            0 => LevelFilter::Warn,
+            1 => LevelFilter::Debug,
+            _ => LevelFilter::Trace,
+        }
+    };
+    builder.filter_module("keep", max_level);
+    builder.init();
 
     debug!("MAIN: Start");
 
+    // Determine default data directory
+    let default_dir = match proj_dirs {
+        Some(ref proj_dirs) => proj_dirs.data_dir().to_path_buf(),
+        None => return Err(anyhow!("Unable to determine data directory")),
+    };
+
+    // Create unified settings using the new config system
+    let settings = Settings::new(&args, default_dir)?;
+
+    debug!("MAIN: Loaded settings: {settings:?}");
+
     let ids = &mut Vec::new();
     let tags = &mut Vec::new();
 
+    // For --info, --get, --export, and --list modes, treat numeric strings as IDs
     for v in args.ids_or_tags.iter() {
+        debug!("MAIN: Parsed value: {v:?}");
         match v.clone() {
-            NumberOrString::Number(num) => ids.push(num),
-            NumberOrString::Str(str) => tags.push(str),
+            NumberOrString::Number(num) => {
+                debug!("MAIN: Adding to ids: {num}");
+                ids.push(num)
+            }
+            NumberOrString::Str(str) => {
+                // For --info, --get, --export, and --list, try to parse strings as numbers to treat them as IDs
+                if (args.mode.info || args.mode.get || args.mode.export || args.mode.list)
+                    && let Ok(num) = str.parse::<i64>()
+                {
+                    debug!("MAIN: Adding parsed string to ids: {num}");
+                    ids.push(num);
+                    continue;
+                }
+                // If not a number, or not using --info/--get/--export/--list, treat as tag
+                debug!("MAIN: Adding to tags: {str}");
+                tags.push(str)
+            }
         }
     }
-
     tags.sort();
     tags.dedup();
 
+    /// Internal enum representing the parsed execution mode.
+    #[derive(PartialEq, Debug)]
+    enum KeepModes {
+        Unknown,
+        Save,
+        Get,
+        Diff,
+        List,
+        Delete,
+        Info,
+        Update,
+        Export,
+        Import,
+        Status,
+        StatusPlugins,
+        #[cfg(feature = "server")]
+        Server,
+        GenerateConfig,
+    }
+
     let mut mode: KeepModes = KeepModes::Unknown;
 
     if args.mode.save {
@@ -259,15 +139,28 @@ fn main() -> Result<(), Error> {
         mode = KeepModes::List;
     } else if args.mode.delete {
         mode = KeepModes::Delete;
-    } else if args.mode.update {
-        mode = KeepModes::Update;
     } else if args.mode.info {
         mode = KeepModes::Info;
+    } else if args.mode.update {
+        mode = KeepModes::Update;
+    } else if args.mode.export {
+        mode = KeepModes::Export;
+    } else if args.mode.import.is_some() {
+        mode = KeepModes::Import;
     } else if args.mode.status {
         mode = KeepModes::Status;
-    } else if args.mode.server.is_some() {
+    } else if args.mode.status_plugins {
+        mode = KeepModes::StatusPlugins;
+    }
+    #[cfg(feature = "server")]
+    {
+        if args.mode.server {
             mode = KeepModes::Server;
         }
+    }
+    if args.mode.generate_config {
+        mode = KeepModes::GenerateConfig;
+    }
 
     if mode == KeepModes::Unknown {
         if !ids.is_empty() {
@@ -278,90 +171,202 @@ fn main() -> Result<(), Error> {
     }
 
     // Validate output format usage
-    if let Some(output_format_str) = &args.options.output_format {
-        if output_format_str != "table" && mode != KeepModes::Info && mode != KeepModes::Status && mode != KeepModes::List {
+    if let Some(output_format_str) = &settings.output_format
+        && output_format_str != "table"
+        && mode != KeepModes::Info
+        && mode != KeepModes::Status
+        && mode != KeepModes::StatusPlugins
+        && mode != KeepModes::List
+    {
         cmd.error(
                 ErrorKind::InvalidValue,
-                "--output-format can only be used with --info, --status, or --list modes"
+                "--output-format can only be used with --info, --status, --status-plugins, or --list modes"
             ).exit();
     }
-    }
 
     // Validate human-readable usage
-    if args.options.human_readable && mode != KeepModes::List && mode != KeepModes::Info {
+    if settings.human_readable && mode != KeepModes::List && mode != KeepModes::Info {
         cmd.error(
             ErrorKind::InvalidValue,
-            "--human-readable can only be used with --list and --info modes"
-        ).exit();
+            "--human-readable can only be used with --list and --info modes",
+        )
+        .exit();
     }
 
     // Validate server password usage
-    if args.options.server_password.is_some() && mode != KeepModes::Server {
+    #[cfg(feature = "server")]
+    if settings.server_password().is_some() && mode != KeepModes::Server {
         cmd.error(
             ErrorKind::InvalidValue,
-            "--server-password can only be used with --server mode"
-        ).exit();
+            "--server-password can only be used with --server mode",
+        )
+        .exit();
     }
 
-    debug!("MAIN: args: {:?}", args);
-    debug!("MAIN: ids: {:?}", ids);
-    debug!("MAIN: tags: {:?}", tags);
-    debug!("MAIN: mode: {:?}", mode);
+    // Validate ids-only usage
+    if settings.ids_only && mode != KeepModes::List {
+        cmd.error(
+            ErrorKind::InvalidValue,
+            "--ids-only can only be used with --list mode",
+        )
+        .exit();
+    }
 
-    if args.options.dir.is_none() {
-        match proj_dirs {
-            Some(proj_dirs) => args.options.dir = Some(proj_dirs.data_dir().to_path_buf()),
-            None => return Err(anyhow!("Unable to determine data directory")),
+    debug!("MAIN: args: {args:?}");
+    debug!("MAIN: ids: {ids:?}");
+    debug!("MAIN: tags: {tags:?}");
+    debug!("MAIN: mode: {mode:?}");
+    debug!("MAIN: settings: {settings:?}");
+
+    // Parse filter chain early for better error reporting
+    let filter_chain = if let Some(filter_str) = &args.item.filters {
+        match keep::filter_plugin::parse_filter_string(filter_str) {
+            Ok(chain) => Some(chain),
+            Err(e) => {
+                cmd.error(
+                    ErrorKind::InvalidValue,
+                    format!("Invalid filter string: {e}"),
+                )
+                .exit();
+            }
+        }
+    } else {
+        None
+    };
+
+    // Check for client mode
+    #[cfg(feature = "client")]
+    {
+        if let Some(ref client_url) = settings.client_url {
+            let client = keep::client::KeepClient::new(
+                client_url,
+                settings.client_username.clone(),
+                settings.client_password.clone(),
+                settings.client_jwt.clone(),
+            )?;
+
+            return match mode {
+                KeepModes::Save => {
+                    let metadata: std::collections::HashMap<String, String> = settings
+                        .meta
+                        .iter()
+                        .filter_map(|(k, v)| v.as_ref().map(|val| (k.clone(), val.clone())))
+                        .collect();
+                    keep::modes::client::save::mode(&client, &mut cmd, &settings, tags, metadata)
+                }
+                KeepModes::Get => keep::modes::client::get::mode(
+                    &client,
+                    &mut cmd,
+                    &settings,
+                    ids,
+                    tags,
+                    filter_chain,
+                ),
+                KeepModes::List => {
+                    keep::modes::client::list::mode(&client, &mut cmd, &settings, ids, tags)
+                }
+                KeepModes::Delete => {
+                    keep::modes::client::delete::mode(&client, &mut cmd, &settings, ids)
+                }
+                KeepModes::Info => {
+                    keep::modes::client::info::mode(&client, &mut cmd, &settings, ids, tags)
+                }
+                KeepModes::Diff => {
+                    keep::modes::client::diff::mode(&client, &mut cmd, &settings, ids)
+                }
+                KeepModes::Status => {
+                    keep::modes::client::status::mode(&client, &mut cmd, &settings)
+                }
+                KeepModes::Update => {
+                    keep::modes::client::update::mode(&client, &mut cmd, &settings, ids, tags)
+                }
+                KeepModes::Export => {
+                    keep::modes::client::export::mode(&client, &mut cmd, &settings, ids, tags)
+                }
+                KeepModes::Import => {
+                    let meta_file = args.mode.import.as_ref().unwrap();
+                    keep::modes::client::import::mode(&client, &mut cmd, &settings, meta_file)
+                }
+                _ => {
+                    cmd.error(
+                        ErrorKind::InvalidValue,
+                        format!("Mode {mode:?} is not supported in client mode"),
+                    )
+                    .exit();
+                }
+            };
         }
     }
 
+    // SAFETY: umask is thread-safe by POSIX spec, and we invoke it exactly once
+    // before any file operations to set a secure default mask. No other threads
+    // exist yet at this point in main(), so there is no data race.
     unsafe {
         libc::umask(0o077);
     }
 
-    let data_path = args.options.dir.clone().unwrap();
+    let data_path = settings.dir.clone();
     let mut db_path = data_path.clone();
     db_path.push("keep-1.db");
 
-    debug!("MAIN: Data directory: {:?}", data_path);
-    debug!("MAIN: DB file: {:?}", db_path);
+    debug!("MAIN: Data directory: {data_path:?}");
+    debug!("MAIN: DB file: {db_path:?}");
 
-    fs::create_dir_all(data_path.clone()).context("Problem creating data directory")?;
-    debug!("MAIN: Data directory created or already exists");
+    // Ensure data directory exists
+    fs::create_dir_all(&data_path)
+        .with_context(|| format!("Unable to create data directory {data_path:?}"))?;
 
-    let mut conn = db::open(db_path.clone()).context("Problem opening database")?;
-    debug!("MAIN: DB opened successfully");
+    // Initialize database
+    let mut conn = db::open(db_path.clone())?;
 
     match mode {
         KeepModes::Save => {
-            crate::modes::save::mode_save(&mut cmd, &args, ids, tags, &mut conn, data_path)?
-        }
-        KeepModes::Get => {
-            crate::modes::get::mode_get(&mut cmd, &args, ids, tags, &mut conn, data_path)?
-        }
-        KeepModes::Diff => {
-            crate::modes::diff::mode_diff(&mut cmd, &args, ids, tags, &mut conn, data_path)?
+            modes::save::mode_save(&mut cmd, &settings, ids, tags, &mut conn, data_path)
         }
+        KeepModes::Get => modes::get::mode_get(
+            &mut cmd,
+            &settings,
+            ids,
+            tags,
+            &mut conn,
+            data_path,
+            filter_chain,
+        ),
+        KeepModes::Diff => modes::diff::mode_diff(&mut cmd, &args, &mut conn),
         KeepModes::List => {
-            crate::modes::list::mode_list(&mut cmd, &args, ids, tags, &mut conn, data_path)?
+            modes::list::mode_list(&mut cmd, &settings, ids, tags, &mut conn, data_path)
+        }
+        KeepModes::Delete => modes::delete::mode_delete(
+            &mut cmd, &settings, &settings, ids, tags, &mut conn, data_path,
+        ),
+        KeepModes::Info => {
+            modes::info::mode_info(&mut cmd, &settings, ids, tags, &mut conn, data_path)
         }
         KeepModes::Update => {
-            crate::modes::update::mode_update(&mut cmd, &args, ids, tags, &mut conn, data_path)?
+            modes::update::mode_update(&mut cmd, &settings, ids, tags, &mut conn, data_path)
         }
-        KeepModes::Info => {
-            crate::modes::info::mode_info(&mut cmd, &args, ids, tags, &mut conn, data_path)?
+        KeepModes::Export => modes::export::mode_export(
+            &mut cmd,
+            &settings,
+            ids,
+            tags,
+            &mut conn,
+            data_path,
+            filter_chain,
+        ),
+        KeepModes::Import => {
+            let meta_file = args.mode.import.as_ref().unwrap();
+            modes::import::mode_import(&mut cmd, &settings, meta_file, &mut conn, data_path)
         }
-        KeepModes::Delete => {
-            crate::modes::delete::mode_delete(&mut cmd, &args, ids, tags, &mut conn, data_path)?
+        KeepModes::Status => modes::status::mode_status(&mut cmd, &settings, data_path, db_path),
+        KeepModes::StatusPlugins => {
+            modes::status_plugins::mode_status_plugins(&mut cmd, &settings, data_path, db_path)
         }
-        KeepModes::Status => {
-            crate::modes::status::mode_status(&mut cmd, &args, data_path, db_path)?
+        #[cfg(feature = "server")]
+        KeepModes::Server => modes::server::mode_server(&mut cmd, &settings, &mut conn, data_path),
+        KeepModes::GenerateConfig => {
+            modes::generate_config::mode_generate_config(&mut cmd, &settings)
         }
-        KeepModes::Server => {
-            crate::modes::server::mode_server(&mut cmd, &args, &mut conn, data_path)?
+        KeepModes::Unknown => unreachable!(),
     }
-        _ => todo!(),
-    }
-
-    Ok(())
 }

src/meta_plugin.rs

@@ -1,86 +0,0 @@
-use anyhow::Result;
-use std::io;
-use std::io::Write;
-
-pub mod program;
-pub mod digest;
-pub mod system;
-
-
-use crate::meta_plugin::program::MetaPluginProgram;
-use crate::meta_plugin::digest::{DigestSha256MetaPlugin, ReadTimeMetaPlugin, ReadRateMetaPlugin};
-use crate::meta_plugin::system::{CwdMetaPlugin, BinaryMetaPlugin, UidMetaPlugin, UserMetaPlugin, GidMetaPlugin, GroupMetaPlugin, ShellMetaPlugin, ShellPidMetaPlugin, KeepPidMetaPlugin, HostnameMetaPlugin, FullHostnameMetaPlugin};
-
-#[derive(Debug, Eq, PartialEq, Clone, strum::EnumIter, strum::Display, strum::EnumString)]
-#[strum(ascii_case_insensitive)]
-pub enum MetaPluginType {
-    FileMagic,
-    FileMime,
-    FileEncoding,
-    LineCount,
-    WordCount,
-    Cwd,
-    Binary,
-    Uid,
-    User,
-    Gid,
-    Group,
-    Shell,
-    ShellPid,
-    KeepPid,
-    DigestSha256,
-    DigestMd5,
-    ReadTime,
-    ReadRate,
-    Hostname,
-    FullHostname,
-}
-
-pub trait MetaPlugin {
-    fn is_supported(&self) -> bool {
-        true
-    }
-    
-    fn is_internal(&self) -> bool {
-        false
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>>;
-    fn finalize(&mut self) -> io::Result<String>;
-
-    // Update the meta plugin with new data
-    fn update(&mut self, data: &[u8]);
-
-    fn meta_name(&mut self) -> String;
-    
-    // Get program information for display in status
-    fn program_info(&self) -> Option<(&str, Vec<&str>)> {
-        None
-    }
-}
-
-pub fn get_meta_plugin(meta_plugin_type: MetaPluginType) -> Box<dyn MetaPlugin> {
-    match meta_plugin_type {
-        MetaPluginType::FileMagic => Box::new(MetaPluginProgram::new("file", vec!["-bE", "-"], "file_magic".to_string(), true)),
-        MetaPluginType::FileMime => Box::new(MetaPluginProgram::new("file", vec!["-b", "--mime-type", "-"], "file_mime".to_string(), true)),
-        MetaPluginType::FileEncoding => Box::new(MetaPluginProgram::new("file", vec!["-b", "--mime-encoding", "-"], "file_encoding".to_string(), true)),
-        MetaPluginType::LineCount => Box::new(MetaPluginProgram::new("wc", vec!["-l"], "line_count".to_string(), true)),
-        MetaPluginType::WordCount => Box::new(MetaPluginProgram::new("wc", vec!["-w"], "word_count".to_string(), true)),
-        MetaPluginType::Cwd => Box::new(CwdMetaPlugin::new()),
-        MetaPluginType::Binary => Box::new(BinaryMetaPlugin::new()),
-        MetaPluginType::Uid => Box::new(UidMetaPlugin::new()),
-        MetaPluginType::User => Box::new(UserMetaPlugin::new()),
-        MetaPluginType::Gid => Box::new(GidMetaPlugin::new()),
-        MetaPluginType::Group => Box::new(GroupMetaPlugin::new()),
-        MetaPluginType::Shell => Box::new(ShellMetaPlugin::new()),
-        MetaPluginType::ShellPid => Box::new(ShellPidMetaPlugin::new()),
-        MetaPluginType::KeepPid => Box::new(KeepPidMetaPlugin::new()),
-        MetaPluginType::DigestSha256 => Box::new(DigestSha256MetaPlugin::new()),
-        MetaPluginType::DigestMd5 => Box::new(MetaPluginProgram::new("md5sum", vec![], "digest_md5".to_string(), true)),
-        MetaPluginType::ReadTime => Box::new(ReadTimeMetaPlugin::new()),
-        MetaPluginType::ReadRate => Box::new(ReadRateMetaPlugin::new()),
-        MetaPluginType::Hostname => Box::new(HostnameMetaPlugin::new()),
-        MetaPluginType::FullHostname => Box::new(FullHostnameMetaPlugin::new()),
-    }
-}
-

src/meta_plugin/cwd.rs

@@ -0,0 +1,141 @@
+use crate::meta_plugin::{MetaPlugin, MetaPluginType};
+use std::env;
+
+#[derive(Debug, Clone, Default)]
+pub struct CwdMetaPlugin {
+    is_finalized: bool,
+    base: crate::meta_plugin::BaseMetaPlugin,
+}
+
+impl CwdMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> CwdMetaPlugin {
+        let mut base = crate::meta_plugin::BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = vec!["cwd".to_string()];
+        for output_name in default_outputs {
+            base.outputs
+                .insert(output_name.clone(), serde_yaml::Value::String(output_name));
+        }
+
+        // Apply provided options and outputs
+        if let Some(opts) = options {
+            for (key, value) in opts {
+                base.options.insert(key, value);
+            }
+        }
+        if let Some(outs) = outputs {
+            for (key, value) in outs {
+                base.outputs.insert(key, value);
+            }
+        }
+
+        CwdMetaPlugin {
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for CwdMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Cwd
+    }
+
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+        let cwd = match env::current_dir() {
+            Ok(path) => path.to_string_lossy().to_string(),
+            Err(_) => "unknown".to_string(),
+        };
+
+        // Use process_metadata_outputs to handle output mapping
+        if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+            "cwd",
+            serde_yaml::Value::String(cwd),
+            self.base.outputs(),
+        ) {
+            metadata.push(meta_data);
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: false,
+        }
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_cwd_plugin() {
+    register_meta_plugin(MetaPluginType::Cwd, |options, outputs| {
+        Box::new(CwdMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register CwdMetaPlugin");
+}

src/meta_plugin/digest.rs

@@ -1,159 +1,284 @@
-use anyhow::Result;
-use sha2::{Digest, Sha256};
-use std::io;
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+use md5;
+use sha2::{Digest, Sha256, Sha512};
 use std::io::Write;
-use std::time::Instant;
 
-use crate::meta_plugin::MetaPlugin;
-
-#[derive(Debug, Clone, Default)]
-pub struct DigestSha256MetaPlugin {
-    hasher: Sha256,
-    meta_name: String,
+#[derive(Clone)]
+enum Hasher {
+    Sha256(Sha256),
+    Md5(md5::Context),
+    Sha512(Sha512),
 }
 
-impl DigestSha256MetaPlugin {
-    pub fn new() -> DigestSha256MetaPlugin {
-        DigestSha256MetaPlugin {
-            hasher: Sha256::new(),
-            meta_name: "digest_sha256".to_string(),
+impl Default for Hasher {
+    fn default() -> Self {
+        Hasher::Sha256(Sha256::default())
+    }
+}
+
+// Manual Debug implementation to avoid md5::Context not implementing Debug
+impl std::fmt::Debug for Hasher {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Hasher::Sha256(_) => write!(f, "Hasher::Sha256"),
+            Hasher::Md5(_) => write!(f, "Hasher::Md5"),
+            Hasher::Sha512(_) => write!(f, "Hasher::Sha512"),
         }
     }
 }
 
-impl MetaPlugin for DigestSha256MetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        // For meta plugins, we don't actually create a writer since we're buffering data internally
-        // This method is required by the trait but not used in the same way as digest engines
-        Ok(Box::new(DummyWriter))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        let result = self.hasher.clone().finalize();
-        Ok(format!("{:x}", result))
-    }
-
+impl Hasher {
     fn update(&mut self, data: &[u8]) {
-        self.hasher.update(data);
+        match self {
+            Hasher::Sha256(hasher) => hasher.update(data),
+            Hasher::Md5(hasher) => {
+                hasher.consume(data);
+            }
+            Hasher::Sha512(hasher) => hasher.update(data),
+        }
     }
 
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
+    fn finalize(&mut self) -> String {
+        match self {
+            Hasher::Sha256(hasher) => {
+                let result = std::mem::replace(hasher, Sha256::new()).finalize_reset();
+                format!("{result:x}")
+            }
+            Hasher::Md5(hasher) => {
+                let result = hasher.clone().compute();
+                format!("{result:x}")
+            }
+            Hasher::Sha512(hasher) => {
+                let result = std::mem::replace(hasher, Sha512::new()).finalize_reset();
+                format!("{result:x}")
+            }
         }
-}
-
-// Dummy writer that implements Write but doesn't do anything
-// This is needed to satisfy the MetaPlugin trait requirements
-struct DummyWriter;
-
-impl Write for DummyWriter {
-    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
-        Ok(buf.len())
     }
 
-    fn flush(&mut self) -> io::Result<()> {
-        Ok(())
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct ReadTimeMetaPlugin {
-    start_time: Option<Instant>,
-    meta_name: String,
-}
-
-impl ReadTimeMetaPlugin {
-    pub fn new() -> ReadTimeMetaPlugin {
-        ReadTimeMetaPlugin {
-            start_time: None,
-            meta_name: "read_time".to_string(),
+    fn output_name(&self) -> &'static str {
+        match self {
+            Hasher::Sha256(_) => "digest_sha256",
+            Hasher::Md5(_) => "digest_md5",
+            Hasher::Sha512(_) => "digest_sha512",
         }
     }
 }
 
-impl MetaPlugin for ReadTimeMetaPlugin {
-    fn is_internal(&self) -> bool {
+#[derive(Debug, Default)]
+pub struct DigestMetaPlugin {
+    hasher: Option<Hasher>,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl DigestMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> DigestMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Apply provided options
+        if let Some(opts) = options {
+            for (key, value) in opts {
+                base.options.insert(key, value);
+            }
+        }
+
+        // Determine the selected method
+        let method = if let Some(method_value) = base.options.get("method") {
+            if let Some(method_str) = method_value.as_str() {
+                match method_str {
+                    "md5" => "md5",
+                    "sha256" => "sha256",
+                    "sha512" => "sha512",
+                    _ => "sha256",
+                }
+            } else {
+                "sha256"
+            }
+        } else {
+            "sha256"
+        };
+
+        // Initialize the hasher based on the method
+        let hasher = match method {
+            "md5" => Some(Hasher::Md5(md5::Context::new())),
+            "sha256" => Some(Hasher::Sha256(Sha256::new())),
+            "sha512" => Some(Hasher::Sha512(Sha512::new())),
+            _ => Some(Hasher::Sha256(Sha256::new())),
+        };
+
+        // Add the method to options so it shows up in the status
+        base.options.insert(
+            "method".to_string(),
+            serde_yaml::Value::String(method.to_string()),
+        );
+
+        // Set outputs based on the selected hash method
+        // Only the selected method's output should be enabled, others should be None
+        let all_outputs = vec!["digest_md5", "digest_sha256", "digest_sha512"];
+        for output_name in &all_outputs {
+            if output_name == &format!("digest_{method}") {
+                base.outputs.insert(
+                    output_name.to_string(),
+                    serde_yaml::Value::String(output_name.to_string()),
+                );
+            } else {
+                base.outputs
+                    .insert(output_name.to_string(), serde_yaml::Value::Null);
+            }
+        }
+
+        // Apply provided outputs, but only for enabled outputs
+        if let Some(outs) = outputs {
+            for (key, value) in outs {
+                // Only update if the output is not disabled (not None)
+                if let Some(current_value) = base.outputs.get_mut(&key)
+                    && !current_value.is_null()
+                {
+                    *current_value = value;
+                }
+            }
+        }
+
+        DigestMetaPlugin {
+            hasher,
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for DigestMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        // Update outputs based on the selected hash method
+        if let Some(hasher) = &mut self.hasher {
+            let hash_value = hasher.finalize();
+            let output_name = hasher.output_name();
+
+            // Use process_metadata_outputs to handle output mapping
+            if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                output_name,
+                serde_yaml::Value::String(hash_value),
+                self.base.outputs(),
+            ) {
+                metadata.push(meta_data);
+            }
+
+            // Set all other digest outputs to None
+            let all_outputs = vec!["digest_md5", "digest_sha256", "digest_sha512"];
+            for output_name in all_outputs {
+                if output_name != hasher.output_name() {
+                    self.base
+                        .outputs
+                        .insert(output_name.to_string(), serde_yaml::Value::Null);
+                }
+            }
+        }
+
+        self.is_finalized = true;
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn update(&mut self, data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Update the active hasher
+        if let Some(hasher) = &mut self.hasher {
+            hasher.update(data);
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Digest
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec![
+            "digest_md5".to_string(),
+            "digest_sha256".to_string(),
+            "digest_sha512".to_string(),
+        ]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+
+    fn parallel_safe(&self) -> bool {
         true
     }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        // For meta plugins, we don't actually create a writer since we're buffering data internally
-        Ok(Box::new(DummyWriter))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        if let Some(start_time) = self.start_time {
-            let duration = start_time.elapsed();
-            Ok(format!("{:.6}s", duration.as_secs_f64()))
-        } else {
-            Ok("0.000000s".to_string())
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        if self.start_time.is_none() {
-            self.start_time = Some(Instant::now());
-        }
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
 }
 
-#[derive(Debug, Clone, Default)]
-pub struct ReadRateMetaPlugin {
-    start_time: Option<Instant>,
-    bytes_read: u64,
-    meta_name: String,
-}
-
-impl ReadRateMetaPlugin {
-    pub fn new() -> ReadRateMetaPlugin {
-        ReadRateMetaPlugin {
-            start_time: None,
-            bytes_read: 0,
-            meta_name: "read_rate".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for ReadRateMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        // For meta plugins, we don't actually create a writer since we're buffering data internally
-        Ok(Box::new(DummyWriter))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        if let Some(start_time) = self.start_time {
-            let duration = start_time.elapsed();
-            if duration.as_secs_f64() > 0.0 {
-                let rate = self.bytes_read as f64 / duration.as_secs_f64();
-                Ok(format!("{:.0} B/s", rate))
-            } else {
-                Ok("0 B/s".to_string())
-            }
-        } else {
-            Ok("0 B/s".to_string())
-        }
-    }
-
-    fn update(&mut self, data: &[u8]) {
-        if self.start_time.is_none() {
-            self.start_time = Some(Instant::now());
-        }
-        self.bytes_read += data.len() as u64;
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_digest_plugin() {
+    register_meta_plugin(MetaPluginType::Digest, |options, outputs| {
+        Box::new(DigestMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register DigestMetaPlugin");
 }

src/meta_plugin/env.rs

@@ -0,0 +1,256 @@
+use super::{BaseMetaPlugin, MetaPlugin, MetaPluginType, process_metadata_outputs};
+
+#[derive(Debug, Clone)]
+/// Meta plugin that extracts environment variables prefixed with KEEP_META_ as metadata.
+pub struct EnvMetaPlugin {
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+    env_vars: Vec<(String, String)>,
+}
+
+impl EnvMetaPlugin {
+    /// Creates a new `EnvMetaPlugin` instance.
+    ///
+    /// Collects environment variables starting with KEEP_META_ and sets up default output mappings.
+    ///
+    /// # Arguments
+    ///
+    /// * `_options` - Optional configuration options for the plugin (unused in this implementation).
+    /// * `outputs` - Optional output mappings for metadata (overrides defaults).
+    ///
+    /// # Returns
+    ///
+    /// A new instance of `EnvMetaPlugin`.
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> Self {
+        let mut env_vars = Vec::new();
+        let mut outputs_map = std::collections::HashMap::new();
+
+        // Use options from --meta-plugin JSON if provided and non-empty,
+        // otherwise fall back to KEEP_META_* environment variables.
+        let use_options = options.as_ref().map(|o| !o.is_empty()).unwrap_or(false);
+
+        if use_options {
+            let opts = options.as_ref().unwrap();
+            for (key, value) in opts {
+                let value_str = match value {
+                    serde_yaml::Value::String(s) => s.clone(),
+                    serde_yaml::Value::Number(n) => n.to_string(),
+                    serde_yaml::Value::Bool(b) => b.to_string(),
+                    _ => serde_yaml::to_string(value).unwrap_or_default(),
+                };
+                env_vars.push((key.clone(), value_str));
+                outputs_map.insert(key.clone(), serde_yaml::Value::String(key.clone()));
+            }
+        } else {
+            // Fall back to KEEP_META_* environment variables
+            for (key, value) in std::env::vars() {
+                if let Some(stripped_key) = key.strip_prefix("KEEP_META_") {
+                    env_vars.push((stripped_key.to_string(), value));
+                    outputs_map.insert(
+                        stripped_key.to_string(),
+                        serde_yaml::Value::String(stripped_key.to_string()),
+                    );
+                }
+            }
+        }
+
+        // Override with provided outputs
+        if let Some(provided_outputs) = outputs {
+            for (key, value) in provided_outputs {
+                outputs_map.insert(key, value);
+            }
+        }
+
+        let mut base = BaseMetaPlugin::new();
+        base.outputs = outputs_map;
+
+        EnvMetaPlugin {
+            is_finalized: false,
+            base,
+            env_vars,
+        }
+    }
+}
+
+impl MetaPlugin for EnvMetaPlugin {
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginType::Env`.
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Env
+    }
+
+    /// Checks if the plugin has been finalized.
+    ///
+    /// # Returns
+    ///
+    /// `true` if finalized, `false` otherwise.
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    /// Sets the finalized state of the plugin.
+    ///
+    /// # Arguments
+    ///
+    /// * `finalized` - The new finalized state.
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    /// Initializes the plugin, processing environment variables.
+    ///
+    /// Processes all KEEP_META_* variables and generates metadata using output mappings.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with environment metadata and finalized state set to `true`.
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Process all collected environment variables
+        let mut metadata = Vec::new();
+        for (name, value) in &self.env_vars {
+            if let Some(meta_data) = process_metadata_outputs(
+                name,
+                serde_yaml::Value::String(value.clone()),
+                self.base.outputs(),
+            ) {
+                metadata.push(meta_data);
+            }
+        }
+
+        // Mark as finalized since this plugin only needs to run once
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    /// Updates the plugin with new data (unused in this implementation).
+    ///
+    /// This plugin does not process streaming data; returns empty response.
+    ///
+    /// # Arguments
+    ///
+    /// * `_data` - The data chunk (unused).
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with empty metadata and current finalized state.
+    fn update(&mut self, _data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    /// Finalizes the plugin, calling initialize if not already done.
+    ///
+    /// Ensures environment metadata is processed if not previously initialized.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with environment metadata if not finalized, or empty if already done.
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If not already finalized, we can call initialize
+        if !self.is_finalized {
+            return self.initialize();
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of outputs.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of outputs.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    /// Returns the default output names based on collected env vars.
+    ///
+    /// # Returns
+    ///
+    /// A vector of environment variable names (stripped of KEEP_META_ prefix).
+    fn default_outputs(&self) -> Vec<String> {
+        self.env_vars.iter().map(|(name, _)| name.clone()).collect()
+    }
+
+    /// Returns a reference to the options mapping (empty for this plugin).
+    ///
+    /// This plugin has no configurable options.
+    ///
+    /// # Returns
+    ///
+    /// An empty `HashMap`.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// # Panics
+    ///
+    /// Panics with "options_mut() not implemented for EnvMetaPlugin".
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+/// Registers the EnvMetaPlugin with the global registry at module initialization.
+#[ctor::ctor]
+fn register_env_plugin() {
+    register_meta_plugin(MetaPluginType::Env, |options, outputs| {
+        Box::new(EnvMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register EnvMetaPlugin");
+}

src/meta_plugin/exec.rs

@@ -0,0 +1,336 @@
+use log::*;
+use std::io::{self, Write};
+use std::process::{Child, Command, Stdio};
+use which::which;
+
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginResponse, MetaPluginType};
+
+/// External program execution meta plugin.
+///
+/// This plugin executes a specified external command during item save operations,
+/// capturing its output as metadata. It supports piping input data to the command's stdin
+/// and processing stdout. Useful for dynamic metadata generation via shell commands.
+///
+/// # Examples
+///
+/// Configured via options like `command: "date"`, the plugin runs `date` and captures output as metadata.
+pub struct MetaPluginExec {
+    pub program: String,
+    pub args: Vec<String>,
+    pub supported: bool,
+    pub split_whitespace: bool,
+    process: Option<Child>,
+    writer: Option<Box<dyn Write + Send>>,
+    result: Option<String>,
+    base: BaseMetaPlugin,
+}
+
+// Manual Debug implementation because Box<dyn Write> doesn't implement Debug
+/// Custom Debug implementation for MetaPluginExec.
+///
+/// Obfuscates the writer field since Box<dyn Write> does not implement Debug.
+impl std::fmt::Debug for MetaPluginExec {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("MetaPluginExec")
+            .field("program", &self.program)
+            .field("args", &self.args)
+            .field("supported", &self.supported)
+            .field("split_whitespace", &self.split_whitespace)
+            .field("process", &self.process)
+            .field("writer", &self.writer.as_ref().map(|_| "Box<dyn Write>"))
+            .field("result", &self.result)
+            .field("base", &self.base)
+            .finish()
+    }
+}
+
+impl MetaPluginExec {
+    /// Creates a new MetaPluginExec instance.
+    ///
+    /// Validates the program availability using `which` and initializes outputs and options.
+    /// The meta_name determines the default output key for captured command output.
+    ///
+    /// # Arguments
+    ///
+    /// * `program` - The executable name or path to run.
+    /// * `args` - Slice of arguments to pass to the program.
+    /// * `meta_name` - Name for the metadata output key.
+    /// * `split_whitespace` - If true, takes the first whitespace-separated word from output; otherwise, trims full output.
+    /// * `_options` - Optional configuration options (currently unused beyond passing through).
+    /// * `outputs` - Optional output mappings to override defaults.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginExec` - New plugin instance, with `supported` set based on program availability.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::meta_plugin::MetaPluginExec;
+    /// let plugin = MetaPluginExec::new("date", &[], "date_output".to_string(), false, None, None);
+    /// ```
+    pub fn new(
+        program: &str,
+        args: &[String],
+        meta_name: String,
+        split_whitespace: bool,
+        _options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> MetaPluginExec {
+        let supported = which(program).is_ok();
+
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default output
+        let default_outputs = &[meta_name.as_str()];
+        base.initialize_plugin(default_outputs, &_options, &outputs);
+
+        MetaPluginExec {
+            program: program.to_string(),
+            args: args.to_vec(),
+            supported,
+            split_whitespace,
+            process: None,
+            writer: None,
+            result: None,
+            base,
+        }
+    }
+
+    /// Starts the external process if not already running.
+    ///
+    /// Spawns the command with piped stdin/stdout and stores the child process and writer.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Empty response, initializes the process.
+    fn start_process(&mut self) -> MetaPluginResponse {
+        if self.process.is_some() {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: false,
+            };
+        }
+
+        if !self.supported {
+            debug!(
+                "META: Exec plugin: program '{}' not supported",
+                self.program
+            );
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut cmd = Command::new(&self.program);
+        cmd.args(&self.args)
+            .stdin(Stdio::piped())
+            .stdout(Stdio::piped())
+            .stderr(Stdio::piped());
+
+        match cmd.spawn() {
+            Ok(mut child) => {
+                let stdin = match child.stdin.take() {
+                    Some(s) => s,
+                    None => {
+                        error!(
+                            "META: Exec plugin: failed to capture stdin for '{}'",
+                            self.program
+                        );
+                        return MetaPluginResponse {
+                            metadata: Vec::new(),
+                            is_finalized: true,
+                        };
+                    }
+                };
+                self.writer = Some(Box::new(stdin));
+                self.process = Some(child);
+                debug!("META: Exec plugin: started process for '{}'", self.program);
+                MetaPluginResponse {
+                    metadata: Vec::new(),
+                    is_finalized: false,
+                }
+            }
+            Err(e) => {
+                error!(
+                    "META: Exec plugin: failed to start '{}': {}",
+                    self.program, e
+                );
+                MetaPluginResponse {
+                    metadata: Vec::new(),
+                    is_finalized: true,
+                }
+            }
+        }
+    }
+}
+
+impl MetaPlugin for MetaPluginExec {
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Exec
+    }
+
+    fn is_supported(&self) -> bool {
+        self.supported
+    }
+
+    fn is_internal(&self) -> bool {
+        false
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn initialize(&mut self) -> MetaPluginResponse {
+        self.start_process()
+    }
+
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        if let Some(writer) = self.writer.as_mut()
+            && let Err(e) = writer.write_all(data)
+        {
+            error!("META: Exec plugin: failed to write to stdin: {e}");
+        }
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn finalize(&mut self) -> MetaPluginResponse {
+        let mut metadata = Vec::new();
+
+        // Close stdin if writer exists
+        drop(self.writer.take());
+
+        // Wait for process to complete and capture output
+        if let Some(child) = self.process.take() {
+            match child.wait_with_output() {
+                Ok(output) => {
+                    if output.status.success() {
+                        let stdout = String::from_utf8_lossy(&output.stdout);
+                        let result = if self.split_whitespace {
+                            stdout
+                                .split_whitespace()
+                                .next()
+                                .unwrap_or(&stdout)
+                                .to_string()
+                        } else {
+                            stdout.trim().to_string()
+                        };
+
+                        self.result = Some(result.clone());
+
+                        if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                            self.base
+                                .outputs()
+                                .keys()
+                                .next()
+                                .unwrap_or(&"exec".to_string()),
+                            serde_yaml::Value::String(result),
+                            self.base.outputs(),
+                        ) {
+                            metadata.push(meta_data);
+                        }
+                    } else {
+                        let stderr = String::from_utf8_lossy(&output.stderr);
+                        error!("META: Exec plugin: command failed: {stderr}");
+                    }
+                }
+                Err(e) => {
+                    error!("META: Exec plugin: failed to wait on process: {e}");
+                }
+            }
+        }
+
+        MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn program_info(&self) -> Option<(&str, Vec<&str>)> {
+        let args_str: Vec<&str> = self.args.iter().map(|s| s.as_str()).collect();
+        Some((&self.program, args_str))
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        &self.base.outputs
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(&mut self.base.outputs)
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        &self.base.options
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(&mut self.base.options)
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["exec".to_string()]
+    }
+
+    fn parallel_safe(&self) -> bool {
+        true
+    }
+}
+
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_exec_plugin() {
+    register_meta_plugin(MetaPluginType::Exec, |options, outputs| {
+        // Parse command from options for registration
+        let mut program_name = String::new();
+        let mut args = Vec::new();
+        let mut meta_name = "exec".to_string();
+        let mut split_whitespace = false;
+
+        if let Some(opts) = &options {
+            if let Some(command_value) = opts.get("command")
+                && let Some(command_str) = command_value.as_str()
+            {
+                let parts: Vec<&str> = command_str.split_whitespace().collect();
+                if !parts.is_empty() {
+                    program_name = parts[0].to_string();
+                    args = parts[1..].iter().map(|s| s.to_string()).collect();
+                }
+            }
+            if let Some(split_value) = opts.get("split_whitespace")
+                && let Some(split_bool) = split_value.as_bool()
+            {
+                split_whitespace = split_bool;
+            }
+            if let Some(name_value) = opts.get("name")
+                && let Some(name_str) = name_value.as_str()
+            {
+                meta_name = name_str.to_string();
+            }
+        }
+
+        Box::new(MetaPluginExec::new(
+            &program_name,
+            &args,
+            meta_name,
+            split_whitespace,
+            options,
+            outputs,
+        ))
+    })
+    .expect("Failed to register ExecMetaPlugin");
+}

src/meta_plugin/hostname.rs

@@ -0,0 +1,419 @@
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+
+use smart_default::SmartDefault;
+
+#[derive(Debug, Clone, SmartDefault)]
+pub struct HostnameMetaPlugin {
+    #[default = false]
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl HostnameMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> HostnameMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["hostname", "hostname_full", "hostname_short"];
+        base.initialize_plugin(default_outputs, &options, &outputs);
+
+        // Start with default options - hostname is now boolean only
+        base.options
+            .insert("hostname".to_string(), serde_yaml::Value::Bool(true));
+        base.options
+            .insert("hostname_full".to_string(), serde_yaml::Value::Bool(true));
+        base.options
+            .insert("hostname_short".to_string(), serde_yaml::Value::Bool(true));
+
+        // Override with provided options
+        if let Some(opts) = &options {
+            for (key, value) in opts {
+                // Convert string "true"/"false" to boolean for hostname option
+                if key == "hostname"
+                    && let serde_yaml::Value::String(s) = value
+                {
+                    if s == "false" {
+                        base.options
+                            .insert(key.clone(), serde_yaml::Value::Bool(false));
+                        continue;
+                    } else if s == "true" {
+                        base.options
+                            .insert(key.clone(), serde_yaml::Value::Bool(true));
+                        continue;
+                    }
+                }
+                base.options.insert(key.clone(), value.clone());
+            }
+        }
+
+        // Determine which outputs are enabled based on options
+        let hostname_enabled = base
+            .options
+            .get("hostname")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+
+        let hostname_full_enabled = base
+            .options
+            .get("hostname_full")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+
+        let hostname_short_enabled = base
+            .options
+            .get("hostname_short")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+
+        // Start with default outputs, setting disabled ones to None
+        let mut final_outputs = std::collections::HashMap::new();
+
+        // Handle hostname output
+        if hostname_enabled {
+            final_outputs.insert(
+                "hostname".to_string(),
+                serde_yaml::Value::String("hostname".to_string()),
+            );
+        } else {
+            final_outputs.insert("hostname".to_string(), serde_yaml::Value::Null);
+        }
+
+        // Handle hostname_full output
+        if hostname_full_enabled {
+            final_outputs.insert(
+                "hostname_full".to_string(),
+                serde_yaml::Value::String("hostname_full".to_string()),
+            );
+        } else {
+            final_outputs.insert("hostname_full".to_string(), serde_yaml::Value::Null);
+        }
+
+        // Handle hostname_short output
+        if hostname_short_enabled {
+            final_outputs.insert(
+                "hostname_short".to_string(),
+                serde_yaml::Value::String("hostname_short".to_string()),
+            );
+        } else {
+            final_outputs.insert("hostname_short".to_string(), serde_yaml::Value::Null);
+        }
+
+        // Override with provided outputs, but only if they're enabled
+        if let Some(outs) = &outputs {
+            for (key, value) in outs {
+                // Only add if the output is enabled
+                match key.as_str() {
+                    "hostname" => {
+                        if hostname_enabled {
+                            final_outputs.insert(key.clone(), value.clone());
+                        }
+                    }
+                    "hostname_full" => {
+                        if hostname_full_enabled {
+                            final_outputs.insert(key.clone(), value.clone());
+                        }
+                    }
+                    "hostname_short" => {
+                        if hostname_short_enabled {
+                            final_outputs.insert(key.clone(), value.clone());
+                        }
+                    }
+                    _ => {
+                        final_outputs.insert(key.clone(), value.clone());
+                    }
+                }
+            }
+        }
+
+        base.outputs = final_outputs;
+
+        HostnameMetaPlugin {
+            is_finalized: false,
+            base,
+        }
+    }
+
+    fn get_hostname(&self) -> String {
+        // First get the short hostname
+        let short_hostname = match gethostname::gethostname().into_string() {
+            Ok(hostname) => hostname,
+            Err(_) => return "unknown".to_string(),
+        };
+
+        // First try DNS resolution for both IPv4 and IPv6 addresses
+        // lookup_host should handle both A and AAAA records
+        if let Ok(addrs_iter) = dns_lookup::lookup_host(&short_hostname) {
+            // Collect addresses into a Vec to be able to use first()
+            let addrs: Vec<std::net::IpAddr> = addrs_iter.collect();
+
+            // Try each address (both IPv4 and IPv6)
+            for addr in &addrs {
+                // Convert to IpAddr for lookup_addr
+                let ip_addr = match addr {
+                    std::net::IpAddr::V4(ipv4) => std::net::IpAddr::V4(*ipv4),
+                    std::net::IpAddr::V6(ipv6) => std::net::IpAddr::V6(*ipv6),
+                };
+                // Perform reverse lookup for each address
+                match dns_lookup::lookup_addr(&ip_addr) {
+                    Ok(full_hostname) => {
+                        // Only use if it's different from the short hostname and looks like a FQDN
+                        if full_hostname != short_hostname && full_hostname.contains('.') {
+                            return full_hostname;
+                        }
+                    }
+                    Err(_) => continue,
+                }
+            }
+
+            // If no reverse lookup worked, but we have addresses, try to construct FQDN
+            // from the first address's domain if the short hostname is part of a domain
+            if let Some(_first_addr) = addrs.first() {
+                // For local addresses, we might not get a reverse lookup, so try to infer
+                // from the system's domain name
+                if let Ok(domain) = std::process::Command::new("domainname").output()
+                    && domain.status.success()
+                {
+                    let domain_str = String::from_utf8_lossy(&domain.stdout).trim().to_string();
+                    if !domain_str.is_empty() && domain_str != "(none)" {
+                        return format!("{short_hostname}.{domain_str}");
+                    }
+                }
+            }
+        }
+
+        // Fallback: try to get the FQDN using the system's hostname resolution
+        // This should give us the full hostname if configured
+        if let Ok(full_hostname) = std::process::Command::new("hostname").arg("-f").output()
+            && full_hostname.status.success()
+        {
+            let full_hostname_str = String::from_utf8_lossy(&full_hostname.stdout)
+                .trim()
+                .to_string();
+            if !full_hostname_str.is_empty() && full_hostname_str != short_hostname {
+                return full_hostname_str;
+            }
+        }
+
+        // Final fallback: return the short hostname
+        short_hostname
+    }
+}
+
+impl MetaPlugin for HostnameMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    fn update(&mut self, _data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Hostname
+    }
+
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Get the full hostname
+        let full_hostname = self.get_hostname();
+        let short_hostname = full_hostname
+            .split('.')
+            .next()
+            .unwrap_or(&full_hostname)
+            .to_string();
+
+        // Determine which hostnames to include based on options
+        let hostname_enabled = self
+            .base
+            .options
+            .get("hostname")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+
+        let hostname_full_enabled = self
+            .base
+            .options
+            .get("hostname_full")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+
+        let hostname_short_enabled = self
+            .base
+            .options
+            .get("hostname_short")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+
+        // Always use gethostname() for the 'hostname' output when enabled
+        let hostname_value = if hostname_enabled {
+            gethostname::gethostname()
+                .into_string()
+                .unwrap_or_else(|_| "unknown".to_string())
+        } else {
+            String::new()
+        };
+
+        // Prepare metadata to return
+        let mut metadata = Vec::new();
+
+        // Add enabled metadata to the response using process_metadata_outputs
+        if hostname_enabled
+            && let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                "hostname",
+                serde_yaml::Value::String(hostname_value.clone()),
+                self.base.outputs(),
+            )
+        {
+            metadata.push(meta_data);
+        }
+        if hostname_full_enabled
+            && let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                "hostname_full",
+                serde_yaml::Value::String(full_hostname.clone()),
+                self.base.outputs(),
+            )
+        {
+            metadata.push(meta_data);
+        }
+        if hostname_short_enabled
+            && let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                "hostname_short",
+                serde_yaml::Value::String(short_hostname.clone()),
+                self.base.outputs(),
+            )
+        {
+            metadata.push(meta_data);
+        }
+
+        // Update outputs based on enabled status
+        // Handle hostname output
+        if hostname_enabled {
+            if let Some(output_value) = self.base.outputs_mut().get_mut("hostname") {
+                *output_value = serde_yaml::Value::String(hostname_value);
+            }
+        } else {
+            self.base
+                .outputs_mut()
+                .insert("hostname".to_string(), serde_yaml::Value::Null);
+        }
+
+        // Handle hostname_full output
+        if hostname_full_enabled {
+            if let Some(output_value) = self.base.outputs_mut().get_mut("hostname_full") {
+                *output_value = serde_yaml::Value::String(full_hostname);
+            }
+        } else {
+            self.base
+                .outputs_mut()
+                .insert("hostname_full".to_string(), serde_yaml::Value::Null);
+        }
+
+        // Handle hostname_short output
+        if hostname_short_enabled {
+            if let Some(output_value) = self.base.outputs_mut().get_mut("hostname_short") {
+                *output_value = serde_yaml::Value::String(short_hostname);
+            }
+        } else {
+            self.base
+                .outputs_mut()
+                .insert("hostname_short".to_string(), serde_yaml::Value::Null);
+        }
+
+        // Mark as finalized since this plugin only needs to run once
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec![
+            "hostname".to_string(),
+            "hostname_full".to_string(),
+            "hostname_short".to_string(),
+        ]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_hostname_plugin() {
+    register_meta_plugin(MetaPluginType::Hostname, |options, outputs| {
+        Box::new(HostnameMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register HostnameMetaPlugin");
+}

src/meta_plugin/infer_plugin.rs

@@ -0,0 +1,177 @@
+use crate::common::PIPESIZE;
+use crate::meta_plugin::{
+    BaseMetaPlugin, MetaPlugin, MetaPluginResponse, MetaPluginType, process_metadata_outputs,
+    register_meta_plugin,
+};
+
+#[derive(Debug, Default)]
+pub struct InferMetaPlugin {
+    buffer: Vec<u8>,
+    max_buffer_size: usize,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl InferMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> InferMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        if let Some(opts) = options {
+            for (key, value) in opts {
+                base.options.insert(key, value);
+            }
+        }
+
+        let max_buffer_size = base
+            .options
+            .get("max_buffer_size")
+            .and_then(|v| v.as_u64())
+            .unwrap_or(PIPESIZE as u64) as usize;
+
+        base.outputs.insert(
+            "infer_mime_type".to_string(),
+            serde_yaml::Value::String("infer_mime_type".to_string()),
+        );
+
+        if let Some(outs) = outputs {
+            for (key, value) in outs {
+                base.outputs.insert(key, value);
+            }
+        }
+
+        InferMetaPlugin {
+            buffer: Vec::new(),
+            max_buffer_size,
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for InferMetaPlugin {
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Infer
+    }
+
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let remaining = self.max_buffer_size.saturating_sub(self.buffer.len());
+        let to_add = &data[..data.len().min(remaining)];
+        self.buffer.extend_from_slice(to_add);
+
+        if self.buffer.len() >= self.max_buffer_size {
+            let mime_type = infer::get(&self.buffer)
+                .map(|kind| kind.mime_type().to_string())
+                .unwrap_or_else(|| "application/octet-stream".to_string());
+
+            self.is_finalized = true;
+
+            let metadata = process_metadata_outputs(
+                "infer_mime_type",
+                serde_yaml::Value::String(mime_type),
+                self.base.outputs(),
+            )
+            .map(|m| vec![m])
+            .unwrap_or_default();
+
+            return MetaPluginResponse {
+                metadata,
+                is_finalized: true,
+            };
+        }
+
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn finalize(&mut self) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mime_type = infer::get(&self.buffer)
+            .map(|kind| kind.mime_type().to_string())
+            .unwrap_or_else(|| "application/octet-stream".to_string());
+
+        self.is_finalized = true;
+
+        let metadata = process_metadata_outputs(
+            "infer_mime_type",
+            serde_yaml::Value::String(mime_type),
+            self.base.outputs(),
+        )
+        .map(|m| vec![m])
+        .unwrap_or_default();
+
+        MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["infer_mime_type".to_string()]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+
+    fn parallel_safe(&self) -> bool {
+        true
+    }
+}
+
+#[ctor::ctor]
+fn register_infer_plugin() {
+    register_meta_plugin(MetaPluginType::Infer, |options, outputs| {
+        Box::new(InferMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register InferMetaPlugin");
+}

src/meta_plugin/keep_pid.rs

@@ -0,0 +1,217 @@
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+use std::process;
+
+#[derive(Debug, Clone, Default)]
+pub struct KeepPidMetaPlugin {
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl KeepPidMetaPlugin {
+    /// Creates a new `KeepPidMetaPlugin` instance.
+    ///
+    /// # Arguments
+    ///
+    /// * `_options` - Optional configuration options for the plugin (unused in this implementation).
+    /// * `outputs` - Optional output mappings for metadata.
+    ///
+    /// # Returns
+    ///
+    /// A new instance of `KeepPidMetaPlugin`.
+    pub fn new(
+        _options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> KeepPidMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["keep_pid"];
+        base.initialize_plugin(default_outputs, &_options, &outputs);
+
+        KeepPidMetaPlugin {
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for KeepPidMetaPlugin {
+    /// Checks if the plugin has been finalized.
+    ///
+    /// # Returns
+    ///
+    /// `true` if finalized, `false` otherwise.
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    /// Sets the finalized state of the plugin.
+    ///
+    /// # Arguments
+    ///
+    /// * `finalized` - The new finalized state.
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    /// Finalizes the plugin, processing any remaining data if needed.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with empty metadata and finalized state set to `true`.
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    /// Updates the plugin with new data chunk.
+    ///
+    /// # Arguments
+    ///
+    /// * `_data` - The data chunk (unused in this implementation).
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with empty metadata and finalized state.
+    fn update(&mut self, _data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginType::KeepPid`.
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::KeepPid
+    }
+
+    /// Initializes the plugin and captures the process PID.
+    ///
+    /// Retrieves the current process ID and adds it to metadata.
+    /// Marks the plugin as finalized after one run.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Response with PID metadata and finalized state.
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+        let pid = process::id().to_string();
+
+        // Use process_metadata_outputs to handle output mapping
+        if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+            "keep_pid",
+            serde_yaml::Value::String(pid),
+            self.base.outputs(),
+        ) {
+            metadata.push(meta_data);
+        }
+
+        // Mark as finalized since this plugin only needs to run once
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of outputs.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of outputs.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    /// Returns the default output names for this plugin.
+    ///
+    /// # Returns
+    ///
+    /// Vector containing "keep_pid".
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["keep_pid".to_string()]
+    }
+
+    /// Returns a reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of options.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of options.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_keep_pid_plugin() {
+    register_meta_plugin(MetaPluginType::KeepPid, |options, outputs| {
+        Box::new(KeepPidMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register KeepPidMetaPlugin");
+}

src/meta_plugin/magic_file.rs

@@ -0,0 +1,455 @@
+#[cfg(feature = "meta_magic")]
+use magic::{Cookie, CookieFlags};
+#[cfg(not(feature = "meta_magic"))]
+use std::process::{Command, Stdio};
+
+use std::io::{self, Write};
+use std::path::Path;
+
+use crate::meta_plugin::{
+    BaseMetaPlugin, MetaData, MetaPlugin, MetaPluginResponse, MetaPluginType,
+    process_metadata_outputs,
+};
+
+// Thread-local libmagic cookie, lazily initialized on first access per thread.
+// Each thread gets its own independent Cookie instance. Libmagic documents that
+// separate cookies can be used from different threads concurrently without
+// synchronization. Using thread_local! avoids unsafe impl Send since the
+// storage is inherently !Send.
+#[cfg(feature = "meta_magic")]
+thread_local! {
+    static MAGIC_COOKIE: std::cell::RefCell<Option<Cookie>> = const { std::cell::RefCell::new(None) };
+}
+
+#[cfg(feature = "meta_magic")]
+#[derive(Debug)]
+pub struct MagicFileMetaPluginImpl {
+    buffer: Vec<u8>,
+    max_buffer_size: usize,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+#[cfg(feature = "meta_magic")]
+impl MagicFileMetaPluginImpl {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> MagicFileMetaPluginImpl {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["mime_type", "mime_encoding", "file_type"];
+        base.initialize_plugin(default_outputs, &options, &outputs);
+
+        // Get max_buffer_size from options, default to PIPESIZE
+        let max_buffer_size = base
+            .options
+            .get("max_buffer_size")
+            .and_then(|v| v.as_u64())
+            .unwrap_or(crate::common::PIPESIZE as u64) as usize;
+
+        MagicFileMetaPluginImpl {
+            buffer: Vec::new(),
+            max_buffer_size,
+            is_finalized: false,
+            base,
+        }
+    }
+
+    fn get_magic_result(&self, flags: CookieFlags) -> io::Result<String> {
+        MAGIC_COOKIE.with(|cell| {
+            // Lazy init: create cookie on first access per thread
+            {
+                let mut opt = cell.borrow_mut();
+                if opt.is_none() {
+                    let cookie = Cookie::open(CookieFlags::default())
+                        .map_err(|e| io::Error::other(format!("Failed to open magic: {e}")))?;
+                    cookie.load(&[] as &[&Path]).map_err(|e| {
+                        io::Error::other(format!("Failed to load magic database: {e}"))
+                    })?;
+                    *opt = Some(cookie);
+                }
+            }
+
+            let cookie_ref = cell.borrow();
+            let cookie = cookie_ref.as_ref().expect("cookie initialized above");
+
+            cookie
+                .set_flags(flags)
+                .map_err(|e| io::Error::other(format!("Failed to set magic flags: {e}")))?;
+
+            let result = cookie
+                .buffer(&self.buffer)
+                .map_err(|e| io::Error::other(format!("Failed to analyze buffer: {e}")))?;
+
+            Ok(result.trim().to_string())
+        })
+    }
+
+    fn process_magic_types(&self) -> Vec<MetaData> {
+        let mut metadata = Vec::new();
+
+        let types_to_process = [
+            ("mime_type", CookieFlags::MIME_TYPE),
+            ("mime_encoding", CookieFlags::MIME_ENCODING),
+            ("file_type", CookieFlags::empty()),
+        ];
+
+        for (name, flags) in types_to_process.iter() {
+            if let Ok(result) = self.get_magic_result(*flags)
+                && !result.is_empty()
+                && let Some(meta_data) = process_metadata_outputs(
+                    name,
+                    serde_yaml::Value::String(result),
+                    self.base.outputs(),
+                )
+            {
+                metadata.push(meta_data);
+            }
+        }
+
+        metadata
+    }
+}
+
+#[cfg(feature = "meta_magic")]
+impl MetaPlugin for MagicFileMetaPluginImpl {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn initialize(&mut self) -> MetaPluginResponse {
+        // Cookie is lazily initialized in the thread-local on first use.
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let remaining_capacity = self.max_buffer_size.saturating_sub(self.buffer.len());
+        if remaining_capacity > 0 {
+            let bytes_to_copy = std::cmp::min(data.len(), remaining_capacity);
+            self.buffer.extend_from_slice(&data[..bytes_to_copy]);
+
+            if self.buffer.len() >= self.max_buffer_size {
+                let metadata = self.process_magic_types();
+                self.is_finalized = true;
+                return MetaPluginResponse {
+                    metadata,
+                    is_finalized: true,
+                };
+            }
+        }
+
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn finalize(&mut self) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let metadata = self.process_magic_types();
+        self.is_finalized = true;
+
+        MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::MagicFile
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec![
+            "mime_type".to_string(),
+            "mime_encoding".to_string(),
+            "file_type".to_string(),
+        ]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+
+    fn parallel_safe(&self) -> bool {
+        true
+    }
+}
+
+#[cfg(feature = "meta_magic")]
+pub use MagicFileMetaPluginImpl as MagicFileMetaPlugin;
+
+#[cfg(not(feature = "meta_magic"))]
+#[derive(Debug)]
+pub struct FallbackMagicFileMetaPlugin {
+    buffer: Vec<u8>,
+    max_buffer_size: usize,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+#[cfg(not(feature = "meta_magic"))]
+impl FallbackMagicFileMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> Self {
+        let mut base = BaseMetaPlugin::new();
+        let default_outputs = &["mime_type", "mime_encoding", "file_type"];
+        base.initialize_plugin(default_outputs, &options, &outputs);
+
+        let max_buffer_size = base
+            .options
+            .get("max_buffer_size")
+            .and_then(|v| v.as_u64())
+            .unwrap_or(crate::common::PIPESIZE as u64) as usize;
+
+        Self {
+            buffer: Vec::new(),
+            max_buffer_size,
+            is_finalized: false,
+            base,
+        }
+    }
+
+    fn run_file_command(&self, args: &[&str]) -> Option<String> {
+        let output = Command::new("file")
+            .args(args)
+            .arg("-")
+            .stdin(Stdio::piped())
+            .stdout(Stdio::piped())
+            .spawn()
+            .and_then(|mut child| {
+                if let Some(mut stdin) = child.stdin.take() {
+                    if stdin.write_all(&self.buffer).is_err() {
+                        // Ignore write error; child will see EOF and likely fail
+                        // the file detection, returning no output.
+                    }
+                }
+                child.wait_with_output()
+            });
+
+        output
+            .ok()
+            .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string())
+    }
+
+    fn detect_type(&self) -> Vec<MetaData> {
+        let mut metadata = Vec::new();
+
+        // Get mime_type and mime_encoding via --mime
+        if let Some(mime_line) = self.run_file_command(&["--brief", "--mime"]) {
+            // Format: "text/plain; charset=us-ascii"
+            if let Some((mime_type, rest)) = mime_line.split_once(';') {
+                let mime_type = mime_type.trim().to_string();
+                let mime_encoding = rest
+                    .trim()
+                    .strip_prefix("charset=")
+                    .unwrap_or("binary")
+                    .to_string();
+
+                if let Some(meta_data) = process_metadata_outputs(
+                    "mime_type",
+                    serde_yaml::Value::String(mime_type),
+                    self.base.outputs(),
+                ) {
+                    metadata.push(meta_data);
+                }
+                if let Some(meta_data) = process_metadata_outputs(
+                    "mime_encoding",
+                    serde_yaml::Value::String(mime_encoding),
+                    self.base.outputs(),
+                ) {
+                    metadata.push(meta_data);
+                }
+            } else {
+                // No charset, just mime type
+                if let Some(meta_data) = process_metadata_outputs(
+                    "mime_type",
+                    serde_yaml::Value::String(mime_line),
+                    self.base.outputs(),
+                ) {
+                    metadata.push(meta_data);
+                }
+            }
+        }
+
+        // Get human-readable file type via --brief
+        if let Some(file_type) = self.run_file_command(&["--brief"])
+            && !file_type.is_empty()
+            && let Some(meta_data) = process_metadata_outputs(
+                "file_type",
+                serde_yaml::Value::String(file_type),
+                self.base.outputs(),
+            )
+        {
+            metadata.push(meta_data);
+        }
+
+        metadata
+    }
+}
+
+#[cfg(not(feature = "meta_magic"))]
+impl MetaPlugin for FallbackMagicFileMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn initialize(&mut self) -> MetaPluginResponse {
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let remaining = self.max_buffer_size.saturating_sub(self.buffer.len());
+        if remaining > 0 {
+            let n = std::cmp::min(data.len(), remaining);
+            self.buffer.extend_from_slice(&data[..n]);
+
+            if self.buffer.len() >= self.max_buffer_size {
+                let metadata = self.detect_type();
+                self.is_finalized = true;
+                return MetaPluginResponse {
+                    metadata,
+                    is_finalized: true,
+                };
+            }
+        }
+
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn finalize(&mut self) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+        self.is_finalized = true;
+        MetaPluginResponse {
+            metadata: self.detect_type(),
+            is_finalized: true,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::MagicFile
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec![
+            "mime_type".to_string(),
+            "mime_encoding".to_string(),
+            "file_type".to_string(),
+        ]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+
+    fn parallel_safe(&self) -> bool {
+        true
+    }
+}
+
+#[cfg(not(feature = "meta_magic"))]
+pub use FallbackMagicFileMetaPlugin as MagicFileMetaPlugin;
+
+use crate::meta_plugin::register_meta_plugin;
+
+#[ctor::ctor]
+fn register_magic_file_plugin() {
+    register_meta_plugin(MetaPluginType::MagicFile, |options, outputs| {
+        Box::new(MagicFileMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register MagicFileMetaPlugin");
+}

src/meta_plugin/mod.rs

@@ -0,0 +1,655 @@
+use log::{debug, warn};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::sync::{Arc, Mutex};
+
+pub mod cwd;
+pub mod digest;
+pub mod env;
+pub mod exec;
+pub mod hostname;
+#[cfg(feature = "meta_infer")]
+pub mod infer_plugin;
+pub mod keep_pid;
+pub mod magic_file;
+pub mod read_rate;
+pub mod read_time;
+pub mod shell;
+pub mod shell_pid;
+pub mod text;
+#[cfg(feature = "meta_tokens")]
+pub mod tokens;
+#[cfg(feature = "meta_tree_magic_mini")]
+pub mod tree_magic_mini;
+pub mod user;
+
+pub use digest::DigestMetaPlugin;
+pub use exec::MetaPluginExec;
+#[cfg(feature = "meta_magic")]
+pub use magic_file::MagicFileMetaPlugin;
+// pub use text::TextMetaPlugin; // Removed duplicate
+pub use cwd::CwdMetaPlugin;
+pub use env::EnvMetaPlugin;
+pub use hostname::HostnameMetaPlugin;
+#[cfg(feature = "meta_infer")]
+pub use infer_plugin::InferMetaPlugin;
+pub use keep_pid::KeepPidMetaPlugin;
+pub use read_rate::ReadRateMetaPlugin;
+pub use read_time::ReadTimeMetaPlugin;
+pub use shell::ShellMetaPlugin;
+pub use shell_pid::ShellPidMetaPlugin;
+#[cfg(feature = "meta_tree_magic_mini")]
+pub use tree_magic_mini::TreeMagicMiniMetaPlugin;
+pub use user::UserMetaPlugin;
+
+#[cfg(not(feature = "meta_magic"))]
+pub use magic_file::FallbackMagicFileMetaPlugin as MagicFileMetaPlugin;
+
+type PluginConstructor = fn(
+    Option<HashMap<String, serde_yaml::Value>>,
+    Option<HashMap<String, serde_yaml::Value>>,
+) -> Box<dyn MetaPlugin>;
+
+/// Represents metadata to be stored.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MetaData {
+    /// The name of the metadata field.
+    pub name: String,
+    /// The value of the metadata field.
+    pub value: String,
+}
+
+/// Response from meta plugin operations.
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct MetaPluginResponse {
+    /// The generated metadata items.
+    pub metadata: Vec<MetaData>,
+    /// Indicates if the plugin has finished processing.
+    pub is_finalized: bool,
+}
+
+/// Type alias for the save_meta callback shared by all plugins.
+pub type SaveMetaFn = Arc<Mutex<dyn FnMut(&str, &str) + Send>>;
+
+/// Creates a no-op save_meta for plugins not wired through MetaService.
+pub fn noop_save_meta() -> SaveMetaFn {
+    Arc::new(Mutex::new(|_: &str, _: &str| {}))
+}
+
+/// Base implementation for meta plugins to reduce boilerplate.
+#[derive(Clone)]
+pub struct BaseMetaPlugin {
+    /// Output mappings for metadata.
+    pub outputs: std::collections::HashMap<String, serde_yaml::Value>,
+    /// Configuration options for the plugin.
+    pub options: std::collections::HashMap<String, serde_yaml::Value>,
+    /// Whether the plugin is finalized.
+    pub is_finalized: bool,
+    /// Callback to store metadata. Called directly by plugins.
+    pub save_meta: SaveMetaFn,
+}
+
+impl std::fmt::Debug for BaseMetaPlugin {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("BaseMetaPlugin")
+            .field("outputs", &self.outputs)
+            .field("options", &self.options)
+            .field("is_finalized", &self.is_finalized)
+            .finish_non_exhaustive()
+    }
+}
+
+impl Default for BaseMetaPlugin {
+    fn default() -> Self {
+        Self {
+            outputs: HashMap::new(),
+            options: HashMap::new(),
+            is_finalized: false,
+            save_meta: noop_save_meta(),
+        }
+    }
+}
+
+impl BaseMetaPlugin {
+    /// Creates a new `BaseMetaPlugin`.
+    ///
+    /// # Returns
+    ///
+    /// A new instance of `BaseMetaPlugin`.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Returns a reference to the outputs mapping.
+    pub fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        &self.outputs
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    pub fn outputs_mut(&mut self) -> &mut std::collections::HashMap<String, serde_yaml::Value> {
+        &mut self.outputs
+    }
+
+    /// Returns a reference to the options mapping.
+    pub fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        &self.options
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    pub fn options_mut(&mut self) -> &mut std::collections::HashMap<String, serde_yaml::Value> {
+        &mut self.options
+    }
+
+    /// Sets the save_meta callback on the base plugin.
+    pub fn set_save_meta(&mut self, save_meta: SaveMetaFn) {
+        self.save_meta = save_meta;
+    }
+
+    /// Saves a metadata entry via the save_meta callback.
+    pub fn save_meta(&self, name: &str, value: &str) {
+        if let Ok(mut f) = self.save_meta.lock() {
+            f(name, value);
+        } else {
+            warn!("META_PLUGIN: save_meta lock poisoned, dropping metadata: {name}={value}");
+        }
+    }
+
+    /// Helper function to initialize plugin options and outputs.
+    ///
+    /// # Arguments
+    ///
+    /// * `default_outputs` - Slice of default output names.
+    /// * `options` - Optional user-provided options.
+    /// * `outputs` - Optional user-provided outputs.
+    pub fn initialize_plugin(
+        &mut self,
+        default_outputs: &[&str],
+        options: &Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: &Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) {
+        // Set default outputs
+        for output_name in default_outputs {
+            self.outputs.insert(
+                output_name.to_string(),
+                serde_yaml::Value::String(output_name.to_string()),
+            );
+        }
+
+        // Apply provided options and outputs
+        if let Some(opts) = options {
+            for (key, value) in opts {
+                self.options.insert(key.clone(), value.clone());
+            }
+        }
+        if let Some(outs) = outputs {
+            for (key, value) in outs {
+                self.outputs.insert(key.clone(), value.clone());
+            }
+        }
+    }
+}
+
+impl MetaPlugin for BaseMetaPlugin {
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginType::Text` (default for base).
+    fn meta_type(&self) -> MetaPluginType {
+        // This is a base implementation, so we need to return something
+        // This might not be used, but we need to satisfy the trait
+        MetaPluginType::Text
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of outputs.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        &self.outputs
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of outputs.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(&mut self.outputs)
+    }
+
+    /// Returns a reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of options.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        &self.options
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of options.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(&mut self.options)
+    }
+}
+
+#[derive(
+    Debug,
+    Eq,
+    PartialEq,
+    Clone,
+    Hash,
+    strum::EnumIter,
+    strum::Display,
+    strum::EnumString,
+    Serialize,
+    Deserialize,
+)]
+#[strum(serialize_all = "snake_case", ascii_case_insensitive)]
+pub enum MetaPluginType {
+    MagicFile,
+    Cwd,
+    Text,
+    User,
+    Shell,
+    ShellPid,
+    KeepPid,
+    Digest,
+    ReadTime,
+    ReadRate,
+    Hostname,
+    Exec,
+    Env,
+    Tokens,
+    TreeMagicMini,
+    Infer,
+}
+
+/// Central function to handle metadata output with name mapping.
+///
+/// # Arguments
+///
+/// * `internal_name` - The internal name of the metadata.
+/// * `value` - The value to process.
+/// * `outputs` - The outputs mapping.
+///
+/// # Returns
+///
+/// An optional `MetaData` if the output is enabled, `None` if disabled.
+pub fn process_metadata_outputs(
+    internal_name: &str,
+    value: serde_yaml::Value,
+    outputs: &std::collections::HashMap<String, serde_yaml::Value>,
+) -> Option<MetaData> {
+    // Check if this output is disabled
+    if let Some(mapping) = outputs.get(internal_name) {
+        // Check for null to disable the output
+        if mapping.is_null() {
+            debug!("META: Skipping disabled output (null): {internal_name}");
+            return None;
+        }
+        // Check for boolean false to disable the output
+        if let Some(false_val) = mapping.as_bool()
+            && !false_val
+        {
+            debug!("META: Skipping disabled output: {internal_name}");
+            return None;
+        }
+        if let Some(custom_name) = mapping.as_str() {
+            let value_str = yaml_value_to_string(&value);
+            debug!(
+                "META: Processing metadata: internal_name={internal_name}, custom_name={custom_name}, value={value_str}"
+            );
+            return Some(MetaData {
+                name: custom_name.to_string(),
+                value: value_str,
+            });
+        }
+    }
+
+    let value_str = yaml_value_to_string(&value);
+
+    // Default: use internal name as output name
+    debug!("META: Processing metadata: name={internal_name}, value={value_str}");
+    Some(MetaData {
+        name: internal_name.to_string(),
+        value: value_str,
+    })
+}
+
+fn yaml_value_to_string(value: &serde_yaml::Value) -> String {
+    match value {
+        serde_yaml::Value::Null => "null".to_string(),
+        serde_yaml::Value::Bool(b) => b.to_string(),
+        serde_yaml::Value::Number(n) => n.to_string(),
+        serde_yaml::Value::String(s) => s.clone(),
+        serde_yaml::Value::Sequence(_)
+        | serde_yaml::Value::Mapping(_)
+        | serde_yaml::Value::Tagged(_) => {
+            serde_yaml::to_string(value).unwrap_or_else(|_| "".to_string())
+        }
+    }
+}
+
+pub trait MetaPlugin: Send
+where
+    Self: 'static,
+{
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// The `MetaPluginType` enum variant for this plugin.
+    fn meta_type(&self) -> MetaPluginType;
+
+    /// Checks if the plugin is supported on the current system.
+    ///
+    /// # Returns
+    ///
+    /// `true` if supported, `false` otherwise.
+    fn is_supported(&self) -> bool {
+        true
+    }
+
+    /// Checks if the plugin is internal (built-in).
+    ///
+    /// # Returns
+    ///
+    /// `true` if internal, `false` otherwise.
+    fn is_internal(&self) -> bool {
+        true
+    }
+
+    /// Checks if the plugin is already finalized.
+    ///
+    /// # Returns
+    ///
+    /// `true` if finalized, `false` otherwise.
+    fn is_finalized(&self) -> bool {
+        false
+    }
+
+    /// Sets the finalized state (only for plugins that can track this).
+    ///
+    /// # Arguments
+    ///
+    /// * `_finalized` - The new finalized state (unused in default).
+    fn set_finalized(&mut self, _finalized: bool) {}
+
+    /// Updates the meta plugin with new data.
+    ///
+    /// # Arguments
+    ///
+    /// * `_data` - The data chunk to process (unused in default).
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with empty metadata and `is_finalized` set to `false`.
+    fn update(&mut self, _data: &[u8]) -> MetaPluginResponse {
+        // Default implementation does nothing
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    /// Finalizes the plugin.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with empty metadata and `is_finalized` set to `true`.
+    fn finalize(&mut self) -> MetaPluginResponse {
+        // Default implementation does nothing
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    /// Gets program information for display in status.
+    ///
+    /// # Returns
+    ///
+    /// An optional tuple of program name and arguments, or `None`.
+    fn program_info(&self) -> Option<(&str, Vec<&str>)> {
+        None
+    }
+
+    /// Initializes the plugin.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with empty metadata and `is_finalized` set to `false`.
+    fn initialize(&mut self) -> MetaPluginResponse {
+        // Default implementation does nothing
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// An empty `HashMap` (default implementation).
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        use std::sync::LazyLock;
+        static EMPTY: LazyLock<std::collections::HashMap<String, serde_yaml::Value>> =
+            LazyLock::new(std::collections::HashMap::new);
+        &EMPTY
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the outputs `HashMap`.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error if the plugin does not support mutable outputs.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        anyhow::bail!("outputs_mut() not supported by this plugin")
+    }
+
+    /// Returns a reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// An empty `HashMap` (default implementation).
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        use std::sync::LazyLock;
+        static EMPTY: LazyLock<std::collections::HashMap<String, serde_yaml::Value>> =
+            LazyLock::new(std::collections::HashMap::new);
+        &EMPTY
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the options `HashMap`.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error if the plugin does not support mutable options.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        anyhow::bail!("options_mut() not supported by this plugin")
+    }
+
+    /// Gets the default output names this plugin can produce.
+    ///
+    /// # Returns
+    ///
+    /// A vector containing the meta type as a string (default).
+    fn default_outputs(&self) -> Vec<String> {
+        // Default implementation returns the meta type as a string
+        vec![self.meta_type().to_string()]
+    }
+
+    /// Returns a description of this plugin for display in config templates.
+    ///
+    /// # Returns
+    ///
+    /// A description string (empty by default).
+    fn description(&self) -> &str {
+        ""
+    }
+
+    /// Returns true if this plugin can execute concurrently with other
+    /// parallel-safe plugins.
+    ///
+    /// Plugins that do significant per-chunk work (hashing, tokenization,
+    /// piping to child processes) should return true. The MetaService will
+    /// run all parallel-safe plugins in separate threads per phase, then
+    /// process results sequentially.
+    fn parallel_safe(&self) -> bool {
+        false
+    }
+
+    /// Builds the schema for this plugin from its options and outputs.
+    ///
+    /// Default implementation infers option types from YAML values and
+    /// collects enabled outputs.
+    ///
+    /// # Returns
+    ///
+    /// A `PluginSchema` describing this plugin's configuration.
+    fn schema(&self) -> crate::common::schema::PluginSchema {
+        use crate::common::schema::{OptionSchema, OptionType, OutputSchema, PluginSchema};
+
+        let options: Vec<OptionSchema> = self
+            .options()
+            .iter()
+            .map(|(key, value)| {
+                let option_type = OptionType::from_yaml_value(value);
+                let (default, required) = if value.is_null() {
+                    (None, true)
+                } else {
+                    (Some(value.clone()), false)
+                };
+                OptionSchema {
+                    name: key.clone(),
+                    option_type,
+                    default,
+                    required,
+                }
+            })
+            .collect();
+
+        let mut outputs: Vec<OutputSchema> = Vec::new();
+        for (key, value) in self.outputs() {
+            if !value.is_null() {
+                outputs.push(OutputSchema {
+                    name: key.clone(),
+                    description: key.clone(),
+                });
+            }
+        }
+        if outputs.is_empty() {
+            for output_name in self.default_outputs() {
+                outputs.push(OutputSchema {
+                    name: output_name.clone(),
+                    description: output_name,
+                });
+            }
+        }
+
+        PluginSchema {
+            name: self.meta_type().to_string(),
+            description: self.description().to_string(),
+            options,
+            outputs,
+        }
+    }
+
+    /// Method to downcast to concrete type (for checking finalization state).
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to `self` as `dyn Any`.
+    fn as_any_mut(&mut self) -> &mut dyn std::any::Any
+    where
+        Self: Sized,
+    {
+        self
+    }
+
+    /// Sets the save_meta callback for this plugin.
+    ///
+    /// Called by MetaService to wire the plugin to the metadata storage.
+    fn set_save_meta(&mut self, _save_meta: SaveMetaFn) {}
+
+    /// Saves a metadata entry via the save_meta callback.
+    ///
+    /// Plugins call this during initialize/update/finalize to persist metadata.
+    fn save_meta(&self, _name: &str, _value: &str) {}
+}
+
+/// Global registry for meta plugins.
+static META_PLUGIN_REGISTRY: std::sync::LazyLock<
+    Mutex<HashMap<MetaPluginType, PluginConstructor>>,
+> = std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
+
+/// Register a meta plugin with the global registry.
+///
+/// # Arguments
+///
+/// * `meta_plugin_type` - The type of the meta plugin to register.
+/// * `constructor` - The constructor function for creating plugin instances.
+pub fn register_meta_plugin(
+    meta_plugin_type: MetaPluginType,
+    constructor: PluginConstructor,
+) -> anyhow::Result<()> {
+    META_PLUGIN_REGISTRY
+        .lock()
+        .map_err(|e| anyhow::anyhow!("plugin registry poisoned: {e}"))?
+        .insert(meta_plugin_type, constructor);
+    Ok(())
+}
+
+pub fn get_meta_plugin(
+    meta_plugin_type: MetaPluginType,
+    options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+) -> anyhow::Result<Box<dyn MetaPlugin>> {
+    get_meta_plugin_with_save(meta_plugin_type, options, outputs, None)
+}
+
+/// Creates a meta plugin instance with an optional save_meta callback.
+///
+/// If `save_meta` is provided, it is wired to the plugin so it can
+/// store metadata directly during initialize/update/finalize.
+pub fn get_meta_plugin_with_save(
+    meta_plugin_type: MetaPluginType,
+    options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    save_meta: Option<SaveMetaFn>,
+) -> anyhow::Result<Box<dyn MetaPlugin>> {
+    let registry = META_PLUGIN_REGISTRY
+        .lock()
+        .map_err(|e| anyhow::anyhow!("plugin registry poisoned: {e}"))?;
+    if let Some(constructor) = registry.get(&meta_plugin_type) {
+        let mut plugin = constructor(options, outputs);
+        if let Some(sm) = save_meta {
+            plugin.set_save_meta(sm);
+        }
+        return Ok(plugin);
+    }
+
+    anyhow::bail!("Meta plugin {meta_plugin_type:?} not registered")
+}

src/meta_plugin/program.rs

@@ -1,151 +0,0 @@
-use crate::plugins::ProgramWriter;
-use anyhow::{Context, Result, anyhow};
-use log::*;
-use std::env;
-use std::fs;
-use std::io;
-use std::io::Write;
-use std::os::unix::fs::PermissionsExt;
-use std::process::{Command, Stdio};
-
-use crate::meta_plugin::MetaPlugin;
-
-#[derive(Clone, Debug)]
-pub struct MetaPluginProgram {
-    pub program: String,
-    pub args: Vec<String>,
-    pub supported: bool,
-    pub meta_name: String,
-    pub split_whitespace: bool,
-    buffer: Vec<u8>,
-}
-
-impl MetaPluginProgram {
-    pub fn new(program: &str, args: Vec<&str>, meta_name: String, split_whitespace: bool) -> MetaPluginProgram {
-        let program_path = get_program_path(program);
-        let supported = program_path.is_ok();
-
-        MetaPluginProgram {
-            program: program_path.unwrap_or(program.to_string()),
-            args: args.iter().map(|s| s.to_string()).collect(),
-            supported,
-            meta_name,
-            split_whitespace,
-            buffer: Vec::new(),
-        }
-    }
-}
-
-impl MetaPlugin for MetaPluginProgram {
-    fn is_supported(&self) -> bool {
-        self.supported
-    }
-    
-    fn is_internal(&self) -> bool {
-        false
-    }
-
-    fn create(&self) -> Result<Box<dyn Write>> {
-        debug!("META: Writing using {:?}", *self);
-
-        let program = self.program.clone();
-        let args = self.args.clone();
-
-        debug!("META: Executing command: {:?} {:?}", program, args);
-
-        let mut process = Command::new(program.clone())
-            .args(args.clone())
-            .stdin(Stdio::piped())
-            .stdout(Stdio::piped())
-            .spawn()
-            .context(anyhow!(
-                "Problem spawning child process: {:?} {:?}",
-                program,
-                args
-            ))?;
-
-        Ok(Box::new(ProgramWriter {
-            stdin: process.stdin.take().unwrap(),
-        }))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        let program = self.program.clone();
-        let args = self.args.clone();
-
-        debug!("META: Executing command for finalize: {:?} {:?}", program, args);
-
-        let mut process = Command::new(program)
-            .args(args)
-            .stdin(Stdio::piped())
-            .stdout(Stdio::piped())
-            .stderr(Stdio::piped())
-            .spawn()
-            .map_err(|e| io::Error::new(io::ErrorKind::Other, format!("Failed to spawn process: {}", e)))?;
-
-        let stdin = process.stdin.as_mut().unwrap();
-        stdin.write_all(&self.buffer)
-            .map_err(|e| io::Error::new(io::ErrorKind::Other, format!("Failed to write to stdin: {}", e)))?;
-
-        let output = process.wait_with_output()
-            .map_err(|e| io::Error::new(io::ErrorKind::Other, format!("Failed to wait for process: {}", e)))?;
-
-        if output.status.success() {
-            let stdout = String::from_utf8_lossy(&output.stdout);
-            let trimmed_result = stdout.trim();
-            
-            // For certain programs, we only want the first part before whitespace
-            if self.split_whitespace {
-                let parts: Vec<&str> = trimmed_result.split_whitespace().collect();
-                if !parts.is_empty() {
-                    Ok(parts[0].to_string())
-                } else {
-                    Ok(trimmed_result.to_string())
-                }
-            } else {
-                Ok(trimmed_result.to_string())
-            }
-        } else {
-            let stderr = String::from_utf8_lossy(&output.stderr);
-            Err(io::Error::new(
-                io::ErrorKind::Other,
-                format!("Command failed: {}", stderr.trim()),
-            ))
-        }
-    }
-
-    fn update(&mut self, data: &[u8]) {
-        self.buffer.extend_from_slice(data);
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-    
-    fn program_info(&self) -> Option<(&str, Vec<&str>)> {
-        if self.supported {
-            Some((&self.program, self.args.iter().map(|s| s.as_str()).collect()))
-        } else {
-            None
-        }
-    }
-
-}
-
-fn get_program_path(program: &str) -> Result<String> {
-    debug!("META: Looking for executable: {}", program);
-    if let Ok(path) = env::var("PATH") {
-        for p in path.split(':') {
-            let p_str = format!("{}/{}", p, program);
-            let stat = fs::metadata(p_str.clone());
-            if let Ok(stat) = stat {
-                let md = stat;
-                let permissions = md.permissions();
-                if md.is_file() && permissions.mode() & 0o111 != 0 {
-                    return Ok(p_str);
-                }
-            }
-        }
-    }
-    Err(anyhow!("Unable to find binary {} in PATH", program))
-}

src/meta_plugin/read_rate.rs

@@ -0,0 +1,250 @@
+use std::time::Instant;
+
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+
+#[derive(Debug, Clone, Default)]
+/// Meta plugin that calculates the read rate (KB/s) of input data.
+///
+/// Tracks bytes read and elapsed time, then computes the rate in finalize().
+/// Outputs the rate via configured mappings. Supports options for customization
+/// (though defaults are used here).
+///
+/// # Fields
+///
+/// * `start_time` - Start time of reading, if begun.
+/// * `bytes_read` - Total bytes accumulated.
+/// * `is_finalized` - Whether processing is complete.
+/// * `base` - Base plugin for outputs and options.
+pub struct ReadRateMetaPlugin {
+    start_time: Option<Instant>,
+    bytes_read: u64,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl ReadRateMetaPlugin {
+    /// Creates a new `ReadRateMetaPlugin` instance.
+    ///
+    /// Initializes with default options and outputs, merging provided ones.
+    /// Starts tracking from zero bytes and no start time.
+    ///
+    /// # Arguments
+    ///
+    /// * `_options` - Optional configuration options (merged with defaults; unused specifics here).
+    /// * `outputs` - Optional output mappings (merged with default "read_rate").
+    ///
+    /// # Returns
+    ///
+    /// A new, un-finalized `ReadRateMetaPlugin` instance.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::meta_plugin::{ReadRateMetaPlugin, MetaPlugin};
+    /// let plugin = ReadRateMetaPlugin::new(None, None);
+    /// assert!(!plugin.is_finalized());
+    /// ```
+    pub fn new(
+        _options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> ReadRateMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["read_rate"];
+        base.initialize_plugin(default_outputs, &_options, &outputs);
+
+        ReadRateMetaPlugin {
+            start_time: None,
+            bytes_read: 0,
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for ReadRateMetaPlugin {
+    /// Checks if the plugin has been finalized.
+    ///
+    /// # Returns
+    ///
+    /// `true` if finalized (processing complete), `false` otherwise.
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    /// Sets the finalized state of the plugin.
+    ///
+    /// Marks the plugin as complete or resets it.
+    ///
+    /// # Arguments
+    ///
+    /// * `finalized` - Whether processing is now complete.
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    /// Finalizes the plugin, calculating the read rate.
+    ///
+    /// Computes KB/s from bytes read and elapsed time. Outputs via mappings.
+    /// Idempotent: skips if already finalized.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with rate metadata (if computable) and finalized=true.
+    ///
+    /// # Errors
+    ///
+    /// None; returns empty metadata if no start time or zero duration.
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        if let Some(start_time) = self.start_time {
+            let duration = start_time.elapsed();
+            let rate = if duration.as_secs_f64() > 0.0 {
+                format!(
+                    "{:.2} KB/s",
+                    (self.bytes_read as f64 / 1024.0) / duration.as_secs_f64()
+                )
+            } else {
+                "N/A".to_string()
+            };
+
+            // Use process_metadata_outputs to handle output mapping
+            if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                "read_rate",
+                serde_yaml::Value::String(rate),
+                self.base.outputs(),
+            ) {
+                metadata.push(meta_data);
+            }
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    /// Updates the plugin with new data, accumulating bytes read.
+    ///
+    /// Starts timer on first update if not set. Accumulates byte count.
+    /// Idempotent post-finalize: ignores data.
+    ///
+    /// # Arguments
+    ///
+    /// * `data` - Byte slice to process (length added to total).
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginResponse` with no metadata and finalized=false (unless already done).
+    fn update(&mut self, data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        if self.start_time.is_none() {
+            self.start_time = Some(Instant::now());
+        }
+        self.bytes_read += data.len() as u64;
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginType::ReadRate`.
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::ReadRate
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// Immutable reference to the outputs HashMap.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// Allows modification of output configurations.
+    ///
+    /// # Returns
+    ///
+    /// Mutable reference to the outputs HashMap.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    /// Returns the default output names for this plugin.
+    ///
+    /// # Returns
+    ///
+    /// Vector containing "read_rate".
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["read_rate".to_string()]
+    }
+
+    /// Returns a reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// Immutable reference to the options HashMap.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// Allows modification of plugin options.
+    ///
+    /// # Returns
+    ///
+    /// Mutable reference to the options HashMap.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_read_rate_plugin() {
+    register_meta_plugin(MetaPluginType::ReadRate, |options, outputs| {
+        Box::new(ReadRateMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register ReadRateMetaPlugin");
+}

src/meta_plugin/read_time.rs

@@ -0,0 +1,137 @@
+use std::time::Instant;
+
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+
+#[derive(Debug, Clone, Default)]
+pub struct ReadTimeMetaPlugin {
+    start_time: Option<Instant>,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl ReadTimeMetaPlugin {
+    pub fn new(
+        _options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> ReadTimeMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["read_time"];
+        base.initialize_plugin(default_outputs, &_options, &outputs);
+
+        ReadTimeMetaPlugin {
+            start_time: None,
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for ReadTimeMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        if let Some(start_time) = self.start_time {
+            let duration = start_time.elapsed();
+            let duration_str = format!("{:.3} seconds", duration.as_secs_f64());
+
+            // Use process_metadata_outputs to handle output mapping
+            if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                "read_time",
+                serde_yaml::Value::String(duration_str),
+                self.base.outputs(),
+            ) {
+                metadata.push(meta_data);
+            }
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn update(&mut self, _data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        if self.start_time.is_none() {
+            self.start_time = Some(Instant::now());
+        }
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::ReadTime
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["read_time".to_string()]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_read_time_plugin() {
+    register_meta_plugin(MetaPluginType::ReadTime, |options, outputs| {
+        Box::new(ReadTimeMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register ReadTimeMetaPlugin");
+}

src/meta_plugin/shell.rs

@@ -0,0 +1,253 @@
+use std::env;
+
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+
+#[derive(Debug, Clone, Default)]
+/// Meta plugin for capturing shell environment information.
+///
+/// This plugin retrieves the current shell from the SHELL environment variable
+/// and provides it as metadata. It runs once during initialization and does not
+/// process input data.
+pub struct ShellMetaPlugin {
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl ShellMetaPlugin {
+    /// Creates a new ShellMetaPlugin instance.
+    ///
+    /// Initializes with default outputs and options, overridden by provided values.
+    /// Defaults to "shell" as the output key.
+    ///
+    /// # Arguments
+    ///
+    /// * `_options` - Optional configuration options (unused currently).
+    /// * `outputs` - Optional output mappings to override defaults.
+    ///
+    /// # Returns
+    ///
+    /// * `ShellMetaPlugin` - A new instance with processed options and outputs.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::meta_plugin::ShellMetaPlugin;
+    /// let plugin = ShellMetaPlugin::new(None, None);
+    /// ```
+    pub fn new(
+        _options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> ShellMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["shell"];
+        base.initialize_plugin(default_outputs, &_options, &outputs);
+
+        ShellMetaPlugin {
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for ShellMetaPlugin {
+    /// Checks if the plugin has been finalized.
+    ///
+    /// # Returns
+    ///
+    /// * `bool` - True if finalized, false otherwise.
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    /// Sets the finalized state of the plugin.
+    ///
+    /// # Arguments
+    ///
+    /// * `finalized` - The new finalized state.
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    /// Finalizes the plugin without processing data.
+    ///
+    /// For this plugin, finalization is handled in `initialize`, so this returns empty metadata.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Response with no metadata and finalized state.
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    /// Updates the plugin with data (not used for shell).
+    ///
+    /// Shell plugin doesn't process data streams; returns empty response unless not finalized.
+    ///
+    /// # Arguments
+    ///
+    /// * `_data` - Byte slice of input data (ignored).
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Empty metadata response.
+    fn update(&mut self, _data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginType::Shell` - The shell plugin type.
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Shell
+    }
+
+    /// Initializes the plugin and extracts shell metadata.
+    ///
+    /// Retrieves the SHELL environment variable and adds it to metadata.
+    /// Marks the plugin as finalized after one run.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Response with shell metadata and finalized state.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// # use keep::meta_plugin::{ShellMetaPlugin, MetaPlugin};
+    /// let mut plugin = ShellMetaPlugin::new(None, None);
+    /// let response = plugin.initialize();
+    /// assert!(response.is_finalized);
+    /// ```
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+        let shell = match env::var("SHELL") {
+            Ok(shell) => shell,
+            Err(_) => "unknown".to_string(),
+        };
+
+        // Use process_metadata_outputs to handle output mapping
+        if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+            "shell",
+            serde_yaml::Value::String(shell),
+            self.base.outputs(),
+        ) {
+            metadata.push(meta_data);
+        }
+
+        // Mark as finalized since this plugin only needs to run once
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    /// Returns a reference to the plugin's outputs.
+    ///
+    /// # Returns
+    ///
+    /// * `&HashMap<String, serde_yaml::Value>` - The outputs map.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    /// Returns a mutable reference to the plugin's outputs.
+    ///
+    /// # Returns
+    ///
+    /// * `&mut HashMap<String, serde_yaml::Value>` - Mutable outputs map.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    /// Returns the default output names for this plugin.
+    ///
+    /// # Returns
+    ///
+    /// Vector containing "shell".
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["shell".to_string()]
+    }
+
+    /// Returns a reference to the plugin's options.
+    ///
+    /// # Returns
+    ///
+    /// * `&HashMap<String, serde_yaml::Value>` - The options map.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    /// Returns a mutable reference to the plugin's options.
+    ///
+    /// # Returns
+    ///
+    /// * `&mut HashMap<String, serde_yaml::Value>` - Mutable options map.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+/// Registers the shell meta plugin with the global registry.
+///
+/// This constructor function is called at module load time using ctor crate.
+/// It creates the plugin with provided options and outputs.
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_shell_plugin() {
+    register_meta_plugin(MetaPluginType::Shell, |options, outputs| {
+        Box::new(ShellMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register ShellMetaPlugin");
+}

src/meta_plugin/shell_pid.rs

@@ -0,0 +1,145 @@
+use crate::meta_plugin::{BaseMetaPlugin, MetaPlugin, MetaPluginType};
+use std::env;
+use std::process;
+
+#[derive(Debug, Clone, Default)]
+pub struct ShellPidMetaPlugin {
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl ShellPidMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> ShellPidMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        // Set default outputs
+        let default_outputs = &["shell_pid"];
+        base.initialize_plugin(default_outputs, &options, &outputs);
+
+        ShellPidMetaPlugin {
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for ShellPidMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn finalize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        // Mark as finalized
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: true,
+        }
+    }
+
+    fn update(&mut self, _data: &[u8]) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::ShellPid
+    }
+
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return crate::meta_plugin::MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+        let pid = match env::var("PPID") {
+            Ok(ppid) => ppid,
+            Err(_) => process::id().to_string(),
+        };
+
+        // Use process_metadata_outputs to handle output mapping
+        if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+            "shell_pid",
+            serde_yaml::Value::String(pid),
+            self.base.outputs(),
+        ) {
+            metadata.push(meta_data);
+        }
+
+        // Mark as finalized since this plugin only needs to run once
+        self.is_finalized = true;
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_shell_pid_plugin() {
+    register_meta_plugin(MetaPluginType::ShellPid, |options, outputs| {
+        Box::new(ShellPidMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register ShellPidMetaPlugin");
+}

src/meta_plugin/system.rs

@@ -1,448 +0,0 @@
-use anyhow::Result;
-use gethostname::gethostname;
-use local_ip_address::local_ip;
-use dns_lookup::lookup_addr;
-use std::io;
-use std::io::Write;
-use std::env;
-use std::process;
-use uzers::{get_current_uid, get_current_gid, get_current_username, get_current_groupname};
-
-use crate::common::is_binary;
-use crate::meta_plugin::MetaPlugin;
-
-#[derive(Debug, Clone, Default)]
-pub struct CwdMetaPlugin {
-    meta_name: String,
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct BinaryMetaPlugin {
-    meta_name: String,
-    buffer: Vec<u8>,
-    max_buffer_size: usize,
-}
-
-impl BinaryMetaPlugin {
-    pub fn new() -> BinaryMetaPlugin {
-        BinaryMetaPlugin {
-            meta_name: "binary".to_string(),
-            buffer: Vec::new(),
-            max_buffer_size: 4096, // 4KB
-        }
-    }
-    
-}
-
-impl MetaPlugin for BinaryMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        let is_binary = is_binary(&self.buffer);
-        Ok(if is_binary { "true".to_string() } else { "false".to_string() })
-    }
-
-    fn update(&mut self, data: &[u8]) {
-        // Only collect up to max_buffer_size
-        let remaining_capacity = self.max_buffer_size.saturating_sub(self.buffer.len());
-        if remaining_capacity > 0 {
-            let bytes_to_copy = std::cmp::min(data.len(), remaining_capacity);
-            self.buffer.extend_from_slice(&data[..bytes_to_copy]);
-        }
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-impl CwdMetaPlugin {
-    pub fn new() -> CwdMetaPlugin {
-        CwdMetaPlugin {
-            meta_name: "cwd".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for CwdMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        match env::current_dir() {
-            Ok(path) => Ok(path.to_string_lossy().to_string()),
-            Err(_) => Ok("unknown".to_string()),
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct UidMetaPlugin {
-    meta_name: String,
-}
-
-impl UidMetaPlugin {
-    pub fn new() -> UidMetaPlugin {
-        UidMetaPlugin {
-            meta_name: "uid".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for UidMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        Ok(get_current_uid().to_string())
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct UserMetaPlugin {
-    meta_name: String,
-}
-
-impl UserMetaPlugin {
-    pub fn new() -> UserMetaPlugin {
-        UserMetaPlugin {
-            meta_name: "user".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for UserMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        match get_current_username() {
-            Some(username) => Ok(username.to_string_lossy().to_string()),
-            None => Ok("unknown".to_string()),
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct GidMetaPlugin {
-    meta_name: String,
-}
-
-impl GidMetaPlugin {
-    pub fn new() -> GidMetaPlugin {
-        GidMetaPlugin {
-            meta_name: "gid".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for GidMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        Ok(get_current_gid().to_string())
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct GroupMetaPlugin {
-    meta_name: String,
-}
-
-impl GroupMetaPlugin {
-    pub fn new() -> GroupMetaPlugin {
-        GroupMetaPlugin {
-            meta_name: "group".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for GroupMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        match get_current_groupname() {
-            Some(groupname) => Ok(groupname.to_string_lossy().to_string()),
-            None => Ok("unknown".to_string()),
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct ShellMetaPlugin {
-    meta_name: String,
-}
-
-impl ShellMetaPlugin {
-    pub fn new() -> ShellMetaPlugin {
-        ShellMetaPlugin {
-            meta_name: "shell".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for ShellMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        match env::var("SHELL") {
-            Ok(shell) => Ok(shell),
-            Err(_) => Ok("unknown".to_string()),
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct ShellPidMetaPlugin {
-    meta_name: String,
-}
-
-impl ShellPidMetaPlugin {
-    pub fn new() -> ShellPidMetaPlugin {
-        ShellPidMetaPlugin {
-            meta_name: "shell_pid".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for ShellPidMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        match env::var("PPID") {
-            Ok(ppid) => Ok(ppid),
-            Err(_) => Ok(process::id().to_string()),
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct KeepPidMetaPlugin {
-    meta_name: String,
-}
-
-impl KeepPidMetaPlugin {
-    pub fn new() -> KeepPidMetaPlugin {
-        KeepPidMetaPlugin {
-            meta_name: "keep_pid".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for KeepPidMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        Ok(process::id().to_string())
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct HostnameMetaPlugin {
-    meta_name: String,
-}
-
-impl HostnameMetaPlugin {
-    pub fn new() -> HostnameMetaPlugin {
-        HostnameMetaPlugin {
-            meta_name: "hostname".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for HostnameMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        match gethostname().into_string() {
-            Ok(hostname) => Ok(hostname),
-            Err(_) => Ok("unknown".to_string()),
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed for hostname
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct FullHostnameMetaPlugin {
-    meta_name: String,
-}
-
-impl FullHostnameMetaPlugin {
-    pub fn new() -> FullHostnameMetaPlugin {
-        FullHostnameMetaPlugin {
-            meta_name: "full_hostname".to_string(),
-        }
-    }
-}
-
-impl MetaPlugin for FullHostnameMetaPlugin {
-    fn is_internal(&self) -> bool {
-        true
-    }
-    
-    fn create(&self) -> Result<Box<dyn Write>> {
-        Ok(Box::new(io::sink()))
-    }
-
-    fn finalize(&mut self) -> io::Result<String> {
-        // Try to get the FQDN through reverse DNS lookup
-        match local_ip() {
-            Ok(my_local_ip) => {
-                match lookup_addr(&my_local_ip) {
-                    Ok(hostname) => Ok(hostname),
-                    Err(_) => {
-                        // Fall back to regular hostname if reverse DNS fails
-                        match gethostname().into_string() {
-                            Ok(hostname) => Ok(hostname),
-                            Err(_) => Ok("unknown".to_string()),
-                        }
-                    }
-                }
-            }
-            Err(_) => {
-                // Fall back to regular hostname if we can't get local IP
-                match gethostname().into_string() {
-                    Ok(hostname) => Ok(hostname),
-                    Err(_) => Ok("unknown".to_string()),
-                }
-            }
-        }
-    }
-
-    fn update(&mut self, _data: &[u8]) {
-        // No update needed for full hostname
-    }
-
-    fn meta_name(&mut self) -> String {
-        self.meta_name.clone()
-    }
-}
-

src/meta_plugin/text.rs

@@ -0,0 +1,831 @@
+use crate::common::PIPESIZE;
+use crate::common::is_binary::is_binary;
+use crate::meta_plugin::{MetaPlugin, MetaPluginResponse, MetaPluginType};
+
+#[derive(Debug, Clone)]
+pub struct TextMetaPlugin {
+    buffer: Option<Vec<u8>>,
+    max_buffer_size: usize,
+    is_finalized: bool,
+    word_count: usize,
+    line_count: usize,
+    is_binary_content: Option<bool>,
+    // State for tracking word boundaries across chunks
+    in_word: bool,
+    // Buffer for handling UTF-8 character boundaries
+    utf8_buffer: Vec<u8>,
+    base: crate::meta_plugin::BaseMetaPlugin,
+    // Options to track specific statistics
+    track_word_count: bool,
+    track_line_count: bool,
+    track_line_lengths: bool,
+    // Flags for which line length statistics to output
+    output_line_max_len: bool,
+    output_line_mean_len: bool,
+    output_line_median_len: bool,
+    // For tracking line lengths
+    line_lengths: Option<Vec<usize>>,
+    current_line_length: usize,
+    // For incremental calculation of max and mean
+    max_line_length: usize,
+    total_line_length: usize,
+    line_count_for_stats: usize,
+}
+
+impl TextMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> TextMetaPlugin {
+        let mut base = crate::meta_plugin::BaseMetaPlugin::new();
+
+        // Initialize with helper function
+        base.initialize_plugin(
+            &[
+                "text",
+                "text_word_count",
+                "text_line_count",
+                "text_line_max_len",
+                "text_line_mean_len",
+                "text_line_median_len",
+            ],
+            &options,
+            &outputs,
+        );
+
+        // Set disabled outputs to null based on options
+        let outputs_to_disable = vec![
+            ("text_word_count", "text_word_count"),
+            ("text_line_count", "text_line_count"),
+            ("text_line_max_len", "text_line_max_len"),
+            ("text_line_mean_len", "text_line_mean_len"),
+            ("text_line_median_len", "text_line_median_len"),
+        ];
+
+        for (option_name, output_name) in outputs_to_disable {
+            if let Some(value) = base.options.get(option_name) {
+                // Handle both boolean false and string "false"
+                let should_disable = match value {
+                    serde_yaml::Value::Bool(b) => !b,
+                    serde_yaml::Value::String(s) => s == "false",
+                    _ => false,
+                };
+                if should_disable {
+                    base.outputs
+                        .insert(output_name.to_string(), serde_yaml::Value::Null);
+                }
+            }
+        }
+
+        // Set default options if not provided
+        let default_options = vec![
+            (
+                "text_detect_size",
+                serde_yaml::Value::Number(PIPESIZE.into()),
+            ),
+            ("text_word_count", serde_yaml::Value::Bool(true)),
+            ("text_line_count", serde_yaml::Value::Bool(true)),
+            ("text_line_max_len", serde_yaml::Value::Bool(true)),
+            ("text_line_mean_len", serde_yaml::Value::Bool(true)),
+            ("text_line_median_len", serde_yaml::Value::Bool(false)),
+        ];
+
+        for (key, value) in default_options {
+            if !base.options.contains_key(key) {
+                base.options.insert(key.to_string(), value);
+            }
+        }
+
+        // Get text_detect_size (previously max_buffer_size)
+        let max_buffer_size = base
+            .options
+            .get("text_detect_size")
+            .or_else(|| base.options.get("max_buffer_size")) // Handle backward compatibility
+            .and_then(|v| v.as_u64())
+            .unwrap_or(PIPESIZE as u64) as usize;
+
+        // Get which statistics to track
+        let track_word_count = base
+            .options
+            .get("text_word_count")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+        let track_line_count = base
+            .options
+            .get("text_line_count")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+        let track_line_max_len = base
+            .options
+            .get("text_line_max_len")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+        let track_line_mean_len = base
+            .options
+            .get("text_line_mean_len")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(true);
+        let track_line_median_len = base
+            .options
+            .get("text_line_median_len")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(false);
+
+        // Track line lengths if any of the line length options are enabled
+        let track_line_lengths = track_line_max_len || track_line_mean_len || track_line_median_len;
+
+        TextMetaPlugin {
+            buffer: Some(Vec::new()),
+            max_buffer_size,
+            is_finalized: false,
+            word_count: 0,
+            line_count: 0,
+            is_binary_content: None,
+            in_word: false,
+            utf8_buffer: Vec::new(),
+            base,
+            // Add fields for line length tracking
+            track_word_count,
+            track_line_count,
+            track_line_lengths,
+            // Set output flags
+            output_line_max_len: track_line_max_len,
+            output_line_mean_len: track_line_mean_len,
+            output_line_median_len: track_line_median_len,
+            line_lengths: if track_line_lengths {
+                Some(Vec::new())
+            } else {
+                None
+            },
+            current_line_length: 0,
+            // Initialize incremental tracking for max and mean
+            max_line_length: 0,
+            total_line_length: 0,
+            line_count_for_stats: 0,
+        }
+    }
+
+    /// Count words and lines in a text chunk, handling block boundaries correctly.
+    ///
+    /// Processes UTF-8 data, tracks word transitions, and updates line length statistics.
+    ///
+    /// # Arguments
+    ///
+    /// * `data` - Byte slice of text content.
+    fn count_text_stats(&mut self, data: &[u8]) {
+        // Count lines (newlines) if needed
+        if self.track_line_count {
+            self.line_count += data.iter().filter(|&&b| b == b'\n').count();
+        }
+
+        // Handle UTF-8 character boundaries by combining with any buffered bytes
+        let combined_data = if !self.utf8_buffer.is_empty() {
+            let mut combined = self.utf8_buffer.clone();
+            combined.extend_from_slice(data);
+            combined
+        } else {
+            data.to_vec()
+        };
+
+        // Clear the UTF-8 buffer
+        self.utf8_buffer.clear();
+
+        // Convert to string, handling potential UTF-8 boundaries
+        let text = match std::str::from_utf8(&combined_data) {
+            Ok(text) => text,
+            Err(e) => {
+                // If we have incomplete UTF-8 at the end, buffer those bytes for next chunk
+                let valid_up_to = e.valid_up_to();
+                if valid_up_to < combined_data.len() {
+                    self.utf8_buffer
+                        .extend_from_slice(&combined_data[valid_up_to..]);
+                }
+                match std::str::from_utf8(&combined_data[..valid_up_to]) {
+                    Ok(text) => text,
+                    Err(_) => return, // Can't process this data
+                }
+            }
+        };
+
+        // Count words if needed
+        if self.track_word_count {
+            for ch in text.chars() {
+                let is_whitespace = ch.is_whitespace();
+
+                if !self.in_word && !is_whitespace {
+                    // Transition from whitespace to word - start of new word
+                    self.word_count += 1;
+                    self.in_word = true;
+                } else if self.in_word && is_whitespace {
+                    // Transition from word to whitespace - end of current word
+                    self.in_word = false;
+                }
+            }
+        }
+
+        // Track line lengths if needed
+        if self.track_line_lengths {
+            for ch in text.chars() {
+                if ch == '\n' {
+                    // Update max line length
+                    if self.current_line_length > self.max_line_length {
+                        self.max_line_length = self.current_line_length;
+                    }
+
+                    // Update total for mean calculation
+                    self.total_line_length += self.current_line_length;
+                    self.line_count_for_stats += 1;
+
+                    // Only store individual lengths if median is needed
+                    if let Some(ref mut lengths) = self.line_lengths {
+                        lengths.push(self.current_line_length);
+                    }
+
+                    self.current_line_length = 0;
+                } else {
+                    self.current_line_length += 1;
+                }
+            }
+        }
+    }
+
+    /// Helper method to perform binary detection and return appropriate metadata.
+    ///
+    /// Uses the is_binary function to check the buffer and sets text-related outputs accordingly.
+    ///
+    /// # Arguments
+    ///
+    /// * `buffer` - Data to check for binary content.
+    ///
+    /// # Returns
+    ///
+    /// * `(Vec<MetaData>, bool)` - Metadata updates and whether content is binary.
+    fn perform_binary_detection(
+        &mut self,
+        buffer: &[u8],
+    ) -> (Vec<crate::meta_plugin::MetaData>, bool) {
+        let mut metadata = Vec::new();
+        let is_binary_result = is_binary(buffer);
+        self.is_binary_content = Some(is_binary_result);
+
+        // Output text status
+        let text_value = if is_binary_result {
+            "false".to_string()
+        } else {
+            "true".to_string()
+        };
+
+        // Use process_metadata_outputs to handle output mapping
+        if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+            "text",
+            serde_yaml::Value::String(text_value),
+            self.base.outputs(),
+        ) {
+            metadata.push(meta_data);
+        }
+
+        // If content is binary, set all text-related outputs to None
+        if is_binary_result {
+            let text_outputs = vec![
+                "text_word_count",
+                "text_line_count",
+                "text_line_max_len",
+                "text_line_mean_len",
+                "text_line_median_len",
+            ];
+            for output_name in text_outputs {
+                if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                    output_name,
+                    serde_yaml::Value::Null,
+                    self.base.outputs(),
+                ) {
+                    metadata.push(meta_data);
+                }
+            }
+        }
+
+        (metadata, is_binary_result)
+    }
+
+    /// Helper method to process the remaining UTF-8 buffer and finalize text statistics.
+    ///
+    /// Calls count_text_stats with empty data to handle any pending UTF-8 bytes.
+    fn process_remaining_utf8_buffer(&mut self) {
+        if !self.utf8_buffer.is_empty() {
+            self.count_text_stats(&[]);
+        }
+    }
+
+    /// Helper method to handle the last line when tracking line lengths.
+    ///
+    /// Updates statistics for any unfinished line at EOF.
+    fn handle_last_line_for_length_tracking(&mut self) {
+        if self.track_line_lengths && self.current_line_length > 0 {
+            // Update max line length for the last line
+            if self.current_line_length > self.max_line_length {
+                self.max_line_length = self.current_line_length;
+            }
+
+            // Update total for mean calculation for the last line
+            self.total_line_length += self.current_line_length;
+            self.line_count_for_stats += 1;
+
+            // Only store individual lengths if median is needed
+            if let Some(ref mut lengths) = self.line_lengths {
+                lengths.push(self.current_line_length);
+            }
+        }
+    }
+
+    /// Helper method to output word count metadata.
+    ///
+    /// # Returns
+    ///
+    /// * `Option<MetaData>` - Metadata entry if tracking is enabled.
+    fn output_word_count_metadata(&self) -> Option<crate::meta_plugin::MetaData> {
+        if self.track_word_count {
+            crate::meta_plugin::process_metadata_outputs(
+                "text_word_count",
+                serde_yaml::Value::String(self.word_count.to_string()),
+                self.base.outputs(),
+            )
+        } else {
+            None
+        }
+    }
+
+    /// Helper method to output line count metadata.
+    ///
+    /// # Returns
+    ///
+    /// * `Option<MetaData>` - Metadata entry if tracking is enabled.
+    fn output_line_count_metadata(&self) -> Option<crate::meta_plugin::MetaData> {
+        if self.track_line_count {
+            crate::meta_plugin::process_metadata_outputs(
+                "text_line_count",
+                serde_yaml::Value::String(self.line_count.to_string()),
+                self.base.outputs(),
+            )
+        } else {
+            None
+        }
+    }
+
+    /// Helper method to output max line length metadata.
+    ///
+    /// # Returns
+    ///
+    /// * `Option<MetaData>` - Metadata entry if enabled and data exists.
+    fn output_max_line_length_metadata(&self) -> Option<crate::meta_plugin::MetaData> {
+        if self.output_line_max_len && self.line_count_for_stats > 0 {
+            crate::meta_plugin::process_metadata_outputs(
+                "text_line_max_len",
+                serde_yaml::Value::String(self.max_line_length.to_string()),
+                self.base.outputs(),
+            )
+        } else {
+            None
+        }
+    }
+
+    /// Helper method to output mean line length metadata.
+    ///
+    /// Computes average line length and rounds to nearest integer.
+    ///
+    /// # Returns
+    ///
+    /// * `Option<MetaData>` - Metadata entry if enabled and data exists.
+    fn output_mean_line_length_metadata(&self) -> Option<crate::meta_plugin::MetaData> {
+        if self.output_line_mean_len && self.line_count_for_stats > 0 {
+            let mean_len = self.total_line_length as f64 / self.line_count_for_stats as f64;
+            // Round to nearest integer
+            let mean_len_int = mean_len.round() as usize;
+            crate::meta_plugin::process_metadata_outputs(
+                "text_line_mean_len",
+                serde_yaml::Value::String(mean_len_int.to_string()),
+                self.base.outputs(),
+            )
+        } else {
+            None
+        }
+    }
+
+    /// Helper method to output median line length metadata.
+    ///
+    /// Sorts line lengths and computes median (average of middle two for even count).
+    ///
+    /// # Returns
+    ///
+    /// * `Option<MetaData>` - Metadata entry if enabled and data exists.
+    fn output_median_line_length_metadata(&self) -> Option<crate::meta_plugin::MetaData> {
+        if self.output_line_median_len
+            && let Some(lengths) = &self.line_lengths
+            && !lengths.is_empty()
+        {
+            let mut sorted_lengths = lengths.clone();
+            sorted_lengths.sort();
+            let median_len = if lengths.len() % 2 == 0 {
+                (sorted_lengths[lengths.len() / 2 - 1] + sorted_lengths[lengths.len() / 2]) as f64
+                    / 2.0
+            } else {
+                sorted_lengths[lengths.len() / 2] as f64
+            };
+
+            return crate::meta_plugin::process_metadata_outputs(
+                "text_line_median_len",
+                serde_yaml::Value::String(median_len.to_string()),
+                self.base.outputs(),
+            );
+        }
+        None
+    }
+
+    /// Helper method to output word and line counts.
+    ///
+    /// Finalizes pending data and collects all enabled text statistics metadata.
+    ///
+    /// # Returns
+    ///
+    /// * `Vec<MetaData>` - List of metadata entries.
+    fn output_word_line_counts(&mut self) -> Vec<crate::meta_plugin::MetaData> {
+        // Process any remaining data in utf8_buffer
+        self.process_remaining_utf8_buffer();
+
+        // Handle the last line if tracking line lengths
+        self.handle_last_line_for_length_tracking();
+
+        // Collect all metadata outputs
+        let mut metadata = Vec::new();
+
+        // Add metadata outputs using a more concise approach
+        let outputs_to_check = vec![
+            (self.output_word_count_metadata(), "word count"),
+            (self.output_line_count_metadata(), "line count"),
+        ];
+
+        for (output, _) in outputs_to_check {
+            if let Some(meta_data) = output {
+                metadata.push(meta_data);
+            }
+        }
+
+        // Output line length statistics if tracked
+        if self.track_line_lengths && self.line_count_for_stats > 0 {
+            let line_stats_outputs = vec![
+                (self.output_max_line_length_metadata(), "max line length"),
+                (self.output_mean_line_length_metadata(), "mean line length"),
+                (
+                    self.output_median_line_length_metadata(),
+                    "median line length",
+                ),
+            ];
+
+            for (output, _) in line_stats_outputs {
+                if let Some(meta_data) = output {
+                    metadata.push(meta_data);
+                }
+            }
+        }
+
+        metadata
+    }
+}
+
+impl MetaPlugin for TextMetaPlugin {
+    /// Checks if the plugin has been finalized.
+    ///
+    /// # Returns
+    ///
+    /// `true` if finalized, `false` otherwise.
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    /// Sets the finalized state of the plugin.
+    ///
+    /// # Arguments
+    ///
+    /// * `finalized` - The new finalized state.
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    /// Updates the plugin with new data chunk.
+    ///
+    /// Accumulates data for binary detection (if pending) or text statistics.
+    /// Finalizes early if binary content is detected.
+    ///
+    /// # Arguments
+    ///
+    /// * `data` - Byte slice of content chunk.
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Current metadata and finalized status.
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        // If already finalized, don't process more data
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        // If we haven't determined if content is binary yet, build buffer and check
+        if self.is_binary_content.is_none() {
+            let should_finalize = if let Some(ref mut buffer) = self.buffer {
+                // Add data to our buffer up to max_buffer_size
+                let remaining_capacity = self.max_buffer_size.saturating_sub(buffer.len());
+                let bytes_to_take = std::cmp::min(data.len(), remaining_capacity);
+                buffer.extend_from_slice(&data[..bytes_to_take]);
+
+                // If we have enough data to make a binary determination, do it now
+                let buffer_len = buffer.len();
+                if buffer_len >= std::cmp::min(1024, self.max_buffer_size) {
+                    // Clone the buffer data for binary detection to avoid borrowing conflicts
+                    let buffer_clone = buffer.clone();
+                    let (binary_metadata, is_binary) = self.perform_binary_detection(&buffer_clone);
+                    metadata.extend(binary_metadata);
+                    self.is_binary_content = Some(is_binary);
+
+                    // If it's binary, we're done with this plugin
+                    if is_binary {
+                        self.buffer = None; // Drop the buffer
+                        self.is_finalized = true;
+                        return MetaPluginResponse {
+                            metadata,
+                            is_finalized: true,
+                        };
+                    }
+
+                    // If it's text, count words and lines for this chunk
+                    self.count_text_stats(&data[..bytes_to_take]);
+
+                    // If we've reached our buffer limit, drop the buffer to save memory
+                    // But don't finalize yet - we need to keep counting words and lines
+                    if buffer_len >= self.max_buffer_size {
+                        self.buffer = None; // Drop the buffer
+                    }
+                    false // Never finalize here for text content
+                } else {
+                    // Still building up buffer, count words and lines for this chunk
+                    self.count_text_stats(&data[..bytes_to_take]);
+                    false
+                }
+            } else {
+                false
+            };
+
+            if should_finalize {
+                return MetaPluginResponse {
+                    metadata,
+                    is_finalized: true,
+                };
+            }
+        } else if self.is_binary_content == Some(false) {
+            // We've already determined it's text, just count words and lines
+            self.count_text_stats(data);
+        }
+        // If is_binary_content == Some(true), we should have already finalized, but just in case:
+        else if self.is_binary_content == Some(true) {
+            self.is_finalized = true;
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        MetaPluginResponse {
+            metadata,
+            is_finalized: self.is_finalized,
+        }
+    }
+
+    /// Finalizes the plugin and emits all pending text statistics.
+    ///
+    /// Performs binary detection if not done, then outputs enabled statistics.
+    /// Handles head/tail options for content preview (future implementation).
+    ///
+    /// # Returns
+    ///
+    /// * `MetaPluginResponse` - Final metadata and finalized status.
+    fn finalize(&mut self) -> MetaPluginResponse {
+        // If already finalized, don't process again
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        // Check if we have head/tail options
+        let head_bytes = self
+            .base
+            .options
+            .get("head_bytes")
+            .and_then(|v| v.as_u64())
+            .map(|v| v as usize);
+        let head_lines = self
+            .base
+            .options
+            .get("head_lines")
+            .and_then(|v| v.as_u64())
+            .map(|v| v as usize);
+        let tail_bytes = self
+            .base
+            .options
+            .get("tail_bytes")
+            .and_then(|v| v.as_u64())
+            .map(|v| v as usize);
+        let tail_lines = self
+            .base
+            .options
+            .get("tail_lines")
+            .and_then(|v| v.as_u64())
+            .map(|v| v as usize);
+
+        // If we haven't determined binary status yet, do it now with whatever we have
+        if self.is_binary_content.is_none()
+            && let Some(buffer) = &self.buffer
+            && !buffer.is_empty()
+        {
+            let buffer = if head_bytes.is_some()
+                || head_lines.is_some()
+                || tail_bytes.is_some()
+                || tail_lines.is_some()
+            {
+                // Build filter string from individual parameters
+                let mut filter_parts = Vec::new();
+                if let Some(bytes) = head_bytes {
+                    filter_parts.push(format!("head_bytes({bytes})"));
+                }
+                if let Some(lines) = head_lines {
+                    filter_parts.push(format!("head_lines({lines})"));
+                }
+                if let Some(bytes) = tail_bytes {
+                    filter_parts.push(format!("tail_bytes({bytes})"));
+                }
+                if let Some(lines) = tail_lines {
+                    filter_parts.push(format!("tail_lines({lines})"));
+                }
+
+                // Apply filters if any are specified
+                let filter_string = filter_parts.join(",");
+                match crate::services::FilterService::new()
+                    .process_with_filter(buffer, Some(&filter_string))
+                {
+                    Ok(filtered) => filtered,
+                    Err(e) => {
+                        log::warn!("Failed to apply filters: {e}");
+                        buffer.clone()
+                    }
+                }
+            } else {
+                buffer.clone()
+            };
+
+            // Clone the processed buffer data for binary detection
+            let (binary_metadata, is_binary) = self.perform_binary_detection(&buffer);
+            metadata.extend(binary_metadata);
+            self.is_binary_content = Some(is_binary);
+
+            // If it's binary, we're done
+            if is_binary {
+                self.buffer = None; // Drop the buffer
+                self.is_finalized = true;
+                // Set all text-related outputs to None since content is binary
+                // Only include outputs that are enabled in the configuration
+                let text_outputs = vec![
+                    ("text_word_count", self.track_word_count),
+                    ("text_line_count", self.track_line_count),
+                    ("text_line_max_len", self.output_line_max_len),
+                    ("text_line_mean_len", self.output_line_mean_len),
+                    ("text_line_median_len", self.output_line_median_len),
+                ];
+
+                for (output_name, is_enabled) in text_outputs {
+                    if is_enabled
+                        && let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                            output_name,
+                            serde_yaml::Value::Null,
+                            self.base.outputs(),
+                        )
+                    {
+                        metadata.push(meta_data);
+                    }
+                }
+                return MetaPluginResponse {
+                    metadata,
+                    is_finalized: true,
+                };
+            }
+        }
+
+        // If content is text, output word and line counts
+        if self.is_binary_content == Some(false) {
+            let word_line_metadata = self.output_word_line_counts();
+            metadata.extend(word_line_metadata);
+        }
+
+        // Only include outputs that are enabled in the configuration
+        // Disabled outputs should not be emitted at all (not even as null)
+        // So we don't need to add anything for disabled outputs
+
+        // Drop the buffer since we're done with it
+        self.buffer = None;
+
+        // Mark as finalized
+        self.is_finalized = true;
+        MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginType::Text`.
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Text
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of outputs.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of outputs.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    /// Returns the default output names for this plugin.
+    ///
+    /// # Returns
+    ///
+    /// Vector of default output field names.
+    fn default_outputs(&self) -> Vec<String> {
+        vec![
+            "text".to_string(),
+            "text_word_count".to_string(),
+            "text_line_count".to_string(),
+            "text_line_max_len".to_string(),
+            "text_line_mean_len".to_string(),
+            "text_line_median_len".to_string(),
+        ]
+    }
+
+    /// Returns a reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of outputs.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of outputs.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_text_plugin() {
+    register_meta_plugin(MetaPluginType::Text, |options, outputs| {
+        Box::new(TextMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register TextMetaPlugin");
+}

src/meta_plugin/tokens.rs

@@ -0,0 +1,325 @@
+use crate::common::PIPESIZE;
+use crate::common::is_binary::is_binary;
+use crate::meta_plugin::{MetaPlugin, MetaPluginResponse, MetaPluginType};
+use crate::tokenizer::{TokenEncoding, get_tokenizer};
+
+#[derive(Debug, Clone)]
+pub struct TokensMetaPlugin {
+    /// Buffer for binary detection (up to PIPESIZE bytes).
+    buffer: Option<Vec<u8>>,
+    max_buffer_size: usize,
+    is_finalized: bool,
+    is_binary_content: Option<bool>,
+    /// Running token count accumulated across chunks.
+    token_count: usize,
+    /// UTF-8 boundary carry buffer.
+    utf8_buffer: Vec<u8>,
+    base: crate::meta_plugin::BaseMetaPlugin,
+    /// The tokenizer encoding.
+    encoding: TokenEncoding,
+}
+
+impl TokensMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> Self {
+        let mut base = crate::meta_plugin::BaseMetaPlugin::new();
+
+        base.initialize_plugin(&["token_count"], &options, &outputs);
+
+        // Set default options
+        let default_options = vec![
+            (
+                "token_detect_size",
+                serde_yaml::Value::Number(PIPESIZE.into()),
+            ),
+            (
+                "encoding",
+                serde_yaml::Value::String("cl100k_base".to_string()),
+            ),
+        ];
+
+        for (key, value) in default_options {
+            if !base.options.contains_key(key) {
+                base.options.insert(key.to_string(), value);
+            }
+        }
+
+        let max_buffer_size = base
+            .options
+            .get("token_detect_size")
+            .and_then(|v| v.as_u64())
+            .unwrap_or(PIPESIZE as u64) as usize;
+
+        let encoding = base
+            .options
+            .get("encoding")
+            .and_then(|v| v.as_str())
+            .and_then(|s| s.parse::<TokenEncoding>().ok())
+            .unwrap_or_default();
+
+        Self {
+            buffer: Some(Vec::new()),
+            max_buffer_size,
+            is_finalized: false,
+            is_binary_content: None,
+            token_count: 0,
+            utf8_buffer: Vec::new(),
+            base,
+            encoding,
+        }
+    }
+
+    /// Tokenize a byte chunk, handling UTF-8 boundaries.
+    ///
+    /// Combines with any pending UTF-8 carry bytes, converts to text,
+    /// and adds the token count to the running total.
+    ///
+    /// Avoids unnecessary allocations when there is no pending UTF-8 carry
+    /// and the data is valid UTF-8.
+    fn count_tokens(&mut self, data: &[u8]) {
+        if data.is_empty() && self.utf8_buffer.is_empty() {
+            return;
+        }
+
+        let tokenizer = get_tokenizer(self.encoding);
+
+        if self.utf8_buffer.is_empty() {
+            // Fast path: no pending carry — try to use data directly
+            match std::str::from_utf8(data) {
+                Ok(text) => {
+                    if !text.is_empty() {
+                        self.token_count += tokenizer.count(text);
+                    }
+                    return;
+                }
+                Err(e) => {
+                    let valid_up_to = e.valid_up_to();
+                    if valid_up_to > 0 {
+                        // Count the valid prefix without copying
+                        let text =
+                            std::str::from_utf8(&data[..valid_up_to]).expect("validated prefix");
+                        self.token_count += tokenizer.count(text);
+                    }
+                    // Save invalid trailing bytes for next call
+                    self.utf8_buffer.extend_from_slice(&data[valid_up_to..]);
+                    return;
+                }
+            }
+        }
+
+        // Slow path: pending carry bytes — must build combined buffer
+        let mut combined = std::mem::take(&mut self.utf8_buffer);
+        combined.extend_from_slice(data);
+
+        match std::str::from_utf8(&combined) {
+            Ok(text) => {
+                if !text.is_empty() {
+                    self.token_count += tokenizer.count(text);
+                }
+            }
+            Err(e) => {
+                let valid_up_to = e.valid_up_to();
+                if valid_up_to > 0 {
+                    let text =
+                        std::str::from_utf8(&combined[..valid_up_to]).expect("validated prefix");
+                    self.token_count += tokenizer.count(text);
+                }
+                self.utf8_buffer.extend_from_slice(&combined[valid_up_to..]);
+            }
+        }
+    }
+
+    /// Perform binary detection on the buffer.
+    fn detect_binary(&mut self, buffer: &[u8]) -> bool {
+        let result = is_binary(buffer);
+        self.is_binary_content = Some(result);
+        result
+    }
+}
+
+impl MetaPlugin for TokensMetaPlugin {
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        if self.is_binary_content.is_none() {
+            // Add data to the buffer
+            let should_detect = if let Some(ref mut buffer) = self.buffer {
+                let remaining = self.max_buffer_size.saturating_sub(buffer.len());
+                let to_take = std::cmp::min(data.len(), remaining);
+                buffer.extend_from_slice(&data[..to_take]);
+                buffer.len() >= std::cmp::min(1024, self.max_buffer_size)
+            } else {
+                false
+            };
+
+            if should_detect {
+                let buffer_data = self.buffer.as_ref().unwrap().clone();
+                let is_binary = self.detect_binary(&buffer_data);
+
+                if is_binary {
+                    if let Some(md) = crate::meta_plugin::process_metadata_outputs(
+                        "token_count",
+                        serde_yaml::Value::Null,
+                        self.base.outputs(),
+                    ) {
+                        metadata.push(md);
+                    }
+                    self.buffer = None;
+                    self.is_finalized = true;
+                    return MetaPluginResponse {
+                        metadata,
+                        is_finalized: true,
+                    };
+                }
+
+                // It's text — tokenize the full buffer (nothing was counted yet),
+                // then clear to avoid double-counting in finalize().
+                self.count_tokens(&buffer_data);
+                self.buffer = Some(Vec::new());
+            }
+        } else if self.is_binary_content == Some(false) {
+            self.count_tokens(data);
+        } else if self.is_binary_content == Some(true) {
+            self.is_finalized = true;
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        MetaPluginResponse {
+            metadata,
+            is_finalized: self.is_finalized,
+        }
+    }
+
+    fn finalize(&mut self) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mut metadata = Vec::new();
+
+        // If binary detection hasn't completed, do it now
+        if self.is_binary_content.is_none()
+            && let Some(buffer) = &self.buffer
+            && !buffer.is_empty()
+        {
+            let buffer_data = buffer.clone();
+            let is_binary = self.detect_binary(&buffer_data);
+
+            if is_binary {
+                if let Some(md) = crate::meta_plugin::process_metadata_outputs(
+                    "token_count",
+                    serde_yaml::Value::Null,
+                    self.base.outputs(),
+                ) {
+                    metadata.push(md);
+                }
+                self.buffer = None;
+                self.is_finalized = true;
+                return MetaPluginResponse {
+                    metadata,
+                    is_finalized: true,
+                };
+            }
+        }
+
+        // Tokenize any bytes in the buffer
+        if let Some(buffer) = &self.buffer {
+            let data = buffer.clone();
+            self.count_tokens(&data);
+        }
+
+        // Process any remaining UTF-8 bytes
+        if !self.utf8_buffer.is_empty() {
+            self.count_tokens(&[]);
+        }
+
+        // Emit token count
+        if let Some(md) = crate::meta_plugin::process_metadata_outputs(
+            "token_count",
+            serde_yaml::Value::String(self.token_count.to_string()),
+            self.base.outputs(),
+        ) {
+            metadata.push(md);
+        }
+
+        self.buffer = None;
+        self.is_finalized = true;
+        MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::Tokens
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["token_count".to_string()]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+
+    fn parallel_safe(&self) -> bool {
+        true
+    }
+}
+
+use crate::meta_plugin::register_meta_plugin;
+
+#[ctor::ctor]
+fn register_tokens_plugin() {
+    register_meta_plugin(MetaPluginType::Tokens, |options, outputs| {
+        Box::new(TokensMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register TokensMetaPlugin");
+}

src/meta_plugin/tree_magic_mini.rs

@@ -0,0 +1,173 @@
+use crate::common::PIPESIZE;
+use crate::meta_plugin::{
+    BaseMetaPlugin, MetaPlugin, MetaPluginResponse, MetaPluginType, process_metadata_outputs,
+    register_meta_plugin,
+};
+
+#[derive(Debug, Default)]
+pub struct TreeMagicMiniMetaPlugin {
+    buffer: Vec<u8>,
+    max_buffer_size: usize,
+    is_finalized: bool,
+    base: BaseMetaPlugin,
+}
+
+impl TreeMagicMiniMetaPlugin {
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> TreeMagicMiniMetaPlugin {
+        let mut base = BaseMetaPlugin::new();
+
+        if let Some(opts) = options {
+            for (key, value) in opts {
+                base.options.insert(key, value);
+            }
+        }
+
+        let max_buffer_size = base
+            .options
+            .get("max_buffer_size")
+            .and_then(|v| v.as_u64())
+            .unwrap_or(PIPESIZE as u64) as usize;
+
+        base.outputs.insert(
+            "tree_magic_mime_type".to_string(),
+            serde_yaml::Value::String("tree_magic_mime_type".to_string()),
+        );
+
+        if let Some(outs) = outputs {
+            for (key, value) in outs {
+                base.outputs.insert(key, value);
+            }
+        }
+
+        TreeMagicMiniMetaPlugin {
+            buffer: Vec::new(),
+            max_buffer_size,
+            is_finalized: false,
+            base,
+        }
+    }
+}
+
+impl MetaPlugin for TreeMagicMiniMetaPlugin {
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::TreeMagicMini
+    }
+
+    fn is_finalized(&self) -> bool {
+        self.is_finalized
+    }
+
+    fn set_finalized(&mut self, finalized: bool) {
+        self.is_finalized = finalized;
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    fn update(&mut self, data: &[u8]) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let remaining = self.max_buffer_size.saturating_sub(self.buffer.len());
+        let to_add = &data[..data.len().min(remaining)];
+        self.buffer.extend_from_slice(to_add);
+
+        if self.buffer.len() >= self.max_buffer_size {
+            let mime_type = tree_magic_mini::from_u8(&self.buffer);
+
+            self.is_finalized = true;
+
+            let metadata = process_metadata_outputs(
+                "tree_magic_mime_type",
+                serde_yaml::Value::String(mime_type.to_string()),
+                self.base.outputs(),
+            )
+            .map(|m| vec![m])
+            .unwrap_or_default();
+
+            return MetaPluginResponse {
+                metadata,
+                is_finalized: true,
+            };
+        }
+
+        MetaPluginResponse {
+            metadata: Vec::new(),
+            is_finalized: false,
+        }
+    }
+
+    fn finalize(&mut self) -> MetaPluginResponse {
+        if self.is_finalized {
+            return MetaPluginResponse {
+                metadata: Vec::new(),
+                is_finalized: true,
+            };
+        }
+
+        let mime_type = tree_magic_mini::from_u8(&self.buffer);
+
+        self.is_finalized = true;
+
+        let metadata = process_metadata_outputs(
+            "tree_magic_mime_type",
+            serde_yaml::Value::String(mime_type.to_string()),
+            self.base.outputs(),
+        )
+        .map(|m| vec![m])
+        .unwrap_or_default();
+
+        MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    fn default_outputs(&self) -> Vec<String> {
+        vec!["tree_magic_mime_type".to_string()]
+    }
+
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+
+    fn parallel_safe(&self) -> bool {
+        true
+    }
+}
+
+#[ctor::ctor]
+fn register_tree_magic_mini_plugin() {
+    register_meta_plugin(MetaPluginType::TreeMagicMini, |options, outputs| {
+        Box::new(TreeMagicMiniMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register TreeMagicMiniMetaPlugin");
+}

src/meta_plugin/user.rs

@@ -0,0 +1,179 @@
+use crate::meta_plugin::{MetaPlugin, MetaPluginType};
+
+#[derive(Debug, Clone, Default)]
+/// Meta plugin for capturing current user and group information.
+///
+/// This plugin collects user ID, group ID, username, and group name for the process
+/// running the keep application, providing context about the creator of items.
+pub struct UserMetaPlugin {
+    base: crate::meta_plugin::BaseMetaPlugin,
+}
+
+impl UserMetaPlugin {
+    /// Creates a new `UserMetaPlugin` instance.
+    ///
+    /// # Arguments
+    ///
+    /// * `options` - Optional configuration options for the plugin.
+    /// * `outputs` - Optional output mappings for metadata.
+    ///
+    /// # Returns
+    ///
+    /// A new instance of `UserMetaPlugin`.
+    pub fn new(
+        options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+        outputs: Option<std::collections::HashMap<String, serde_yaml::Value>>,
+    ) -> UserMetaPlugin {
+        let mut base = crate::meta_plugin::BaseMetaPlugin::new();
+
+        // Initialize with helper function
+        base.initialize_plugin(
+            &["user_uid", "user_gid", "user_name", "user_group"],
+            &options,
+            &outputs,
+        );
+
+        UserMetaPlugin { base }
+    }
+
+    /// Gets the current username.
+    ///
+    /// # Returns
+    ///
+    /// An `Option<String>` with the username, or `None` if unavailable.
+    fn get_current_username() -> Option<String> {
+        uzers::get_user_by_uid(uzers::get_current_uid())
+            .map(|user| user.name().to_string_lossy().to_string())
+    }
+
+    /// Gets the current group name.
+    ///
+    /// # Returns
+    ///
+    /// An `Option<String>` with the group name, or `None` if unavailable.
+    fn get_current_groupname() -> Option<String> {
+        uzers::get_group_by_gid(uzers::get_current_gid())
+            .map(|group| group.name().to_string_lossy().to_string())
+    }
+}
+
+impl MetaPlugin for UserMetaPlugin {
+    /// Initializes the plugin, capturing user information.
+    ///
+    /// # Returns
+    ///
+    /// A `MetaPluginResponse` with user metadata and `is_finalized` set to `true`.
+    fn initialize(&mut self) -> crate::meta_plugin::MetaPluginResponse {
+        let mut metadata = Vec::new();
+
+        // Get user info
+        let uid = uzers::get_current_uid().to_string();
+        let gid = uzers::get_current_gid().to_string();
+        let username = Self::get_current_username().unwrap_or_else(|| "unknown".to_string());
+        let groupname = Self::get_current_groupname().unwrap_or_else(|| "unknown".to_string());
+
+        // Process each output
+        let values = [
+            ("user_uid", uid),
+            ("user_gid", gid),
+            ("user_name", username),
+            ("user_group", groupname),
+        ];
+
+        for (name, value) in values {
+            if let Some(meta_data) = crate::meta_plugin::process_metadata_outputs(
+                name,
+                serde_yaml::Value::String(value),
+                self.base.outputs(),
+            ) {
+                metadata.push(meta_data);
+            }
+        }
+
+        crate::meta_plugin::MetaPluginResponse {
+            metadata,
+            is_finalized: true,
+        }
+    }
+
+    /// Returns the type of this meta plugin.
+    ///
+    /// # Returns
+    ///
+    /// `MetaPluginType::User`.
+    fn meta_type(&self) -> MetaPluginType {
+        MetaPluginType::User
+    }
+
+    fn set_save_meta(&mut self, save_meta: crate::meta_plugin::SaveMetaFn) {
+        self.base.set_save_meta(save_meta);
+    }
+
+    fn save_meta(&self, name: &str, value: &str) {
+        self.base.save_meta(name, value);
+    }
+
+    /// Returns a reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of outputs.
+    fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.outputs()
+    }
+
+    /// Returns a mutable reference to the outputs mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of outputs.
+    fn outputs_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.outputs_mut())
+    }
+
+    /// Returns the default output names.
+    ///
+    /// # Returns
+    ///
+    /// A vector of default output names.
+    fn default_outputs(&self) -> Vec<String> {
+        vec![
+            "user_uid".to_string(),
+            "user_gid".to_string(),
+            "user_name".to_string(),
+            "user_group".to_string(),
+        ]
+    }
+
+    /// Returns a reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A reference to the `HashMap` of options.
+    fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
+        self.base.options()
+    }
+
+    /// Returns a mutable reference to the options mapping.
+    ///
+    /// # Returns
+    ///
+    /// A mutable reference to the `HashMap` of options.
+    fn options_mut(
+        &mut self,
+    ) -> anyhow::Result<&mut std::collections::HashMap<String, serde_yaml::Value>> {
+        Ok(self.base.options_mut())
+    }
+}
+use crate::meta_plugin::register_meta_plugin;
+
+// Register the plugin at module initialization time
+#[ctor::ctor]
+fn register_user_plugin() {
+    register_meta_plugin(MetaPluginType::User, |options, outputs| {
+        Box::new(UserMetaPlugin::new(options, outputs))
+    })
+    .expect("Failed to register UserMetaPlugin");
+}

src/modes/client/delete.rs

@@ -0,0 +1,21 @@
+use crate::client::KeepClient;
+use clap::Command;
+use log::debug;
+
+pub fn mode(
+    client: &KeepClient,
+    _cmd: &mut Command,
+    settings: &crate::config::Settings,
+    ids: &[i64],
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_DELETE: Deleting items via remote server");
+
+    for &id in ids {
+        client.delete_item(id)?;
+        if !settings.quiet {
+            eprintln!("Deleted item {id}");
+        }
+    }
+
+    Ok(())
+}

src/modes/client/diff.rs

@@ -0,0 +1,24 @@
+use crate::client::KeepClient;
+use clap::Command;
+use log::debug;
+
+pub fn mode(
+    client: &KeepClient,
+    _cmd: &mut Command,
+    _settings: &crate::config::Settings,
+    ids: &[i64],
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_DIFF: Getting diff via remote server");
+
+    if ids.len() != 2 {
+        return Err(anyhow::anyhow!("Diff requires exactly 2 item IDs"));
+    }
+
+    let diff_lines = client.diff_items(ids[0], ids[1])?;
+
+    for line in &diff_lines {
+        println!("{line}");
+    }
+
+    Ok(())
+}

src/modes/client/export.rs

@@ -0,0 +1,77 @@
+use anyhow::{Context, Result, anyhow};
+use chrono::Utc;
+use clap::Command;
+use log::debug;
+use std::collections::HashMap;
+use std::fs;
+
+use crate::client::KeepClient;
+use crate::common::sanitize_ts_string;
+use crate::config;
+
+/// Export items to a `.keep.tar` archive via client.
+///
+/// Sends a request to the server's `/api/export` endpoint and
+/// streams the response to a local tar file.
+pub fn mode(
+    client: &KeepClient,
+    cmd: &mut Command,
+    settings: &config::Settings,
+    ids: &[i64],
+    tags: &[String],
+) -> Result<()> {
+    // Validate: IDs XOR tags
+    if !ids.is_empty() && !tags.is_empty() {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "Cannot use both IDs and tags with --export",
+        )
+        .exit();
+    }
+    if ids.is_empty() && tags.is_empty() {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "Must provide either IDs or tags with --export",
+        )
+        .exit();
+    }
+
+    // We need to resolve items on the server to compute the filename.
+    // First, get the item info to build the filename template variables.
+    // For the tar filename, we use {name}_{ts}.keep.tar where name comes from
+    // --export-name or default export_<common-tags>.
+    let dir_name = if let Some(ref name) = settings.export_name {
+        name.clone()
+    } else {
+        "export".to_string()
+    };
+
+    let now = Utc::now();
+    let ts_str = sanitize_ts_string(&now.format("%Y-%m-%dT%H:%M:%SZ").to_string());
+
+    let mut vars = HashMap::new();
+    vars.insert("name".to_string(), dir_name);
+    vars.insert("ts".to_string(), ts_str);
+
+    let basename = strfmt::strfmt(&settings.export_filename_format, &vars).map_err(|e| {
+        anyhow!(
+            "Invalid export filename format '{}': {}",
+            settings.export_filename_format,
+            e
+        )
+    })?;
+
+    let tar_filename = format!("{basename}.keep.tar");
+
+    client
+        .export_items_to_file(ids, tags, std::path::Path::new(&tar_filename))
+        .map_err(|e| anyhow!("Export failed: {e}"))?;
+
+    if !settings.quiet {
+        eprintln!("{tar_filename}");
+    }
+
+    debug!("CLIENT_EXPORT: Wrote items to {tar_filename}");
+
+    Ok(())
+}

src/modes/client/get.rs

@@ -0,0 +1,75 @@
+use crate::client::KeepClient;
+use crate::compression_engine::CompressionType;
+use crate::filter_plugin::FilterChain;
+use crate::modes::common::{check_binary_tty, resolve_item_id};
+use crate::services::compression_service::CompressionService;
+use anyhow::Result;
+use clap::Command;
+use log::debug;
+use std::io::{Read, Write};
+use std::str::FromStr;
+
+pub fn mode(
+    client: &KeepClient,
+    cmd: &mut Command,
+    settings: &crate::config::Settings,
+    ids: &[i64],
+    tags: &[String],
+    filter_chain: Option<FilterChain>,
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_GET: Getting item via remote server");
+
+    if !ids.is_empty() && !tags.is_empty() {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "Both ID and tags given, you must supply either IDs or tags when using --get",
+        )
+        .exit();
+    }
+
+    let item_id = resolve_item_id(client, ids, tags)?;
+
+    // Get item info for metadata
+    let item_info = client.get_item_info(item_id)?;
+    let metadata = &item_info.metadata;
+
+    // Get streaming reader for raw content
+    let (reader, compression) = client.get_item_content_stream(item_id)?;
+    let compression_type = CompressionType::from_str(&compression).unwrap_or(CompressionType::Raw);
+
+    // Decompress through streaming readers
+    let mut decompressed_reader: Box<dyn Read> =
+        CompressionService::decompressing_reader(reader, &compression_type)?;
+
+    // Binary detection: sample first chunk
+    let mut sample_buf = [0u8; crate::common::PIPESIZE];
+    let sample_len = decompressed_reader.read(&mut sample_buf)?;
+    check_binary_tty(metadata, &sample_buf[..sample_len], settings.force)?;
+
+    // If filters present, buffer through filter chain; otherwise stream directly
+    if let Some(mut chain) = filter_chain {
+        // Apply filter to sample first, then remaining
+        let mut output = Vec::new();
+        chain.filter(&mut &sample_buf[..sample_len], &mut output)?;
+        crate::common::stream_copy(&mut decompressed_reader, |chunk| {
+            chain.filter(&mut std::io::Cursor::new(chunk), &mut output)?;
+            Ok(())
+        })?;
+        let stdout = std::io::stdout();
+        let mut stdout = stdout.lock();
+        stdout.write_all(&output)?;
+        stdout.flush()?;
+    } else {
+        // Stream decompressed content to stdout
+        let stdout = std::io::stdout();
+        let mut stdout = stdout.lock();
+        stdout.write_all(&sample_buf[..sample_len])?;
+        crate::common::stream_copy(&mut decompressed_reader, |chunk| {
+            stdout.write_all(chunk)?;
+            Ok(())
+        })?;
+        stdout.flush()?;
+    }
+
+    Ok(())
+}

src/modes/client/import.rs

@@ -0,0 +1,160 @@
+use anyhow::{Context, Result, anyhow};
+use clap::Command;
+use log::debug;
+use std::collections::HashMap;
+use std::fs;
+use std::io::Read;
+use std::path::Path;
+
+use crate::client::KeepClient;
+use crate::compression_engine::CompressionType;
+use crate::config;
+use crate::modes::common::ImportMeta;
+use std::str::FromStr;
+
+/// Import items from a `.keep.tar` archive or legacy `.meta.yml` file via client.
+///
+/// For `.keep.tar` files, streams the archive to the server's `/api/import` endpoint.
+/// For `.meta.yml` files, uses the legacy single-item import path.
+pub fn mode(
+    client: &KeepClient,
+    cmd: &mut Command,
+    settings: &config::Settings,
+    import_path: &str,
+) -> Result<()> {
+    if import_path.ends_with(".keep.tar") {
+        import_tar(client, cmd, settings, import_path)
+    } else if import_path.ends_with(".meta.yml") {
+        import_legacy(client, cmd, settings, import_path)
+    } else {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            format!("Unsupported import format: {}", import_path),
+        )
+        .exit();
+    }
+}
+
+/// Import from a `.keep.tar` archive via the server API.
+fn import_tar(
+    client: &KeepClient,
+    _cmd: &mut Command,
+    settings: &config::Settings,
+    tar_path: &str,
+) -> Result<()> {
+    let path = Path::new(tar_path);
+
+    let imported_ids = client
+        .import_tar_file(path)
+        .map_err(|e| anyhow!("Import failed: {e}"))?;
+
+    if !settings.quiet {
+        println!(
+            "KEEP: Imported {} item(s): {:?}",
+            imported_ids.len(),
+            imported_ids
+        );
+    }
+
+    debug!(
+        "CLIENT_IMPORT: Imported {} items from {}",
+        imported_ids.len(),
+        tar_path
+    );
+
+    Ok(())
+}
+
+/// Legacy single-item import from a `.meta.yml` file.
+fn import_legacy(
+    client: &KeepClient,
+    cmd: &mut Command,
+    settings: &config::Settings,
+    meta_file: &str,
+) -> Result<()> {
+    // Read and parse metadata
+    let meta_yaml = fs::read_to_string(meta_file)
+        .with_context(|| format!("Cannot read metadata file: {meta_file}"))?;
+    let import_meta: ImportMeta = serde_yaml::from_str(&meta_yaml)
+        .with_context(|| format!("Cannot parse metadata file: {meta_file}"))?;
+
+    // Validate compression type
+    CompressionType::from_str(&import_meta.compression).map_err(|_| {
+        anyhow!(
+            "Invalid compression type '{}' in metadata file",
+            import_meta.compression
+        )
+    })?;
+
+    debug!(
+        "CLIENT_IMPORT: Parsed meta: ts={}, compression={}, tags={:?}",
+        import_meta.ts, import_meta.compression, import_meta.tags
+    );
+
+    // Build query parameters
+    let ts_str = import_meta.ts.to_rfc3339();
+    let params = [
+        ("compress".to_string(), "false".to_string()),
+        ("meta".to_string(), "false".to_string()),
+        ("tags".to_string(), import_meta.tags.join(",")),
+        (
+            "compression_type".to_string(),
+            import_meta.compression.clone(),
+        ),
+        ("ts".to_string(), ts_str),
+    ];
+    let param_refs: Vec<(&str, &str)> = params
+        .iter()
+        .map(|(k, v)| (k.as_str(), v.as_str()))
+        .collect();
+
+    // Stream data to server without buffering entire file
+    let item_info = if let Some(ref data_file) = settings.import_data_file {
+        let mut reader = fs::File::open(data_file)
+            .with_context(|| format!("Cannot read data file: {}", data_file.display()))?;
+        client.post_stream("/api/item/", &mut reader, &param_refs)?
+    } else {
+        // For stdin, we need to buffer since stdin can't be seeked
+        // and post_stream may need to retry.
+        let mut buf = Vec::new();
+        std::io::stdin()
+            .read_to_end(&mut buf)
+            .context("Cannot read data from stdin")?;
+        if buf.is_empty() {
+            cmd.error(
+                clap::error::ErrorKind::InvalidValue,
+                "No data provided (empty stdin)",
+            )
+            .exit();
+        }
+        let mut cursor = std::io::Cursor::new(&buf);
+        client.post_stream("/api/item/", &mut cursor, &param_refs)?
+    };
+
+    let item_id = item_info.id;
+    debug!("CLIENT_IMPORT: Created item {} via server", item_id);
+
+    // Set uncompressed size if known from metadata
+    if let Some(size) = import_meta.uncompressed_size {
+        client.set_item_size(item_id, size as u64)?;
+        debug!("CLIENT_IMPORT: Set size to {}", size);
+    }
+
+    // Post metadata
+    if !import_meta.metadata.is_empty() {
+        client.post_metadata(item_id, &import_meta.metadata)?;
+        debug!(
+            "CLIENT_IMPORT: Set {} metadata entries",
+            import_meta.metadata.len()
+        );
+    }
+
+    if !settings.quiet {
+        println!(
+            "KEEP: Imported item {} tags: {:?}",
+            item_id, import_meta.tags
+        );
+    }
+
+    Ok(())
+}

src/modes/client/info.rs

@@ -0,0 +1,52 @@
+use crate::client::KeepClient;
+use crate::modes::common::{
+    DisplayItemInfo, OutputFormat, format_size, render_item_info_table, resolve_item_ids,
+    settings_output_format,
+};
+use clap::Command;
+use log::debug;
+
+pub fn mode(
+    client: &KeepClient,
+    _cmd: &mut Command,
+    settings: &crate::config::Settings,
+    ids: &[i64],
+    tags: &[String],
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_INFO: Getting item info via remote server");
+
+    let output_format = settings_output_format(settings);
+    let item_ids = resolve_item_ids(client, ids, tags)?;
+
+    for &id in &item_ids {
+        let item = client.get_item_info(id)?;
+
+        match output_format {
+            OutputFormat::Json | OutputFormat::Yaml => {
+                crate::modes::common::print_serialized(&item, &output_format)?;
+            }
+            OutputFormat::Table => {
+                let display = DisplayItemInfo {
+                    id: item.id,
+                    timestamp: item.ts.clone(),
+                    path: String::new(),
+                    stream_size: item
+                        .uncompressed_size
+                        .map(|s| format_size(s as u64, settings.human_readable))
+                        .unwrap_or_else(|| "N/A".to_string()),
+                    compression: item.compression.clone(),
+                    file_size: String::new(),
+                    tags: item.tags.clone(),
+                    metadata: item
+                        .metadata
+                        .iter()
+                        .map(|(k, v)| (k.clone(), v.clone()))
+                        .collect(),
+                };
+                render_item_info_table(&display, &settings.table_config);
+            }
+        }
+    }
+
+    Ok(())
+}

src/modes/client/list.rs

@@ -0,0 +1,71 @@
+use crate::client::KeepClient;
+use crate::modes::common::{
+    ColumnType, OutputFormat, format_size, render_list_table_with_format, settings_output_format,
+};
+use clap::Command;
+use log::debug;
+use std::str::FromStr;
+
+pub fn mode(
+    client: &KeepClient,
+    _cmd: &mut Command,
+    settings: &crate::config::Settings,
+    ids: &[i64],
+    tags: &[String],
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_LIST: Listing items via remote server");
+
+    let items = client.list_items(ids, tags, "newest", 0, 100, &settings.meta_filter())?;
+
+    if settings.ids_only {
+        for item in &items {
+            println!("{}", item.id);
+        }
+        return Ok(());
+    }
+
+    let output_format = settings_output_format(settings);
+
+    match output_format {
+        OutputFormat::Json | OutputFormat::Yaml => {
+            crate::modes::common::print_serialized(&items, &output_format)?;
+        }
+        OutputFormat::Table => {
+            let rows: Vec<Vec<String>> = items
+                .iter()
+                .map(|item| {
+                    let mut row = Vec::new();
+                    for column in &settings.list_format {
+                        let col_type = ColumnType::from_str(&column.name).ok();
+                        let cell = match col_type {
+                            Some(ColumnType::Id) => item.id.to_string(),
+                            Some(ColumnType::Time) => item.ts.clone(),
+                            Some(ColumnType::Size) => item
+                                .uncompressed_size
+                                .map(|s| format_size(s as u64, settings.human_readable))
+                                .unwrap_or_default(),
+                            Some(ColumnType::Compression) => item.compression.clone(),
+                            Some(ColumnType::Tags) => item.tags.join(" "),
+                            Some(ColumnType::Meta) => {
+                                let meta_key = column.name.strip_prefix("meta:");
+                                match meta_key {
+                                    Some(key) => {
+                                        item.metadata.get(key).cloned().unwrap_or_default()
+                                    }
+                                    None => String::new(),
+                                }
+                            }
+                            _ => String::new(),
+                        };
+                        row.push(cell);
+                    }
+                    row
+                })
+                .collect();
+
+            render_list_table_with_format(&settings.list_format, &rows, &settings.table_config);
+        }
+    }
+
+    Ok(())
+}

src/modes/client/mod.rs

@@ -0,0 +1,10 @@
+pub mod delete;
+pub mod diff;
+pub mod export;
+pub mod get;
+pub mod import;
+pub mod info;
+pub mod list;
+pub mod save;
+pub mod status;
+pub mod update;

src/modes/client/save.rs

@@ -0,0 +1,180 @@
+use crate::client::KeepClient;
+use crate::compression_engine::CompressionType;
+use crate::config::Settings;
+use crate::meta_plugin::SaveMetaFn;
+use crate::modes::common::settings_compression_type;
+use crate::services::ItemInfo;
+use crate::services::compression_service::CompressionService;
+use crate::services::meta_service::MetaService;
+use anyhow::Result;
+use clap::Command;
+use is_terminal::IsTerminal;
+use log::debug;
+use std::collections::HashMap;
+use std::io::{Read, Write};
+use std::sync::{Arc, Mutex};
+
+/// Streaming save mode for client.
+///
+/// Uses three threads for true streaming with constant memory:
+/// - Reader thread: reads stdin, tees to stdout, runs meta plugins,
+///   compresses data, writes to OS pipe
+/// - Pipe: zero-copy transfer of compressed bytes between threads
+/// - Streamer thread: reads from pipe, streams to server via chunked HTTP
+///
+/// Meta plugins run on the client side during streaming. Collected metadata
+/// is sent to the server via a separate POST after streaming completes.
+///
+/// Memory usage is O(PIPESIZE) regardless of data size.
+pub fn mode(
+    client: &KeepClient,
+    cmd: &mut Command,
+    settings: &Settings,
+    tags: &mut Vec<String>,
+    metadata: HashMap<String, String>,
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_SAVE: Saving item via remote server (streaming)");
+
+    crate::modes::common::ensure_default_tag(tags);
+
+    // Determine compression type from settings
+    let compression_type = settings_compression_type(cmd, settings);
+    let compression_type_str = compression_type.to_string();
+    // In client mode, the client always handles compression (even "raw").
+    // The server should never re-compress client data.
+    let server_compress = false;
+
+    // Shared metadata collection: plugins write here via save_meta closure
+    let collected_meta: Arc<Mutex<HashMap<String, String>>> = Arc::new(Mutex::new(HashMap::new()));
+    let meta_collector = collected_meta.clone();
+    let save_meta: SaveMetaFn = Arc::new(Mutex::new(move |name: &str, value: &str| {
+        if let Ok(mut map) = meta_collector.lock() {
+            map.insert(name.to_string(), value.to_string());
+        }
+    }));
+
+    // Create MetaService and get plugins (must happen before spawning reader thread)
+    let meta_service = MetaService::new(save_meta);
+    let mut plugins = meta_service.get_plugins(cmd, settings);
+
+    // Create OS pipe for streaming compressed bytes between threads
+    let (pipe_reader, pipe_writer) = os_pipe::pipe()?;
+
+    // Reader thread: stdin → tee(stdout) → meta plugins → compress → pipe
+    let compression_type_clone = compression_type.clone();
+    let reader_handle = std::thread::spawn(move || -> Result<u64> {
+        let stdin = std::io::stdin();
+        let stdout = std::io::stdout();
+        let mut stdin_lock = stdin.lock();
+        let mut stdout_lock = stdout.lock();
+
+        let mut total_bytes = 0u64;
+        let mut buffer = [0u8; 8192];
+
+        // Initialize meta plugins
+        meta_service.initialize_plugins(&mut plugins);
+
+        // Wrap pipe writer with appropriate compression
+        let mut compressor: Box<dyn Write> =
+            CompressionService::compressing_writer(Box::new(pipe_writer), &compression_type_clone)?;
+
+        loop {
+            let n = stdin_lock.read(&mut buffer)?;
+            if n == 0 {
+                break;
+            }
+
+            // Tee to stdout
+            stdout_lock.write_all(&buffer[..n])?;
+
+            // Feed chunk to meta plugins
+            meta_service.process_chunk(&mut plugins, &buffer[..n]);
+
+            total_bytes += n as u64;
+
+            // Compress and write to pipe
+            compressor.write_all(&buffer[..n])?;
+        }
+
+        // Finalize meta plugins (digest, text, tokens produce final output here)
+        meta_service.finalize_plugins(&mut plugins);
+
+        // Explicitly flush and finalize compression before dropping.
+        compressor.flush()?;
+        drop(compressor);
+
+        Ok(total_bytes)
+    });
+
+    // Streamer thread: reads compressed bytes from pipe → POST to server
+    let client_url = client.base_url().to_string();
+    let client_username = client.username().cloned();
+    let client_password = client.password().cloned();
+    let client_jwt = client.jwt().cloned();
+    let tags_clone = tags.clone();
+    let compression_type_str_clone = compression_type_str.clone();
+
+    let streamer_handle = std::thread::spawn(move || -> Result<ItemInfo> {
+        let streaming_client =
+            KeepClient::new(&client_url, client_username, client_password, client_jwt)?;
+        let params = [
+            ("compress".to_string(), server_compress.to_string()),
+            ("meta".to_string(), "false".to_string()),
+            ("tags".to_string(), tags_clone.join(",")),
+            // Always send compression_type when compress=false (client handled compression)
+            ("compression_type".to_string(), compression_type_str_clone),
+        ];
+        // Filter out empty params
+        let params: Vec<(String, String)> =
+            params.into_iter().filter(|(_, v)| !v.is_empty()).collect();
+        let param_refs: Vec<(&str, &str)> = params
+            .iter()
+            .map(|(k, v)| (k.as_str(), v.as_str()))
+            .collect();
+
+        let mut reader: Box<dyn Read> = Box::new(pipe_reader);
+        let item_info = streaming_client.post_stream("/api/item/", &mut reader, &param_refs)?;
+        Ok(item_info)
+    });
+
+    // Wait for streaming to complete, capture item info
+    let item_info = streamer_handle
+        .join()
+        .map_err(|e| anyhow::anyhow!("Streamer thread panicked: {:?}", e))??;
+
+    // Wait for reader thread (should complete quickly after pipe is drained)
+    let uncompressed_size = reader_handle
+        .join()
+        .map_err(|e| anyhow::anyhow!("Reader thread panicked: {:?}", e))??;
+
+    // Merge plugin-collected metadata with CLI metadata
+    let mut local_metadata = metadata;
+
+    // Add plugin-collected metadata (digest, hostname, text stats, etc.)
+    if let Ok(plugin_meta) = collected_meta.lock() {
+        for (k, v) in plugin_meta.iter() {
+            local_metadata.entry(k.clone()).or_insert_with(|| v.clone());
+        }
+    }
+
+    // Send uncompressed size to server (proper field, not metadata)
+    client.set_item_size(item_info.id, uncompressed_size)?;
+
+    // Send metadata to server
+    if !local_metadata.is_empty() {
+        client.post_metadata(item_info.id, &local_metadata)?;
+    }
+
+    // Print status to stderr (item ID is known immediately from server response)
+    if !settings.quiet {
+        if std::io::stderr().is_terminal() {
+            eprintln!("KEEP: New item: {}  tags: {}", item_info.id, tags.join(" "));
+        } else {
+            eprintln!("KEEP: New item: {}  tags: {tags:?}", item_info.id);
+        }
+    }
+
+    debug!("CLIENT_SAVE: Streaming complete, {uncompressed_size} bytes uncompressed");
+
+    Ok(())
+}

src/modes/client/status.rs

@@ -0,0 +1,91 @@
+use crate::client::KeepClient;
+use crate::modes::common::OutputFormat;
+use crate::modes::common::settings_output_format;
+use clap::Command;
+use comfy_table::{Attribute, Cell, Table};
+use log::debug;
+
+pub fn mode(
+    client: &KeepClient,
+    _cmd: &mut Command,
+    settings: &crate::config::Settings,
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_STATUS: Getting status from remote server");
+
+    let status_info = client.get_status()?;
+
+    let output_format = settings_output_format(settings);
+
+    match output_format {
+        OutputFormat::Json | OutputFormat::Yaml => {
+            crate::modes::common::print_serialized(&status_info, &output_format)?;
+        }
+        OutputFormat::Table => {
+            // Paths
+            let path_table =
+                crate::modes::common::build_path_table(&status_info.paths, &settings.table_config);
+            println!("PATHS:");
+            println!(
+                "{}",
+                crate::modes::common::trim_lines_end(&path_table.trim_fmt())
+            );
+            println!();
+
+            // Configured meta plugins
+            if let Some(ref configured) = status_info.configured_meta_plugins
+                && !configured.is_empty()
+            {
+                let mut sorted = configured.clone();
+                sorted.sort_by(|a, b| a.name.cmp(&b.name));
+
+                let mut table =
+                    crate::modes::common::create_table_with_config(&settings.table_config);
+                table.set_header(vec![
+                    Cell::new("Plugin Name").add_attribute(Attribute::Bold),
+                    Cell::new("Enabled").add_attribute(Attribute::Bold),
+                ]);
+                for plugin in &sorted {
+                    let enabled = status_info.enabled_meta_plugins.contains(&plugin.name);
+                    table.add_row(vec![
+                        plugin.name.clone(),
+                        if enabled { "Yes" } else { "No" }.to_string(),
+                    ]);
+                }
+                println!("META PLUGINS:");
+                println!(
+                    "{}",
+                    crate::modes::common::trim_lines_end(&table.trim_fmt())
+                );
+                println!();
+            }
+
+            // Compression
+            if !status_info.compression.is_empty() {
+                let mut table =
+                    crate::modes::common::create_table_with_config(&settings.table_config);
+                table.set_header(vec![
+                    Cell::new("Type").add_attribute(Attribute::Bold),
+                    Cell::new("Found").add_attribute(Attribute::Bold),
+                    Cell::new("Default").add_attribute(Attribute::Bold),
+                    Cell::new("Binary").add_attribute(Attribute::Bold),
+                ]);
+                for comp in &status_info.compression {
+                    table.add_row(vec![
+                        comp.compression_type.clone(),
+                        if comp.found { "Yes" } else { "No" }.to_string(),
+                        if comp.default { "Yes" } else { "No" }.to_string(),
+                        comp.binary.clone(),
+                    ]);
+                }
+                println!("COMPRESSION:");
+                println!(
+                    "{}",
+                    crate::modes::common::trim_lines_end(&table.trim_fmt())
+                );
+                println!();
+            }
+        }
+    }
+
+    Ok(())
+}

src/modes/client/update.rs

@@ -0,0 +1,102 @@
+use crate::client::KeepClient;
+use crate::config::Settings;
+use anyhow::Result;
+use clap::Command;
+use log::debug;
+use std::collections::HashMap;
+
+/// Client update mode: runs meta plugins on the server for an existing item.
+///
+/// Sends the list of plugin names (from --meta-plugin config) and any direct
+/// metadata (--meta key=value) to the server. The server reads the stored file,
+/// runs the specified plugins, and stores the results.
+pub fn mode(
+    client: &KeepClient,
+    cmd: &mut Command,
+    settings: &Settings,
+    ids: &mut [i64],
+    tags: &mut [String],
+) -> Result<(), anyhow::Error> {
+    debug!("CLIENT_UPDATE: Updating item via remote server");
+
+    if ids.len() != 1 {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "--update requires exactly one numeric ID",
+        )
+        .exit();
+    }
+
+    let item_id = ids[0];
+
+    // Collect plugin names from settings (--meta-plugin config)
+    let plugin_names: Vec<String> = settings
+        .meta_plugins_names()
+        .into_iter()
+        .flat_map(|s| {
+            s.split(',')
+                .map(|p| p.trim().to_string())
+                .collect::<Vec<_>>()
+        })
+        .filter(|p| !p.is_empty())
+        .collect();
+
+    // Collect direct metadata from --meta flags
+    let metadata: HashMap<String, String> = settings
+        .meta
+        .iter()
+        .filter_map(|(k, v)| v.as_ref().map(|val| (k.clone(), val.clone())))
+        .collect();
+
+    // Build query params
+    let mut params: Vec<(String, String)> = Vec::new();
+
+    if !plugin_names.is_empty() {
+        params.push(("plugins".to_string(), plugin_names.join(",")));
+    }
+
+    if !metadata.is_empty() {
+        let meta_json = serde_json::to_string(&metadata)?;
+        params.push(("metadata".to_string(), meta_json));
+    }
+
+    if !tags.is_empty() {
+        params.push(("tags".to_string(), tags.join(",")));
+    }
+
+    // Nothing to update
+    if params.is_empty() {
+        if !settings.quiet {
+            eprintln!("KEEP: No changes specified for item {item_id}");
+        }
+        return Ok(());
+    }
+
+    let param_refs: Vec<(&str, &str)> = params
+        .iter()
+        .map(|(k, v)| (k.as_str(), v.as_str()))
+        .collect();
+
+    let url_path = format!("/api/item/{item_id}/update");
+
+    // POST to update endpoint
+    let _item_info = client.post_bytes(&url_path, &[], &param_refs)?;
+
+    if !settings.quiet {
+        let mut parts = Vec::new();
+        if !plugin_names.is_empty() {
+            parts.push(format!("plugins: {}", plugin_names.join(", ")));
+        }
+        if !metadata.is_empty() {
+            parts.push(format!("{} metadata", metadata.len()));
+        }
+        if !tags.is_empty() {
+            parts.push(format!("tags: {}", tags.join(" ")));
+        }
+        let action = parts.join(", ");
+
+        eprintln!("KEEP: Updated item {item_id} ({action})");
+    }
+
+    Ok(())
+}

src/modes/common.rs

@@ -1,83 +1,130 @@
-use crate::Args;
+use crate::common::status::PathInfo;
 use crate::compression_engine::CompressionType;
+/// Common utilities shared across different modes in the Keep application.
+///
+/// This module provides helper functions for formatting, configuration parsing,
+/// table creation, and environment variable handling used by various CLI modes.
+///
+/// # Usage
+///
+/// These utilities are typically used internally by mode implementations:
+///
+/// ```
+/// # use keep::modes::common::{format_size, OutputFormat};
+/// let formatted = format_size(1024, true); // "1.0K"
+/// // let format = OutputFormat::from_str("json")?;
+/// ```
+use crate::config;
 use crate::meta_plugin::MetaPluginType;
+use anyhow::{Result, anyhow};
+use chrono::{DateTime, Utc};
 use clap::Command;
 use clap::error::ErrorKind;
+use comfy_table::{Attribute, Cell, ContentArrangement, Table};
 use log::debug;
-use prettytable::format::TableFormat;
-use regex::Regex;
+use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::env;
+use std::io::IsTerminal;
 use std::str::FromStr;
 use strum::IntoEnumIterator;
-use serde::{Deserialize, Serialize};
+
+#[derive(Debug, Clone, strum::EnumString, strum::Display, PartialEq)]
+#[strum(ascii_case_insensitive)]
+/// Enum representing supported output formats for structured data.
+///
+/// Used to determine how to display lists, info, and status information in CLI modes.
+/// Defaults to Table for human-readable output; JSON/YAML for machine parsing.
+///
+/// # Variants
+///
+/// * `Table` - Formatted table output (default).
+/// * `Json` - JSON structured output.
+/// * `Yaml` - YAML structured output.
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::OutputFormat;
+/// # use std::str::FromStr;
+/// assert_eq!(OutputFormat::from_str("json").unwrap(), OutputFormat::Json);
+/// ```
+pub enum OutputFormat {
+    Table,
+    Json,
+    Yaml,
+}
+
+pub const IMPORT_FORMAT_ERROR: &str =
+    "Unsupported import format: {} (expected .keep.tar or .meta.yml)";
 
 pub fn get_meta_from_env() -> HashMap<String, String> {
     debug!("COMMON: Getting meta from KEEP_META_*");
-    let re = Regex::new(r"^KEEP_META_(.+)$").unwrap();
     let mut meta_env: HashMap<String, String> = HashMap::new();
+    const PREFIX: &str = "KEEP_META_";
     for (key, value) in env::vars() {
-        if let Some(meta_name_caps) = re.captures(key.as_str()) {
-            let name = String::from(meta_name_caps.get(1).unwrap().as_str());
-            // Ignore KEEP_META_PLUGINS
-            if name != "PLUGINS" {
-                debug!("COMMON: Found meta: {}={}", name.clone(), value.clone());
-                meta_env.insert(name, value.clone());
+        if let Some(name) = key.strip_prefix(PREFIX) {
+            if !name.is_empty() && name != "PLUGINS" {
+                debug!("COMMON: Found meta: {}={}", name, value);
+                meta_env.insert(name.to_string(), value);
             }
         }
     }
     meta_env
 }
 
-pub fn format_size_human_readable(size: u64) -> String {
-    const UNITS: &[&str] = &["", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei"];
-    const THRESHOLD: u64 = 1024;
-    
-    if size == 0 {
-        return "0".to_string();
-    }
-    
-    let mut size_f = size as f64;
-    let mut unit_index = 0;
-    
-    while size_f >= THRESHOLD as f64 && unit_index < UNITS.len() - 1 {
-        size_f /= THRESHOLD as f64;
-        unit_index += 1;
-    }
-    
-    if unit_index == 0 {
-        format!("{}", size)
-    } else if size_f.fract() == 0.0 {
-        format!("{}{}", size_f as u64, UNITS[unit_index])
-    } else {
-        format!("{:.1}{}", size_f, UNITS[unit_index])
-    }
-}
-
+/// Formats a file size in bytes to human-readable or raw format.
+///
+/// Uses the humansize crate for human-readable output with decimal units (KB, MB, etc.).
+///
+/// # Arguments
+///
+/// * `size` - Size in bytes as u64.
+/// * `human_readable` - If true, use units like KB, MB; otherwise, raw bytes as string.
+///
+/// # Returns
+///
+/// `String` - Formatted size string, e.g., "1.0K" or "1024".
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::format_size;
+/// let raw = format_size(1024, false); // "1024"
+/// let human = format_size(1024, true); // "1.0K"
+/// ```
 pub fn format_size(size: u64, human_readable: bool) -> String {
     match human_readable {
-        true => format_size_human_readable(size),
+        true => humansize::format_size(size, humansize::DECIMAL),
         false => size.to_string(),
     }
 }
 
-pub fn string_column(s: String, column_width: usize) -> String {
-    if column_width > 0 {
-        match s.char_indices().nth(column_width) {
-            None => s.to_string(),
-            Some((idx, _)) => s[..idx].to_string(),
-        }
-    } else {
-        s.to_string()
-    }
-}
-
-pub fn size_column(size: u64, human_readable: bool, column_width: usize) -> String {
-    string_column(format_size(size, human_readable), column_width)
-}
-
-#[derive(Debug, Eq, PartialEq, Clone, strum::EnumIter, strum::Display, strum::EnumString)]
+#[derive(Debug, Eq, PartialEq, Clone, strum::EnumIter, strum::Display)]
 #[strum(ascii_case_insensitive)]
+/// Enum representing column types for table display.
+///
+/// Defines standard and meta columns for list/info modes. Supports "meta:<name>" for specific metadata columns.
+///
+/// # Variants
+///
+/// * `Id` - Item ID column.
+/// * `Time` - Timestamp column.
+/// * `Size` - Content size column.
+/// * `Compression` - Compression type column.
+/// * `FileSize` - On-disk file size column.
+/// * `FilePath` - File path column.
+/// * `Tags` - Tags column.
+/// * `Meta` - Metadata column (with sub-type via string parsing).
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::ColumnType;
+/// # use std::str::FromStr;
+/// assert_eq!(ColumnType::from_str("id").unwrap(), ColumnType::Id);
+/// assert_eq!(ColumnType::from_str("meta:hostname").unwrap(), ColumnType::Meta);
+/// ```
 pub enum ColumnType {
     Id,
     Time,
@@ -89,111 +136,48 @@ pub enum ColumnType {
     Meta,
 }
 
-impl ColumnType {
-    /// Returns a Result with error message if the string is not a valid ColumnType
-    pub fn from_str(s: &str) -> anyhow::Result<Self> {
-        Ok(Self::try_from(s)?)
-    }
-}
-
-// impl TryFrom<&str> for ColumnType is already implemented by strum_macros
-// so we remove this conflicting implementation
-
-pub fn get_format_box_chars_no_border_line_separator() -> TableFormat {
-    prettytable::format::FormatBuilder::new()
-        .column_separator('│')
-        .borders('│')
-        .separators(
-            &[prettytable::format::LinePosition::Top],
-            prettytable::format::LineSeparator::new('─', '┬', '┌', '┐'),
-        )
-        .separators(
-            &[prettytable::format::LinePosition::Title],
-            prettytable::format::LineSeparator::new('─', '┼', '├', '┤'),
-        )
-        .separators(
-            &[prettytable::format::LinePosition::Bottom],
-            prettytable::format::LineSeparator::new('─', '┴', '└', '┘'),
-        )
-        .padding(1, 1)
-        .build()
-}
-
-pub fn get_digest_type_meta(digest_type: MetaPluginType) -> String {
-    format!("digest_{}", digest_type.to_string().to_lowercase())
-}
-
-
-pub fn cmd_args_digest_type(cmd: &mut Command, args: &Args) -> MetaPluginType {
-    let digest_name = args
-        .item
-        .digest
-        .clone()
-        .unwrap_or(MetaPluginType::DigestSha256.to_string());
-
-    let digest_type_opt = MetaPluginType::from_str(&digest_name);
-    if digest_type_opt.is_err() {
-        cmd.error(
-            ErrorKind::InvalidValue,
-            format!("Invalid digest algorithm '{}'. Use 'sha256' or 'md5'", digest_name),
-        )
-        .exit();
-    }
-
-    digest_type_opt.unwrap()
-}
-
-pub fn cmd_args_compression_type(cmd: &mut Command, args: &Args) -> CompressionType {
-    let compression_name = args
-        .item
-        .compression
-        .clone()
-        .unwrap_or(CompressionType::LZ4.to_string());
-
-    let compression_type_opt = CompressionType::from_str(&compression_name);
-    if compression_type_opt.is_err() {
-        cmd.error(
-            ErrorKind::InvalidValue,
-            format!("Invalid compression algorithm '{}'. Supported algorithms: lz4, gzip, xz, zstd", compression_name),
-        )
-        .exit();
-    }
-
-    compression_type_opt.unwrap()
-}
-
-#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
-pub enum OutputFormat {
-    Table,
-    Json,
-    Yaml,
-}
-
-impl FromStr for OutputFormat {
+impl std::str::FromStr for ColumnType {
     type Err = anyhow::Error;
-    fn from_str(s: &str) -> Result<Self, Self::Err> {
-        match s.to_lowercase().as_str() {
-            "table" => Ok(OutputFormat::Table),
-            "json" => Ok(OutputFormat::Json),
-            "yaml" => Ok(OutputFormat::Yaml),
-            _ => Err(anyhow::anyhow!("Invalid output format. Supported formats: table, json, yaml")),
+
+    fn from_str(s: &str) -> anyhow::Result<Self> {
+        let lower_s = s.to_lowercase();
+        if s.starts_with("meta:") {
+            Ok(ColumnType::Meta)
+        } else {
+            for variant in ColumnType::iter() {
+                if variant.to_string().to_lowercase() == lower_s {
+                    return Ok(variant);
+                }
+            }
+            Err(anyhow::anyhow!("Invalid column type: {}", s))
         }
     }
 }
 
-pub fn get_output_format(args: &Args) -> OutputFormat {
-    args.options.output_format
-        .as_ref()
-        .and_then(|s| OutputFormat::from_str(s).ok())
-        .unwrap_or(OutputFormat::Table)
-}
-
-pub fn cmd_args_meta_plugin_types(cmd: &mut Command, args: &Args) -> Vec<MetaPluginType> {
+/// Extracts configured meta plugin types from settings and command.
+///
+/// Handles comma-separated plugin names and validates against registered types.
+///
+/// # Arguments
+///
+/// * `cmd` - Mutable Clap command for error reporting.
+/// * `settings` - Application settings with plugin config.
+///
+/// # Returns
+///
+/// `Vec<MetaPluginType>` - List of enabled plugin types.
+///
+/// # Panics
+///
+/// Exits via Clap error if unknown plugin type specified.
+pub fn settings_meta_plugin_types(
+    cmd: &mut Command,
+    settings: &config::Settings,
+) -> Vec<MetaPluginType> {
     let mut meta_plugin_types = Vec::new();
 
     // Handle comma-separated values in each meta_plugins argument
-    for meta_plugin_names_str in &args.item.meta_plugins {
+    for meta_plugin_names_str in &settings.meta_plugins_names() {
         let meta_plugin_names: Vec<&str> = meta_plugin_names_str.split(',').collect();
 
         for name in meta_plugin_names {
@@ -205,8 +189,10 @@ pub fn cmd_args_meta_plugin_types(cmd: &mut Command, args: &Args) -> Vec<MetaPlu
             // Try to find the MetaPluginType by meta name
             let mut found = false;
             for meta_plugin_type in MetaPluginType::iter() {
-                let mut meta_plugin = crate::meta_plugin::get_meta_plugin(meta_plugin_type.clone());
-                if meta_plugin.meta_name() == trimmed_name {
+                if let Ok(meta_plugin) =
+                    crate::meta_plugin::get_meta_plugin(meta_plugin_type.clone(), None, None)
+                    && meta_plugin.meta_type().to_string() == trimmed_name
+                {
                     meta_plugin_types.push(meta_plugin_type);
                     found = true;
                     break;
@@ -216,7 +202,7 @@ pub fn cmd_args_meta_plugin_types(cmd: &mut Command, args: &Args) -> Vec<MetaPlu
             if !found {
                 cmd.error(
                     ErrorKind::InvalidValue,
-                    format!("Unknown meta plugin type: {}", trimmed_name),
+                    format!("Unknown meta plugin type: {trimmed_name}"),
                 )
                 .exit();
             }
@@ -225,3 +211,493 @@ pub fn cmd_args_meta_plugin_types(cmd: &mut Command, args: &Args) -> Vec<MetaPlu
 
     meta_plugin_types
 }
+
+/// Determines compression type from settings and command arguments.
+///
+/// Validates the compression name and returns the corresponding enum variant.
+///
+/// # Arguments
+///
+/// * `cmd` - Mutable Clap command for error reporting.
+/// * `settings` - Application settings.
+///
+/// # Returns
+///
+/// `CompressionType` - The resolved compression type.
+///
+/// # Panics
+///
+/// Exits via Clap error if invalid compression specified.
+pub fn settings_compression_type(
+    cmd: &mut Command,
+    settings: &config::Settings,
+) -> CompressionType {
+    let compression_name = settings
+        .compression()
+        .unwrap_or(CompressionType::LZ4.to_string());
+
+    let compression_type_opt = CompressionType::from_str(&compression_name);
+    if compression_type_opt.is_err() {
+        cmd.error(
+            ErrorKind::InvalidValue,
+            format!("Invalid compression algorithm '{compression_name}'. Supported algorithms: lz4, gzip, xz, zstd"),
+        )
+        .exit();
+    }
+
+    compression_type_opt.unwrap()
+}
+
+/// Parses output format from settings.
+///
+/// Defaults to `Table` if not specified or invalid. Uses case-insensitive string parsing.
+///
+/// # Arguments
+///
+/// * `settings` - Application settings with optional output_format field.
+///
+/// # Returns
+///
+/// `OutputFormat` - Parsed enum variant or Table as default.
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::{settings_output_format, OutputFormat};
+/// // Example usage requires a Settings instance
+/// // let format = settings_output_format(&settings);
+/// ```
+pub fn settings_output_format(settings: &config::Settings) -> OutputFormat {
+    settings
+        .output_format
+        .as_ref()
+        .and_then(|s| OutputFormat::from_str(s).ok())
+        .unwrap_or(OutputFormat::Table)
+}
+
+/// Trims trailing whitespace from each line in a multi-line string.
+///
+/// Useful for cleaning up table output before printing. Preserves newlines but removes spaces/tabs at line ends.
+///
+/// # Arguments
+///
+/// * `s` - Input string with potential trailing whitespace, e.g., "line1  \nline2   ".
+///
+/// # Returns
+///
+/// `String` - Cleaned string with trimmed lines, e.g., "line1\nline2".
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::trim_lines_end;
+/// let cleaned = trim_lines_end("line1  \nline2   ");
+/// assert_eq!(cleaned, "line1\nline2");
+/// ```
+pub fn trim_lines_end(s: &str) -> String {
+    s.lines()
+        .map(|line| line.trim_end())
+        .collect::<Vec<&str>>()
+        .join("\n")
+}
+
+/// Creates a new table with styling based on terminal detection.
+///
+/// Loads appropriate preset (UTF8 or ASCII) if styling is enabled.
+///
+/// # Arguments
+///
+/// * `use_styling` - If true, apply visual styling.
+///
+/// # Returns
+///
+/// `Table` - Configured table instance.
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::create_table;
+/// let mut table = create_table(true);
+/// table.add_row(vec!["Header1", "Header2"]);
+/// ```
+pub fn create_table(_use_styling: bool) -> Table {
+    create_table_with_config(&crate::config::TableConfig::default())
+}
+
+/// Creates a table configured from application table settings.
+///
+/// Applies style presets, modifiers, content arrangement, and truncation indicators.
+///
+/// # Arguments
+///
+/// * `table_config` - Table configuration from settings.
+///
+/// # Returns
+///
+/// `Table` - Fully configured table.
+///
+/// # Examples
+///
+/// ```
+/// # use keep::modes::common::create_table_with_config;
+/// # use keep::config::TableConfig;
+/// let config = TableConfig::default();
+/// let table = create_table_with_config(&config);
+/// ```
+pub fn create_table_with_config(table_config: &crate::config::TableConfig) -> Table {
+    let mut table = Table::new();
+
+    // Set content arrangement
+    match table_config.content_arrangement {
+        crate::config::ContentArrangement::Dynamic => {
+            table.set_content_arrangement(comfy_table::ContentArrangement::Dynamic)
+        }
+        crate::config::ContentArrangement::DynamicFullWidth => {
+            table.set_content_arrangement(comfy_table::ContentArrangement::DynamicFullWidth)
+        }
+        crate::config::ContentArrangement::Disabled => {
+            table.set_content_arrangement(comfy_table::ContentArrangement::Disabled)
+        }
+    };
+
+    // Set style preset
+    match &table_config.style {
+        crate::config::TableStyle::Ascii => {
+            table.load_preset(comfy_table::presets::ASCII_FULL);
+        }
+        crate::config::TableStyle::Utf8 => {
+            table.load_preset(comfy_table::presets::UTF8_FULL);
+        }
+        crate::config::TableStyle::Utf8Full => {
+            table.load_preset(comfy_table::presets::UTF8_FULL);
+        }
+        crate::config::TableStyle::Nothing => {
+            table.load_preset(comfy_table::presets::NOTHING);
+        }
+        crate::config::TableStyle::Custom(preset) => {
+            // For custom presets, we'd need to parse the string
+            // This is a placeholder for custom preset handling
+            if preset == "ASCII_FULL" {
+                table.load_preset(comfy_table::presets::ASCII_FULL);
+            } else if preset == "UTF8_FULL" {
+                table.load_preset(comfy_table::presets::UTF8_FULL);
+            } else if preset == "NOTHING" {
+                table.load_preset(comfy_table::presets::NOTHING);
+            }
+            // Add more presets as needed
+        }
+    };
+
+    // Apply modifiers
+    for modifier in &table_config.modifiers {
+        match modifier.as_str() {
+            "UTF8_SOLID_INNER_BORDERS" => {
+                table.apply_modifier(comfy_table::modifiers::UTF8_SOLID_INNER_BORDERS);
+            }
+            "UTF8_ROUND_CORNERS" => {
+                table.apply_modifier(comfy_table::modifiers::UTF8_ROUND_CORNERS);
+            }
+            _ => {} // Ignore unknown modifiers
+        }
+    }
+
+    // Set truncation indicator if specified
+    if !table_config.truncation_indicator.is_empty() {
+        table.set_truncation_indicator(&table_config.truncation_indicator);
+    }
+
+    if !std::io::stdout().is_terminal() {
+        table.force_no_tty();
+    }
+
+    table
+}
+
+/// Display data for a single item's detail view (used by --info).
+pub struct DisplayItemInfo {
+    pub id: i64,
+    pub timestamp: String,
+    pub path: String,
+    pub stream_size: String,
+    pub compression: String,
+    pub file_size: String,
+    pub tags: Vec<String>,
+    pub metadata: Vec<(String, String)>,
+}
+
+/// Renders item detail table. Shared by local and client info modes.
+pub fn render_item_info_table(info: &DisplayItemInfo, table_config: &config::TableConfig) {
+    use comfy_table::{Attribute, Cell};
+
+    let mut table = create_table_with_config(table_config);
+
+    table.add_row(vec![
+        Cell::new("ID").add_attribute(Attribute::Bold),
+        Cell::new(info.id.to_string()),
+    ]);
+    table.add_row(vec![
+        Cell::new("Time").add_attribute(Attribute::Bold),
+        Cell::new(&info.timestamp),
+    ]);
+    table.add_row(vec![
+        Cell::new("Size").add_attribute(Attribute::Bold),
+        Cell::new(&info.stream_size),
+    ]);
+    table.add_row(vec![
+        Cell::new("Compression").add_attribute(Attribute::Bold),
+        Cell::new(&info.compression),
+    ]);
+    table.add_row(vec![
+        Cell::new("Tags").add_attribute(Attribute::Bold),
+        Cell::new(info.tags.join(" ")),
+    ]);
+
+    for (key, value) in &info.metadata {
+        table.add_row(vec![
+            Cell::new(format!("Meta: {key}")).add_attribute(Attribute::Bold),
+            Cell::new(value),
+        ]);
+    }
+
+    println!("{}", trim_lines_end(&table.trim_fmt()));
+}
+
+/// Renders list table with column format from config. Shared by local and client list modes.
+pub fn render_list_table_with_format(
+    columns: &[config::ColumnConfig],
+    rows: &[Vec<String>],
+    table_config: &config::TableConfig,
+) {
+    let mut table = create_table_with_config(table_config);
+
+    let header_cells: Vec<Cell> = columns
+        .iter()
+        .map(|col| Cell::new(&col.label).add_attribute(Attribute::Bold))
+        .collect();
+    table.set_header(header_cells);
+
+    for row in rows {
+        let cells: Vec<Cell> = row
+            .iter()
+            .enumerate()
+            .map(|(i, val)| {
+                let mut cell = Cell::new(val);
+                if let Some(col) = columns.get(i) {
+                    if let Some(ref fg) = col.fg_color {
+                        cell = apply_color(cell, fg, true);
+                    }
+                    if let Some(ref bg) = col.bg_color {
+                        cell = apply_color(cell, bg, false);
+                    }
+                    for attr in &col.attributes {
+                        cell = apply_table_attribute(cell, attr);
+                    }
+                }
+                cell
+            })
+            .collect();
+        table.add_row(cells);
+    }
+
+    println!("{}", trim_lines_end(&table.trim_fmt()));
+}
+
+/// Applies config TableColor to a comfy-table Cell.
+pub fn apply_color(mut cell: Cell, color: &config::TableColor, is_foreground: bool) -> Cell {
+    use comfy_table::Color;
+
+    let comfy_color = match color {
+        config::TableColor::Black => Color::Black,
+        config::TableColor::Red => Color::Red,
+        config::TableColor::Green => Color::Green,
+        config::TableColor::Yellow => Color::Yellow,
+        config::TableColor::Blue => Color::Blue,
+        config::TableColor::Magenta => Color::Magenta,
+        config::TableColor::Cyan => Color::Cyan,
+        config::TableColor::White => Color::White,
+        config::TableColor::Gray => Color::Grey,
+        config::TableColor::DarkRed => Color::DarkRed,
+        config::TableColor::DarkGreen => Color::DarkGreen,
+        config::TableColor::DarkYellow => Color::DarkYellow,
+        config::TableColor::DarkBlue => Color::DarkBlue,
+        config::TableColor::DarkMagenta => Color::DarkMagenta,
+        config::TableColor::DarkCyan => Color::DarkCyan,
+        config::TableColor::Rgb(r, g, b) => Color::Rgb {
+            r: *r,
+            g: *g,
+            b: *b,
+        },
+    };
+
+    if is_foreground {
+        cell = cell.fg(comfy_color);
+    } else {
+        cell = cell.bg(comfy_color);
+    }
+
+    cell
+}
+
+/// Ensures tags has at least one entry, adding "none" if empty.
+pub fn ensure_default_tag(tags: &mut Vec<String>) {
+    if tags.is_empty() {
+        tags.push("none".to_string());
+    }
+}
+
+/// Prints a serializable value in JSON or YAML format based on output format.
+///
+/// Only handles Json and Yaml variants; Table should be handled separately.
+pub fn print_serialized<T: serde::Serialize>(
+    value: &T,
+    format: &OutputFormat,
+) -> anyhow::Result<()> {
+    match format {
+        OutputFormat::Json => println!("{}", serde_json::to_string_pretty(value)?),
+        OutputFormat::Yaml => println!("{}", serde_yaml::to_string(value)?),
+        OutputFormat::Table => unreachable!(),
+    }
+    Ok(())
+}
+
+/// Applies config TableAttribute to a comfy-table Cell.
+pub fn apply_table_attribute(mut cell: Cell, attribute: &config::TableAttribute) -> Cell {
+    match attribute {
+        config::TableAttribute::Bold => cell = cell.add_attribute(Attribute::Bold),
+        config::TableAttribute::Dim => cell = cell.add_attribute(Attribute::Dim),
+        config::TableAttribute::Italic => cell = cell.add_attribute(Attribute::Italic),
+        config::TableAttribute::Underlined => cell = cell.add_attribute(Attribute::Underlined),
+        config::TableAttribute::SlowBlink => cell = cell.add_attribute(Attribute::SlowBlink),
+        config::TableAttribute::RapidBlink => cell = cell.add_attribute(Attribute::RapidBlink),
+        config::TableAttribute::Reverse => cell = cell.add_attribute(Attribute::Reverse),
+        config::TableAttribute::Hidden => cell = cell.add_attribute(Attribute::Hidden),
+        config::TableAttribute::CrossedOut => cell = cell.add_attribute(Attribute::CrossedOut),
+    }
+
+    cell
+}
+
+/// Builds a table showing data and database path information.
+pub fn build_path_table(path_info: &PathInfo, table_config: &config::TableConfig) -> Table {
+    let mut path_table = create_table_with_config(table_config);
+
+    path_table.set_header(vec![
+        Cell::new("Type").add_attribute(Attribute::Bold),
+        Cell::new("Path").add_attribute(Attribute::Bold),
+    ]);
+
+    path_table.add_row(vec!["Data", &path_info.data]);
+    path_table.add_row(vec!["Database", &path_info.database]);
+
+    path_table
+}
+
+/// Sanitize tags for use in filenames.
+///
+/// Replaces non-alphanumeric characters with underscores and joins with `_`.
+/// Empty tags are filtered out to avoid double underscores.
+pub fn sanitize_tags(tags: &[String]) -> String {
+    tags.iter()
+        .filter(|t| !t.is_empty())
+        .map(|t| {
+            t.chars()
+                .map(|c| if c.is_alphanumeric() { c } else { '_' })
+                .collect::<String>()
+        })
+        .collect::<Vec<_>>()
+        .join("_")
+}
+
+/// Metadata structure for export to YAML. Shared by local and client export modes.
+#[derive(Debug, Serialize)]
+pub struct ExportMeta {
+    pub ts: DateTime<Utc>,
+    pub compression: String,
+    pub uncompressed_size: Option<i64>,
+    pub tags: Vec<String>,
+    pub metadata: HashMap<String, String>,
+}
+
+/// Metadata structure for import from YAML. Shared by local and client import modes.
+#[derive(Debug, Deserialize)]
+pub struct ImportMeta {
+    pub ts: DateTime<Utc>,
+    pub compression: String,
+    #[serde(default, alias = "size")]
+    pub uncompressed_size: Option<i64>,
+    #[serde(default)]
+    pub tags: Vec<String>,
+    #[serde(default)]
+    pub metadata: HashMap<String, String>,
+}
+
+/// Resolve a single item ID from explicit IDs, tags, or latest item.
+///
+/// Returns the first ID if provided, the newest item matching tags,
+/// or the newest item overall if neither is specified.
+#[cfg(feature = "client")]
+pub fn resolve_item_id(
+    client: &crate::client::KeepClient,
+    ids: &[i64],
+    tags: &[String],
+) -> Result<i64> {
+    if !ids.is_empty() {
+        Ok(ids[0])
+    } else if !tags.is_empty() {
+        let items = client.list_items(&[], tags, "newest", 0, 1, &HashMap::new())?;
+        if items.is_empty() {
+            return Err(anyhow!("No items found matching tags: {:?}", tags));
+        }
+        Ok(items[0].id)
+    } else {
+        let items = client.list_items(&[], &[], "newest", 0, 1, &HashMap::new())?;
+        if items.is_empty() {
+            return Err(anyhow!("No items found"));
+        }
+        Ok(items[0].id)
+    }
+}
+
+/// Resolve item IDs from explicit IDs or tags (multi-item variant).
+#[cfg(feature = "client")]
+pub fn resolve_item_ids(
+    client: &crate::client::KeepClient,
+    ids: &[i64],
+    tags: &[String],
+) -> Result<Vec<i64>> {
+    if !ids.is_empty() {
+        Ok(ids.to_vec())
+    } else if !tags.is_empty() {
+        let items = client.list_items(&[], tags, "newest", 0, 0, &HashMap::new())?;
+        if items.is_empty() {
+            return Err(anyhow!("No items found matching tags: {:?}", tags));
+        }
+        Ok(items.into_iter().map(|i| i.id).collect())
+    } else {
+        let items = client.list_items(&[], &[], "newest", 0, 1, &HashMap::new())?;
+        if items.is_empty() {
+            return Err(anyhow!("No items found"));
+        }
+        Ok(vec![items[0].id])
+    }
+}
+
+/// Check if binary content should be blocked from TTY output.
+///
+/// Uses metadata `text` field as fast path, then falls back to byte sampling.
+/// Returns Err if content is binary and should not be displayed.
+pub fn check_binary_tty(
+    metadata: &HashMap<String, String>,
+    data_sample: &[u8],
+    force: bool,
+) -> Result<()> {
+    if force || !std::io::stdout().is_terminal() {
+        return Ok(());
+    }
+    if crate::common::is_binary::is_content_binary_from_metadata(metadata, data_sample) {
+        return Err(anyhow!(
+            "Refusing to output binary data to TTY, use --force to override"
+        ));
+    }
+    Ok(())
+}

src/modes/delete.rs

@@ -1,53 +1,77 @@
-use anyhow::{Context, Result, anyhow};
-use std::fs;
+use anyhow::Result;
 use std::path::PathBuf;
 
-use crate::db;
+use crate::config;
+use crate::services::error::CoreError;
+use crate::services::item_service::ItemService;
 use clap::Command;
-use clap::error::ErrorKind;
-use log::{debug, warn};
+use log::warn;
 use rusqlite::Connection;
 
+/// Handles the delete mode: removes items by ID from the database and storage.
+///
+/// This function processes a list of item IDs, attempting to delete each from
+/// both the database and the underlying file storage. It skips items that are
+/// not found and logs warnings for them. Validation of arguments (e.g., ensuring
+/// IDs are provided and tags are empty) is handled at the clap parsing level.
+///
+/// # Arguments
+///
+/// * `_cmd` - Clap command for error handling (unused).
+/// * `_settings` - Global settings (unused).
+/// * `_config` - Configuration settings (unused).
+/// * `ids` - List of item IDs to delete.
+/// * `_tags` - Tags (unused, as delete only supports IDs).
+/// * `conn` - Database connection.
+/// * `data_path` - Path to data directory for storage cleanup.
+///
+/// # Returns
+///
+/// `Result<()>` on success, or an error if deletion fails for any item.
+///
+/// # Errors
+///
+/// Returns an `anyhow::Error` if a deletion operation fails due to database
+/// or I/O issues (excluding `ItemNotFound`, which is handled gracefully).
+///
+/// # Examples
+///
+/// ```ignore
+/// // This would be called from main after parsing args
+/// mode_delete(&mut cmd, &settings, &config, &mut vec![1, 2], &mut vec![], &mut conn, data_path)?;
+/// ```
+///
+/// # Panics
+///
+/// None.
 pub fn mode_delete(
-    cmd: &mut Command,
-    _args: &crate::Args,
-    ids: &mut Vec<i64>,
-    tags: &mut Vec<String>,
+    _cmd: &mut Command,
+    _settings: &config::Settings,
+    _config: &config::Settings,
+    ids: &mut [i64],
+    _tags: &mut [String],
     conn: &mut Connection,
     data_path: PathBuf,
 ) -> Result<()> {
-    if ids.is_empty() {
-        cmd.error(
-            ErrorKind::InvalidValue,
-            "No ID given, you must supply atleast one ID when using --delete",
-        )
-        .exit();
-    } else if !tags.is_empty() {
-        cmd.error(
-            ErrorKind::InvalidValue,
-            "Tags given but not supported, you must supply atleast one ID when using --delete",
-        )
-        .exit();
-    }
+    // Validation is now handled at the argument parsing level
+    // So we can assume ids is not empty and tags is empty
+
+    let item_service = ItemService::new(data_path);
 
     for item_id in ids.iter() {
-        if let Some(item) = db::get_item(conn, *item_id)? {
-            debug!("MAIN: Found item {:?}", item);
-            db::delete_item(conn, item)?;
-
-            // Validate that item ID is positive to prevent path traversal issues
-            if *item_id <= 0 {
-                return Err(anyhow!("Invalid item ID: {}", item_id));
-            }
-            
-            let mut item_path = data_path.clone();
-            item_path.push(item_id.to_string());
-
-            fs::remove_file(&item_path)
-                .context(anyhow!("Unable to remove item file {:?}", item_path))?;
-        } else {
+        match item_service.delete_item(conn, *item_id) {
+            Ok(_) => {}
+            Err(e) => match e {
+                CoreError::ItemNotFound(_) => {
                     warn!("Unable to find item {item_id} in database");
                 }
+                _ => {
+                    return Err(
+                        anyhow::Error::from(e).context(format!("Failed to delete item {item_id}"))
+                    );
+                }
+            },
+        }
     }
 
     Ok(())

src/modes/diff.rs

@@ -1,405 +1,203 @@
-use anyhow::{anyhow, Result};
+/// Diff mode implementation.
+///
+/// This module provides functionality for comparing two items and displaying their
+/// differences using external diff tools. Decompressed content is streamed to diff
+/// via pipes and /dev/fd file descriptors — no temporary files are created.
+use crate::config;
+use crate::services::compression_service::CompressionService;
+use crate::services::item_service::ItemService;
+use anyhow::{Context, Result, anyhow};
 use clap::Command;
+use command_fds::{CommandFdExt, FdMapping};
+use log::debug;
+use nix::fcntl::OFlag;
+use nix::unistd::pipe2;
 use std::io::Read;
-use std::os::fd::FromRawFd;
-use std::str::FromStr;
+use std::os::unix::io::{AsRawFd, OwnedFd};
 
-fn validate_diff_args(cmd: &mut Command, ids: &Vec<i64>, tags: &Vec<String>) {
+fn validate_diff_args(_cmd: &mut Command, ids: &[i64], tags: &[String]) -> anyhow::Result<()> {
     if !tags.is_empty() {
-        cmd.error(
-            clap::error::ErrorKind::InvalidValue,
-            "Tags are not supported with --diff. Please provide exactly two IDs.",
-        )
-        .exit();
+        return Err(anyhow::anyhow!(
+            "Tags are not supported with --diff. Please provide exactly two IDs."
+        ));
     }
     if ids.len() != 2 {
-        cmd.error(
-            clap::error::ErrorKind::InvalidValue,
-            "You must supply exactly two IDs when using --diff.",
-        )
-        .exit();
+        return Err(anyhow::anyhow!(
+            "You must supply exactly two IDs when using --diff."
+        ));
     }
+    Ok(())
 }
 
+/// Fetches and validates items from the database for diff operation.
 fn fetch_and_validate_items(
     conn: &mut rusqlite::Connection,
-    ids: &Vec<i64>,
-) -> Result<(crate::db::Item, crate::db::Item), anyhow::Error> {
-    // Fetch items, ensuring they exist.
-    let item_a = crate::db::get_item(conn, ids[0])?
-        .ok_or_else(|| anyhow::anyhow!("Unable to find first item (ID: {}) in database", ids[0]))?;
-    let item_b = crate::db::get_item(conn, ids[1])?
-        .ok_or_else(|| anyhow::anyhow!("Unable to find second item (ID: {}) in database", ids[1]))?;
+    ids: &[i64],
+    item_service: &ItemService,
+) -> Result<(
+    crate::services::types::ItemWithMeta,
+    crate::services::types::ItemWithMeta,
+)> {
+    let item_a = item_service
+        .get_item(conn, ids[0])
+        .with_context(|| format!("Unable to find first item (ID: {}) in database", ids[0]))?;
+    let item_b = item_service
+        .get_item(conn, ids[1])
+        .with_context(|| format!("Unable to find second item (ID: {}) in database", ids[1]))?;
 
-    log::debug!("MAIN: Found item A {:?}", item_a);
-    log::debug!("MAIN: Found item B {:?}", item_b);
-
-    let item_a_id = item_a.id.ok_or_else(|| anyhow!("Item A missing ID"))?;
-    let item_b_id = item_b.id.ok_or_else(|| anyhow!("Item B missing ID"))?;
-    
-    // Validate that item IDs are positive to prevent path traversal issues
-    if item_a_id <= 0 || item_b_id <= 0 {
-        return Err(anyhow::anyhow!("Invalid item ID: {} or {}", item_a_id, item_b_id));
-    }
+    debug!("DIFF: Found item A {:?}", item_a.item);
+    debug!("DIFF: Found item B {:?}", item_b.item);
 
     Ok((item_a, item_b))
 }
 
-fn get_item_tags(conn: &mut rusqlite::Connection, item: &crate::db::Item) -> Result<Vec<String>, anyhow::Error> {
-    let tags: Vec<String> = crate::db::get_item_tags(conn, item)?
-        .into_iter()
-        .map(|x| x.name)
+pub fn mode_diff(
+    cmd: &mut Command,
+    args: &crate::args::Args,
+    conn: &mut rusqlite::Connection,
+) -> anyhow::Result<()> {
+    let ids: Vec<i64> = args
+        .ids_or_tags
+        .iter()
+        .filter_map(|x| {
+            if let crate::args::NumberOrString::Number(n) = x {
+                Some(*n)
+            } else {
+                None
+            }
+        })
         .collect();
-    Ok(tags)
-}
 
-fn setup_diff_paths_and_compression(
-    data_path: &std::path::PathBuf,
-    item_a: &crate::db::Item,
-    item_b: &crate::db::Item,
-) -> Result<(std::path::PathBuf, crate::compression_engine::CompressionType, std::path::PathBuf, crate::compression_engine::CompressionType), anyhow::Error> {
-    let item_a_id = item_a.id.ok_or_else(|| anyhow::anyhow!("Item A missing ID"))?;
-    let item_b_id = item_b.id.ok_or_else(|| anyhow::anyhow!("Item B missing ID"))?;
-
-    let mut item_path_a = data_path.clone();
-    item_path_a.push(item_a_id.to_string());
-    let compression_type_a = crate::compression_engine::CompressionType::from_str(&item_a.compression)?;
-    log::debug!("MAIN: Item A has compression type {:?}", compression_type_a);
-
-    let mut item_path_b = data_path.clone();
-    item_path_b.push(item_b_id.to_string());
-    let compression_type_b = crate::compression_engine::CompressionType::from_str(&item_b.compression)?;
-    log::debug!("MAIN: Item B has compression type {:?}", compression_type_b);
-
-    Ok((item_path_a, compression_type_a, item_path_b, compression_type_b))
-}
-
-fn setup_diff_pipes() -> Result<((libc::c_int, libc::c_int), (libc::c_int, libc::c_int)), anyhow::Error> {
-    use nix::unistd::pipe;
-    use nix::Error as NixError;
-
-    // Create pipes for diff's input
-    let (fd_a_read, fd_a_write) =
-        pipe().map_err(|e: NixError| anyhow::anyhow!("Failed to create pipe A: {}", e))?;
-    let (fd_b_read, fd_b_write) =
-        pipe().map_err(|e: NixError| anyhow::anyhow!("Failed to create pipe B: {}", e))?;
-
-    Ok(((fd_a_read, fd_a_write), (fd_b_read, fd_b_write)))
-}
-
-fn setup_fd_guards(fd_a_read: libc::c_int, fd_b_read: libc::c_int) -> (FdGuard, FdGuard) {
-    // Wrap file descriptors in RAII guards
-    let fd_a_read_guard = FdGuard::new(fd_a_read);
-    let fd_b_read_guard = FdGuard::new(fd_b_read);
-    (fd_a_read_guard, fd_b_read_guard)
-}
-
-fn set_fd_cloexec(fd_a_write: libc::c_int, fd_b_write: libc::c_int) -> Result<(), anyhow::Error> {
-    use nix::fcntl::{fcntl, FcntlArg, FdFlag};
-    
-    // Set FD_CLOEXEC on write ends
-    fcntl(
-        fd_a_write,
-        FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC),
-    )
-    .map_err(|e| anyhow::anyhow!("Failed to set FD_CLOEXEC on fd_a_write: {}", e))?;
-    fcntl(
-        fd_b_write,
-        FcntlArg::F_SETFD(FdFlag::FD_CLOEXEC),
-    )
-    .map_err(|e| anyhow::anyhow!("Failed to set FD_CLOEXEC on fd_b_write: {}", e))?;
-    
-    Ok(())
-}
-
-fn spawn_diff_process(
-    item_a_id: i64,
-    item_a_tags: Vec<String>,
-    item_b_id: i64,
-    item_b_tags: Vec<String>,
-    fd_a_read: libc::c_int,
-    fd_b_read: libc::c_int,
-) -> Result<std::process::Child, anyhow::Error> {
-    log::debug!("MAIN: Creating child process for diff");
-    let mut diff_command = std::process::Command::new("diff");
-    diff_command
-        .arg("-u")
-        .arg("--label")
-        .arg(format!(
-            "Keep item A: {} {}",
-            item_a_id,
-            item_a_tags.join(" ")
-        ))
-        .arg(format!("/dev/fd/{}", fd_a_read))
-        .arg("--label")
-        .arg(format!(
-            "Keep item B: {} {}",
-            item_b_id,
-            item_b_tags.join(" ")
-        ))
-        .arg(format!("/dev/fd/{}", fd_b_read))
-        .stdin(std::process::Stdio::null())
-        .stdout(std::process::Stdio::piped())
-        .stderr(std::process::Stdio::piped());
-
-    let child_process = diff_command
-        .spawn()
-        .map_err(|e| anyhow::anyhow!("Failed to execute diff command: {}", e))?;
-
-    Ok(child_process)
-}
-
-// RAII guard for file descriptors to ensure they're closed
-struct FdGuard {
-    fd: libc::c_int,
-}
-
-impl FdGuard {
-    fn new(fd: libc::c_int) -> Self {
-        Self { fd }
+    let tags: Vec<String> = args
+        .ids_or_tags
+        .iter()
+        .filter_map(|x| {
+            if let crate::args::NumberOrString::Str(s) = x {
+                Some(s.clone())
+            } else {
+                None
             }
+        })
+        .collect();
+
+    validate_diff_args(cmd, &ids, &tags)?;
+
+    let settings = config::Settings::new(args, config::Settings::default_dir()?)?;
+    let item_service = ItemService::new(settings.dir.clone());
+    let (item_a, item_b) = fetch_and_validate_items(conn, &ids, &item_service)?;
+
+    run_external_diff(&item_service, &item_a, &item_b)
 }
 
-impl Drop for FdGuard {
-    fn drop(&mut self) {
-        let _ = nix::unistd::close(self.fd);
-    }
+/// Creates a pipe with CLOEXEC set atomically, returns (read_fd, write_fd).
+fn create_pipe() -> Result<(OwnedFd, OwnedFd)> {
+    pipe2(OFlag::O_CLOEXEC).context("Failed to create pipe")
 }
 
-// Create a function to write item data to a pipe
-fn write_item_to_pipe(
-    item_path: std::path::PathBuf,
-    compression_type: crate::compression_engine::CompressionType,
-    pipe_writer_raw: std::fs::File,
-) -> Result<(), anyhow::Error> {
-    use std::io::BufWriter;
-    let mut buffered_pipe_writer = BufWriter::new(pipe_writer_raw);
-    let engine =
-        crate::compression_engine::get_compression_engine(compression_type).expect("Unable to get compression engine");
-    log::debug!("THREAD: Sending item to diff");
-    engine
-        .copy(item_path, &mut buffered_pipe_writer)
-        .map_err(|e| anyhow::anyhow!("Failed to copy/compress item: {}", e))?;
-    log::debug!("THREAD: Done sending item to diff");
-    Ok(())
-}
-
-// Function to spawn a writer thread for an item
+/// Streams decompressed item content through a pipe fd.
+///
+/// Returns a JoinHandle for the writer thread. The thread writes decompressed
+/// data to write_fd and closes it when done (causing EOF for the reader).
 fn spawn_writer_thread(
-    item_path: std::path::PathBuf,
-    compression_type: crate::compression_engine::CompressionType,
-    fd_write: libc::c_int,
-) -> std::thread::JoinHandle<Result<(), anyhow::Error>> {
-    let pipe_writer_raw = unsafe { std::fs::File::from_raw_fd(fd_write) };
-    std::thread::spawn(move || {
-        write_item_to_pipe(item_path, compression_type, pipe_writer_raw)
+    item_service: &ItemService,
+    item: &crate::services::types::ItemWithMeta,
+    write_fd: OwnedFd,
+) -> std::thread::JoinHandle<Result<()>> {
+    let data_path = item_service.get_data_path().clone();
+    let id = match item.item.id {
+        Some(id) => id,
+        None => return std::thread::spawn(|| Err(anyhow!("item missing ID"))),
+    };
+    let compression = item.item.compression.clone();
+    let mut item_path = data_path;
+    item_path.push(id.to_string());
+
+    std::thread::spawn(move || -> Result<()> {
+        let compression_service = CompressionService::new();
+        let mut reader = compression_service
+            .stream_item_content(item_path, &compression)
+            .map_err(|e| anyhow::anyhow!("Failed to stream item {id}: {e}"))?;
+
+        // Convert OwnedFd to File — safe, takes ownership, closes on drop
+        let mut writer = std::fs::File::from(write_fd);
+        crate::common::stream_copy(&mut reader, |chunk| {
+            use std::io::Write;
+            writer.write_all(chunk)
+        })
+        .map_err(|e| anyhow::anyhow!("Error reading item {id}: {e}"))?;
+        // writer dropped here, closing write_fd → diff sees EOF
+        Ok(())
     })
 }
 
-fn execute_diff_command(
-    child_process: &mut std::process::Child,
-) -> Result<(Vec<u8>, Vec<u8>), anyhow::Error> {
-    let mut child_stdout_pipe = child_process
-        .stdout
-        .take()
-        .expect("BUG: Failed to capture diff stdout pipe");
-    let mut child_stderr_pipe = child_process
-        .stderr
-        .take()
-        .expect("BUG: Failed to capture diff stderr pipe");
+/// Runs external diff command, streaming decompressed content via /dev/fd pipes.
+///
+/// Creates two pipes, spawns writer threads to decompress each item into its pipe,
+/// and runs `diff -u /dev/fd/N /dev/fd/M` where N and M are the pipe read fds.
+/// The `command-fds` crate handles CLOEXEC clearing safely — no unsafe needed.
+fn run_external_diff(
+    item_service: &ItemService,
+    item_a: &crate::services::types::ItemWithMeta,
+    item_b: &crate::services::types::ItemWithMeta,
+) -> Result<()> {
+    if which::which_global("diff").is_err() {
+        return Err(anyhow::anyhow!(
+            "diff command not found. Please install diffutils."
+        ));
+    }
 
-    log::debug!("MAIN: Creating threads for diff I/O");
+    let (read_fd_a, write_fd_a) = create_pipe()?;
+    let (read_fd_b, write_fd_b) = create_pipe()?;
 
-    // Thread to read diff's standard output
-    let stdout_reader_thread = std::thread::spawn(move || {
-        let mut output_buffer = Vec::new();
-        log::debug!("STDOUT_READER: Reading diff stdout");
-        // child_stdout_pipe is a ChildStdout, which implements std::io::Read
-        child_stdout_pipe
-            .read_to_end(&mut output_buffer)
-            .map_err(|e| anyhow::anyhow!("Failed to read diff stdout: {}", e))
-            .map(|_| output_buffer) // Return the Vec<u8> on success
-    });
+    // Spawn writer threads — they take ownership of write fds and close them on exit
+    let writer_a = spawn_writer_thread(item_service, item_a, write_fd_a);
+    let writer_b = spawn_writer_thread(item_service, item_b, write_fd_b);
 
-    // Thread to read diff's standard error
-    let stderr_reader_thread = std::thread::spawn(move || {
-        let mut error_buffer = Vec::new();
-        log::debug!("STDERR_READER: Reading diff stderr");
-        child_stderr_pipe
-            .read_to_end(&mut error_buffer)
-            .map_err(|e| anyhow::anyhow!("Failed to read diff stderr: {}", e))
-            .map(|_| error_buffer)
-    });
+    // Get fd numbers for /dev/fd paths (borrows, does not consume)
+    let raw_read_a = read_fd_a.as_raw_fd();
+    let raw_read_b = read_fd_b.as_raw_fd();
 
-    // Retrieve the captured output from the reader threads.
-    let stdout_capture_result = stdout_reader_thread
+    debug!("DIFF: pipe fds: a(r={raw_read_a}) b(r={raw_read_b})");
+
+    // Spawn diff with /dev/fd/N paths. command-fds handles CLOEXEC clearing
+    // and fd inheritance safely — the fds are released from OwnedFd to the
+    // child process. If spawn fails, the OwnedFd values in FdMapping are
+    // dropped and the fds are properly closed.
+    let mut command = std::process::Command::new("diff");
+    command
+        .arg("-u")
+        .arg(format!("/dev/fd/{raw_read_a}"))
+        .arg(format!("/dev/fd/{raw_read_b}"))
+        .stdout(std::process::Stdio::inherit())
+        .stderr(std::process::Stdio::inherit())
+        .stdin(std::process::Stdio::null())
+        .fd_mappings(vec![
+            FdMapping {
+                parent_fd: read_fd_a,
+                child_fd: raw_read_a,
+            },
+            FdMapping {
+                parent_fd: read_fd_b,
+                child_fd: raw_read_b,
+            },
+        ])
+        .map_err(|e| anyhow::anyhow!("FD mapping collision: {e}"))?;
+    let mut child = command.spawn().context("Failed to spawn diff command")?;
+
+    let status = child.wait().context("Failed to wait for diff command")?;
+
+    // Join writer threads and propagate errors
+    writer_a
         .join()
-        .map_err(|panic_payload| {
-            anyhow::anyhow!("Stdout reader thread panicked: {:?}", panic_payload)
-        })?
-        .map_err(|e| anyhow::anyhow!("Failed to read diff stdout: {}", e))?;
-
-    let stderr_capture_result = stderr_reader_thread
+        .map_err(|e| anyhow::anyhow!("Writer A panicked: {e:?}"))??;
+    writer_b
         .join()
-        .map_err(|panic_payload| {
-            anyhow::anyhow!("Stderr reader thread panicked: {:?}", panic_payload)
-        })?
-        .map_err(|e| anyhow::anyhow!("Failed to read diff stderr: {}", e))?;
-
-    Ok((stdout_capture_result, stderr_capture_result))
-}
-
-fn handle_diff_output(
-    diff_status: std::process::ExitStatus,
-    stdout_capture_result: Vec<u8>,
-    stderr_capture_result: Vec<u8>,
-) -> Result<(), anyhow::Error> {
-    // Handle diff's exit status and output
-    match diff_status.code() {
-        Some(0) => {
-            // Exit code 0: No differences
-            log::debug!("MAIN: Diff successful, no differences found.");
-            // Typically, diff -u doesn't print to stdout if no differences.
-            // But if it did, it would be shown here.
-            if !stdout_capture_result.is_empty() {
-                println!("{}", String::from_utf8_lossy(&stdout_capture_result));
-            }
-        }
-        Some(1) => {
-            // Exit code 1: Differences found
-            log::debug!("MAIN: Diff successful, differences found.");
-            println!("{}", String::from_utf8_lossy(&stdout_capture_result));
-        }
-        Some(error_code) => {
-            // Exit code > 1: Error in diff utility
-            eprintln!("Diff command failed with exit code: {}", error_code);
-            if !stdout_capture_result.is_empty() {
-                eprintln!(
-                    "Diff stdout before error:\n{}",
-                    String::from_utf8_lossy(&stdout_capture_result)
-                );
-            }
-            if !stderr_capture_result.is_empty() {
-                eprintln!(
-                    "Diff stderr:\n{}",
-                    String::from_utf8_lossy(&stderr_capture_result)
-                );
-            }
-            return Err(anyhow::anyhow!(
-                "Diff command reported an error (exit code {})",
-                error_code
-            ));
-        }
-        None => {
-            // Process terminated by a signal
-            eprintln!("Diff command terminated by signal.");
-            if !stderr_capture_result.is_empty() {
-                eprintln!(
-                    "Diff stderr before signal termination:\n{}",
-                    String::from_utf8_lossy(&stderr_capture_result)
-                );
-            }
-            return Err(anyhow::anyhow!("Diff command terminated by signal"));
-        }
-    }
-
-    Ok(())
-}
-
-pub fn mode_diff(
-    cmd: &mut Command,
-    _args: &crate::Args,
-    ids: &mut Vec<i64>,
-    tags: &mut Vec<String>,
-    conn: &mut rusqlite::Connection,
-    data_path: std::path::PathBuf,
-) -> Result<(), anyhow::Error> {
-    validate_diff_args(cmd, ids, tags);
-    let (item_a, item_b) = fetch_and_validate_items(conn, ids)?;
-
-    let item_a_tags = get_item_tags(conn, &item_a)?;
-    let item_b_tags = get_item_tags(conn, &item_b)?;
-
-    let (item_path_a, compression_type_a, item_path_b, compression_type_b) = 
-        setup_diff_paths_and_compression(&data_path, &item_a, &item_b)?;
-
-    let ((fd_a_read, fd_a_write), (fd_b_read, fd_b_write)) = setup_diff_pipes()?;
-    let (_fd_a_read_guard, _fd_b_read_guard) = setup_fd_guards(fd_a_read, fd_b_read);
-    set_fd_cloexec(fd_a_write, fd_b_write)?;
-
-    let item_a_id = item_a.id.ok_or_else(|| anyhow::anyhow!("Item A missing ID"))?;
-    let item_b_id = item_b.id.ok_or_else(|| anyhow::anyhow!("Item B missing ID"))?;
-
-    let mut child_process = spawn_diff_process(
-        item_a_id,
-        item_a_tags,
-        item_b_id,
-        item_b_tags,
-        fd_a_read,
-        fd_b_read,
-    )?;
-
-    // Close read ends in parent process - they're now guarded by FdGuard
-    drop(_fd_a_read_guard);
-    drop(_fd_b_read_guard);
-
-    // Spawn writer threads for both items
-    let writer_thread_a =
-        spawn_writer_thread(item_path_a.clone(), compression_type_a.clone(), fd_a_write);
-
-    let writer_thread_b =
-        spawn_writer_thread(item_path_b.clone(), compression_type_b.clone(), fd_b_write);
-
-    // Wait for writer threads to complete (meaning all input has been sent to diff)
-    log::debug!("MAIN: Waiting on writer thread for item A");
-    match writer_thread_a.join() {
-        Ok(Ok(())) => {
-            log::debug!("MAIN: Writer thread for item A completed successfully.");
-        }
-        Ok(Err(e)) => {
-            return Err(anyhow::anyhow!("Writer thread for item A failed: {}", e));
-        }
-        Err(panic_payload) => {
-            return Err(anyhow::anyhow!(
-                "Writer thread for item A (ID: {}) panicked: {:?}",
-                ids[0],
-                panic_payload
-            ));
-        }
-    }
-
-    log::debug!("MAIN: Waiting on writer thread for item B");
-    match writer_thread_b.join() {
-        Ok(Ok(())) => {
-            log::debug!("MAIN: Writer thread for item B completed successfully.");
-        }
-        Ok(Err(e)) => {
-            return Err(anyhow::anyhow!("Writer thread for item B failed: {}", e));
-        }
-        Err(panic_payload) => {
-            return Err(anyhow::anyhow!(
-                "Writer thread for item B (ID: {}) panicked: {:?}",
-                ids[1],
-                panic_payload
-            ));
-        }
-    }
-    
-    log::debug!("MAIN: Done waiting on input-writer threads.");
-
-    // Now that all input has been sent and input pipes will be closed by threads exiting,
-    // wait for the diff child process to terminate.
-    log::debug!("MAIN: Waiting for diff child process to finish...");
-    let diff_status = child_process
-        .wait()
-        .map_err(|e| anyhow::anyhow!("Failed to wait on diff command: {}", e))?;
-    log::debug!(
-        "MAIN: Diff child process finished with status: {}",
-        diff_status
-    );
-
-    let (stdout_capture_result, stderr_capture_result) = execute_diff_command(&mut child_process)?;
-    handle_diff_output(diff_status, stdout_capture_result, stderr_capture_result)?;
+        .map_err(|e| anyhow::anyhow!("Writer B panicked: {e:?}"))??;
 
+    // diff returns 0 if identical, 1 if different, 2 on error
+    if status.code() == Some(2) {
+        Err(anyhow::anyhow!("diff command failed with an error"))
+    } else {
         Ok(())
+    }
 }

src/modes/export.rs

@@ -0,0 +1,145 @@
+use anyhow::{Context, Result, anyhow};
+use chrono::Utc;
+use clap::Command;
+use log::debug;
+use std::collections::HashMap;
+use std::fs;
+use std::path::PathBuf;
+
+use crate::common::sanitize_ts_string;
+use crate::config;
+use crate::export_tar;
+use crate::filter_plugin::FilterChain;
+use crate::modes::common::sanitize_tags;
+use crate::services::item_service::ItemService;
+use crate::services::types::ItemWithMeta;
+
+/// Export items to a `.keep.tar` archive.
+///
+/// Requires either IDs or tags (mutually exclusive). If IDs are given,
+/// ALL must exist. Archives contain per-item data and metadata files.
+pub fn mode_export(
+    cmd: &mut Command,
+    settings: &config::Settings,
+    ids: &[i64],
+    tags: &[String],
+    conn: &mut rusqlite::Connection,
+    data_path: PathBuf,
+    filter_chain: Option<FilterChain>,
+) -> Result<()> {
+    // Validate: IDs XOR tags
+    if !ids.is_empty() && !tags.is_empty() {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "Cannot use both IDs and tags with --export",
+        )
+        .exit();
+    }
+    if ids.is_empty() && tags.is_empty() {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "Must provide either IDs or tags with --export",
+        )
+        .exit();
+    }
+
+    let item_service = ItemService::new(data_path.clone());
+    let meta_filter = settings.meta_filter();
+
+    // Resolve items
+    let items: Vec<ItemWithMeta> = if !ids.is_empty() {
+        // Fetch each ID individually; ALL must exist
+        let mut result = Vec::new();
+        for &id in ids {
+            match item_service.get_item(conn, id) {
+                Ok(item) => result.push(item),
+                Err(_) => {
+                    cmd.error(
+                        clap::error::ErrorKind::InvalidValue,
+                        format!("Item {id} not found"),
+                    )
+                    .exit();
+                }
+            }
+        }
+        result
+    } else {
+        // Search by tags
+        item_service
+            .list_items(conn, tags, &meta_filter)
+            .map_err(|e| anyhow!("Unable to find matching items: {}", e))?
+    };
+
+    if items.is_empty() {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "No items found matching the given criteria",
+        )
+        .exit();
+    }
+
+    // Validate: --export-filename-format doesn't use per-item vars with multiple items
+    if items.len() > 1 {
+        let fmt = &settings.export_filename_format;
+        if fmt.contains("{id}") || fmt.contains("{tags}") || fmt.contains("{compression}") {
+            cmd.error(
+                clap::error::ErrorKind::InvalidValue,
+                "Cannot use {id}, {tags}, or {compression} in --export-filename-format when exporting multiple items",
+            )
+            .exit();
+        }
+    }
+
+    // Compute export name
+    let dir_name = export_tar::export_name(&settings.export_name, &items);
+
+    // Compute tar filename from format template
+    let now = Utc::now();
+    let ts_str = sanitize_ts_string(&now.format("%Y-%m-%dT%H:%M:%SZ").to_string());
+
+    let mut vars = HashMap::new();
+    vars.insert("name".to_string(), dir_name.clone());
+    vars.insert("ts".to_string(), ts_str.clone());
+
+    // For single-item exports, also provide per-item vars
+    if items.len() == 1 {
+        let item = &items[0];
+        let item_id = item.item.id.context("Item missing ID")?;
+        let item_tags = item.tag_names();
+        vars.insert("id".to_string(), item_id.to_string());
+        vars.insert("tags".to_string(), sanitize_tags(&item_tags));
+        vars.insert("compression".to_string(), item.item.compression.clone());
+    }
+
+    let basename = strfmt::strfmt(&settings.export_filename_format, &vars).map_err(|e| {
+        anyhow!(
+            "Invalid export filename format '{}': {}",
+            settings.export_filename_format,
+            e
+        )
+    })?;
+
+    let tar_filename = format!("{basename}.keep.tar");
+
+    // Write the tar archive
+    let tar_file = fs::File::create(&tar_filename)
+        .with_context(|| format!("Cannot create tar file: {tar_filename}"))?;
+
+    export_tar::write_export_tar(
+        tar_file,
+        &dir_name,
+        &items,
+        &data_path,
+        filter_chain.as_ref(),
+        &item_service,
+        conn,
+    )?;
+
+    if !settings.quiet {
+        eprintln!("{tar_filename}");
+    }
+
+    debug!("EXPORT: Wrote {} items to {tar_filename}", items.len());
+
+    Ok(())
+}

src/modes/generate_config.rs

@@ -0,0 +1,264 @@
+use anyhow::Result;
+use clap::Command;
+use std::collections::HashMap;
+use strum::IntoEnumIterator;
+
+use crate::common::schema::{gather_filter_plugin_schemas, gather_meta_plugin_schemas};
+use crate::compression_engine::CompressionType;
+use crate::config;
+
+/// Generates and prints a default commented YAML configuration template.
+///
+/// Discovers all registered meta plugins, filter plugins, and compression engines
+/// at runtime via the plugin schema system. Outputs a commented YAML template
+/// with all available plugins and their default options/outputs.
+///
+/// # Arguments
+///
+/// * `_cmd` - Unused Clap command reference.
+/// * `_settings` - Unused settings reference.
+///
+/// # Returns
+///
+/// `Ok(())` on success.
+pub fn mode_generate_config(_cmd: &mut Command, _settings: &crate::config::Settings) -> Result<()> {
+    let meta_schemas = gather_meta_plugin_schemas();
+    let filter_schemas = gather_filter_plugin_schemas();
+
+    // Build list_format defaults matching config.rs
+    let list_format = default_list_format();
+
+    // Build meta_plugins with env as the default (active), rest commented
+    let meta_plugins = build_meta_plugins_section(&meta_schemas);
+
+    // Build the full YAML
+    let mut lines = Vec::with_capacity(128);
+
+    lines.push("# Keep configuration file".to_string());
+    lines.push("# Uncomment and modify the settings you need.".to_string());
+    lines.push(String::new());
+
+    // Core settings
+    lines.push("# Data directory for storing items".to_string());
+    lines.push("dir: ~/.local/share/keep".to_string());
+    lines.push(String::new());
+
+    // List format
+    lines.push("# Column configuration for --list output".to_string());
+    lines.push("list_format:".to_string());
+    for col in &list_format {
+        lines.push(format!("  - name: {}", col.name));
+        lines.push(format!("    label: {}", col.label));
+        lines.push(format!("    align: {}", col.align));
+    }
+    lines.push(String::new());
+
+    // Table config
+    lines.push("# Table display configuration".to_string());
+    lines.push("#table_config:".to_string());
+    lines.push("#  style: nothing".to_string());
+    lines.push("#  modifiers: []".to_string());
+    lines.push("#  content_arrangement: dynamic".to_string());
+    lines.push("#  truncination_indicator: \"\"".to_string());
+    lines.push(String::new());
+
+    // Other settings
+    lines.push("human_readable: false".to_string());
+    lines.push("output_format: table".to_string());
+    lines.push("quiet: false".to_string());
+    lines.push("force: false".to_string());
+    lines.push(String::new());
+
+    // Server config
+    lines.push("# Server configuration (only used with --server)".to_string());
+    lines.push("server:".to_string());
+    lines.push("  address: 127.0.0.1".to_string());
+    lines.push("  port: 8080".to_string());
+    lines.push("#  username: keep".to_string());
+    lines.push("#  password: null".to_string());
+    lines.push("#  password_file: null".to_string());
+    lines.push("#  password_hash: null".to_string());
+    lines.push("#  jwt_secret: null".to_string());
+    lines.push("#  jwt_secret_file: null".to_string());
+    lines.push("#  cert_file: null".to_string());
+    lines.push("#  key_file: null".to_string());
+    lines.push("#  cors_origin: null".to_string());
+    lines.push(String::new());
+
+    // Compression plugin
+    lines.push("# Compression plugin to use".to_string());
+    lines.push("#compression_plugin:".to_string());
+    let mut comp_types: Vec<String> = CompressionType::iter().map(|ct| ct.to_string()).collect();
+    comp_types.sort();
+    for ct in &comp_types {
+        lines.push(format!("#  name: {ct}  # {}", compression_description(ct)));
+    }
+    lines.push(String::new());
+
+    // Meta plugins
+    lines.push("# Meta plugins to run when saving items".to_string());
+    lines.push("meta_plugins:".to_string());
+    for line in &meta_plugins {
+        lines.push(line.clone());
+    }
+    lines.push(String::new());
+
+    // Filter plugins reference
+    if !filter_schemas.is_empty() {
+        lines.push("# Available filter plugins (use with --filter)".to_string());
+        for schema in &filter_schemas {
+            lines.push(format!("#   {}", schema.name));
+            if !schema.description.is_empty() {
+                lines.push(format!("#     {}", schema.description));
+            }
+            for opt in &schema.options {
+                let req = if opt.required { "required" } else { "optional" };
+                lines.push(format!(
+                    "#       {} ({:?}, {})",
+                    opt.name, opt.option_type, req
+                ));
+            }
+        }
+        lines.push(String::new());
+    }
+
+    // Client config
+    lines.push("# Client configuration (requires client feature)".to_string());
+    lines.push("#client:".to_string());
+    lines.push("#  url: null".to_string());
+    lines.push("#  username: null".to_string());
+    lines.push("#  password: null".to_string());
+    lines.push("#  jwt: null".to_string());
+
+    // Print
+    for line in &lines {
+        println!("{line}");
+    }
+
+    Ok(())
+}
+
+struct ListColumn {
+    name: String,
+    label: String,
+    align: String,
+}
+
+fn default_list_format() -> Vec<ListColumn> {
+    vec![
+        ListColumn {
+            name: "id".into(),
+            label: "Item".into(),
+            align: "right".into(),
+        },
+        ListColumn {
+            name: "time".into(),
+            label: "Time".into(),
+            align: "right".into(),
+        },
+        ListColumn {
+            name: "size".into(),
+            label: "Size".into(),
+            align: "right".into(),
+        },
+        ListColumn {
+            name: "meta:text_line_count".into(),
+            label: "Lines".into(),
+            align: "right".into(),
+        },
+        ListColumn {
+            name: "tags".into(),
+            label: "Tags".into(),
+            align: "left".into(),
+        },
+        ListColumn {
+            name: "meta:hostname_short".into(),
+            label: "Host".into(),
+            align: "left".into(),
+        },
+        ListColumn {
+            name: "meta:command".into(),
+            label: "Command".into(),
+            align: "left".into(),
+        },
+    ]
+}
+
+fn build_meta_plugins_section(schemas: &[crate::common::schema::PluginSchema]) -> Vec<String> {
+    let mut lines = Vec::new();
+
+    for (i, schema) in schemas.iter().enumerate() {
+        let is_default = schema.name == "env";
+        let prefix = if is_default { "" } else { "# " };
+
+        if i > 0 {
+            lines.push(format!("{prefix}# --- {name} ---", name = schema.name));
+        }
+
+        lines.push(format!("{prefix}- name: {}", schema.name));
+
+        // Options
+        if !schema.options.is_empty() {
+            lines.push(format!("{prefix}  options:"));
+            for opt in &schema.options {
+                if let Some(ref default) = opt.default {
+                    let default_str = format_yaml_value(default);
+                    lines.push(format!("{prefix}    {}: {}", opt.name, default_str));
+                } else if opt.required {
+                    lines.push(format!("{prefix}    {}: null  # required", opt.name));
+                }
+            }
+        } else {
+            lines.push(format!("{prefix}  options: {{}}"));
+        }
+
+        // Outputs
+        if !schema.outputs.is_empty() {
+            lines.push(format!("{prefix}  outputs:"));
+            for output in &schema.outputs {
+                lines.push(format!("{prefix}    {}: {}", output.name, output.name));
+            }
+        } else {
+            lines.push(format!("{prefix}  outputs: {{}}"));
+        }
+    }
+
+    lines
+}
+
+fn format_yaml_value(value: &serde_yaml::Value) -> String {
+    match value {
+        serde_yaml::Value::Null => "null".into(),
+        serde_yaml::Value::Bool(b) => b.to_string(),
+        serde_yaml::Value::Number(n) => n.to_string(),
+        serde_yaml::Value::String(s) => {
+            if s.contains(' ') || s.contains(':') || s.contains('#') {
+                format!("\"{s}\"")
+            } else {
+                s.clone()
+            }
+        }
+        serde_yaml::Value::Sequence(_) | serde_yaml::Value::Mapping(_) => {
+            serde_yaml::to_string(value)
+                .unwrap_or_default()
+                .trim()
+                .to_string()
+        }
+        serde_yaml::Value::Tagged(_) => serde_yaml::to_string(value)
+            .unwrap_or_default()
+            .trim()
+            .to_string(),
+    }
+}
+
+fn compression_description(name: &str) -> &str {
+    match name {
+        "lz4" => "Fast compression (native)",
+        "gzip" => "Good compression ratio (native)",
+        "bzip2" => "High compression (requires bzip2 binary)",
+        "xz" => "Very high compression (requires xz binary)",
+        "zstd" => "Modern fast compression (requires zstd binary)",
+        "raw" => "No compression (alias: none)",
+        _ => "",
+    }
+}

src/modes/get.rs

@@ -1,114 +1,111 @@
-use anyhow::anyhow;
-use std::io::{Read, Write};
+use anyhow::{Context, Result, anyhow};
+use std::io::Write;
 
-use crate::compression_engine::{CompressionType, get_compression_engine};
-use crate::common::is_binary;
+use crate::common::PIPESIZE;
+use crate::common::is_binary::is_binary;
+use crate::config;
+use crate::filter_plugin::FilterChain;
+use crate::services::item_service::ItemService;
 use clap::Command;
+use is_terminal::IsTerminal;
+use std::io::Read;
 use std::path::PathBuf;
-use std::str::FromStr;
 
+/// Handles the get mode: retrieves and streams item content to stdout, applying filters if specified.
+///
+/// # Arguments
+///
+/// * `cmd` - Clap command for error handling.
+/// * `settings` - Global settings, including force output flag.
+/// * `ids` - List of item IDs (at most one).
+/// * `tags` - List of tags to match (mutually exclusive with IDs).
+/// * `conn` - Database connection.
+/// * `data_path` - Path to data directory.
+/// * `filter_chain` - Optional pre-parsed filter chain to apply to content.
+///
+/// # Returns
+///
+/// `Result<()>` on success, or an error if item not found or output fails.
 pub fn mode_get(
     cmd: &mut Command,
-    args: &crate::Args,
-    ids: &mut Vec<i64>,
-    tags: &mut Vec<String>,
+    settings: &config::Settings,
+    ids: &mut [i64],
+    tags: &mut [String],
     conn: &mut rusqlite::Connection,
     data_path: PathBuf,
-) -> anyhow::Result<()> {
+    filter_chain: Option<FilterChain>,
+) -> Result<()> {
     if !ids.is_empty() && !tags.is_empty() {
-        cmd.error(clap::error::ErrorKind::InvalidValue, "Both ID and tags given, you must supply exactly one ID or at least one tag when using --get").exit();
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "Both ID and tags given, you must supply either IDs or tags when using --get",
+        )
+        .exit();
     } else if ids.len() > 1 {
-        cmd.error(clap::error::ErrorKind::InvalidValue, "More than one ID given, you must supply exactly one ID or at least one tag when using --get").exit();
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "More than one ID given, you must supply exactly one ID when using --get",
+        )
+        .exit();
     }
+    // If both are empty, find_item will find the last item
 
-    let mut meta: std::collections::HashMap<String, String> = std::collections::HashMap::new();
-    for item in args.item.meta.iter() {
-        let item = item.clone();
-        meta.insert(item.key, item.value);
-    }
+    let item_service = ItemService::new(data_path.clone());
+    let item_with_meta = item_service
+        .find_item(conn, ids, tags, &settings.meta_filter())
+        .map_err(|e| anyhow!("Unable to find matching item in database: {}", e))?;
 
-    let item_maybe = match tags.is_empty() && meta.is_empty() {
-        true => match ids.iter().next() {
-            Some(item_id) => crate::db::get_item(conn, *item_id)?,
-            None => crate::db::get_item_last(conn)?,
-        },
-        false => crate::db::get_item_matching(conn, tags, &meta)?,
-    };
-
-    if let Some(item) = item_maybe {
-        let item_id = item.id.ok_or_else(|| anyhow!("Item missing ID"))?;
-        // Validate that item ID is positive to prevent path traversal issues
-        if item_id <= 0 {
-            return Err(anyhow!("Invalid item ID: {}", item_id));
-        }
-        
-        let mut item_path = data_path.clone();
-        item_path.push(item_id.to_string());
+    let item_id = item_with_meta.item.id.context("Item missing ID")?;
 
     // Determine if we should detect binary data
-        let mut detect_binary = !args.options.force && is_stdout_tty();
+    let mut detect_binary = !settings.force && std::io::stdout().is_terminal();
 
-        // If we're detecting binary and there's binary metadata, check it
     if detect_binary {
-            let item_meta = crate::db::get_item_meta(conn, &item)?;
-            let binary_meta = item_meta.into_iter().find(|meta| meta.name == "binary");
-            if let Some(binary_meta) = binary_meta {
-                if binary_meta.value == "false" {
-                    // If metadata says it's not binary, don't detect
+        let meta_map = item_with_meta.meta_as_map();
+        if let Some(text_val) = meta_map.get("text") {
+            if text_val == "true" {
                 detect_binary = false;
-                } else if binary_meta.value == "true" {
-                    // If metadata says it's binary, error immediately
-                    return Err(anyhow!("Refusing to output binary data to TTY, use --force to override"));
+            } else if text_val == "false" {
+                return Err(anyhow!(
+                    "Refusing to output binary data to TTY, use --force to override"
+                ));
             }
         }
     }
 
-        let compression_type = CompressionType::from_str(&item.compression)?;
-        let compression_engine = get_compression_engine(compression_type)?;
-        
-        // If we need to detect binary, read first 4KB and check
     if detect_binary {
-            // Open the file through compression engine to read first 4KB
-            let mut reader = compression_engine.open(item_path.clone())?;
-            let mut buffer = [0u8; 4096];
-            let bytes_read = reader.read(&mut buffer)?;
-            
-            // Check if this data is binary
-            if is_binary(&buffer[..bytes_read]) {
-                return Err(anyhow!("Refusing to output binary data to TTY, use --force to override"));
+        // Binary detection: sample first 8KB, then create a fresh reader for the full output.
+        let (mut sample_reader, _, _) = item_service
+            .get_item_content_info_streaming_with_item(item_with_meta, filter_chain.as_ref())?;
+        let mut sample_buffer = vec![0; PIPESIZE];
+        let bytes_read = sample_reader.read(&mut sample_buffer)?;
+        if is_binary(&sample_buffer[..bytes_read]) {
+            return Err(anyhow!(
+                "Refusing to output binary data to TTY, use --force to override"
+            ));
         }
-            
-            // If not binary, output the data we've read
-            std::io::stdout().write_all(&buffer[..bytes_read])?;
-            
-            // Continue reading and outputting the rest of the data
-            let mut stdout = std::io::stdout();
-            std::io::copy(&mut reader, &mut stdout)?;
+        // Create fresh reader for actual output (sampling consumed the first reader)
+        let (reader, _, _) = item_service.get_item_content_info_streaming_with_chain(
+            conn,
+            item_id,
+            filter_chain.as_ref(),
+        )?;
+        stream_to_stdout(reader)?;
     } else {
-            // No binary detection needed, just output the data
-            compression_engine.cat(item_path.clone())?;
+        // No binary detection needed, use the already-fetched item with meta
+        let (reader, _, _) = item_service
+            .get_item_content_info_streaming_with_item(item_with_meta, filter_chain.as_ref())?;
+        stream_to_stdout(reader)?;
     }
 
     Ok(())
-    } else {
-        Err(anyhow!("Unable to find matching item in database"))
-    }
 }
 
-fn is_stdout_tty() -> bool {
-    #[cfg(unix)]
-    unsafe {
-        libc::isatty(libc::STDOUT_FILENO) != 0
-    }
-    
-    #[cfg(windows)]
-    unsafe {
-        let stdout_handle = winapi::um::processenv::GetStdHandle(winapi::um::winbase::STD_OUTPUT_HANDLE);
-        let mut console_mode: winapi::shared::minwindef::DWORD = 0;
-        winapi::um::consoleapi::GetConsoleMode(stdout_handle, &mut console_mode) != 0
-    }
-    
-    // Fallback for non-unix platforms or if we can't determine
-    #[cfg(not(any(unix, windows)))]
-    false
+fn stream_to_stdout(mut reader: Box<dyn Read + Send>) -> Result<()> {
+    let mut stdout = std::io::stdout();
+    crate::common::stream_copy(&mut reader, |chunk| {
+        stdout.write_all(chunk)?;
+        Ok(())
+    })?;
+    Ok(())
 }

src/modes/import.rs

@@ -0,0 +1,192 @@
+use anyhow::{Context, Result, anyhow};
+use chrono::{DateTime, Utc};
+use clap::Command;
+use log::debug;
+use std::collections::HashMap;
+use std::fs;
+use std::io::{Read, Write};
+use std::path::PathBuf;
+use std::str::FromStr;
+
+use crate::common::PIPESIZE;
+use crate::compression_engine::CompressionType;
+use crate::config;
+use crate::db;
+use crate::import_tar;
+use crate::modes::common::ImportMeta;
+
+/// Import items from a `.keep.tar` archive or legacy `.meta.yml` file.
+///
+/// For `.keep.tar` files, all items are imported in their original ID order,
+/// each receiving a new auto-incremented ID from the database.
+/// For `.meta.yml` files, the legacy single-item import is used.
+pub fn mode_import(
+    cmd: &mut Command,
+    settings: &config::Settings,
+    import_path: &str,
+    conn: &mut rusqlite::Connection,
+    data_path: PathBuf,
+) -> Result<()> {
+    let path = PathBuf::from(import_path);
+
+    if import_path.ends_with(".keep.tar") {
+        // New tar-based import
+        let imported_ids = import_tar::import_from_tar(&path, conn, &data_path)?;
+
+        if !settings.quiet {
+            println!(
+                "KEEP: Imported {} item(s): {:?}",
+                imported_ids.len(),
+                imported_ids
+            );
+        }
+
+        debug!(
+            "IMPORT: Imported {} items from {}",
+            imported_ids.len(),
+            import_path
+        );
+    } else if import_path.ends_with(".meta.yml") {
+        // Legacy single-item import
+        import_legacy(cmd, settings, import_path, conn, data_path)?;
+    } else {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            format!("Unsupported import format: {}", import_path),
+        )
+        .exit();
+    }
+
+    Ok(())
+}
+
+/// Legacy single-item import from a `.meta.yml` file.
+fn import_legacy(
+    cmd: &mut Command,
+    settings: &config::Settings,
+    meta_file: &str,
+    conn: &mut rusqlite::Connection,
+    data_path: PathBuf,
+) -> Result<()> {
+    // Read metadata
+    let meta_yaml = fs::read_to_string(meta_file)
+        .with_context(|| format!("Cannot read metadata file: {meta_file}"))?;
+    let import_meta: ImportMeta = serde_yaml::from_str(&meta_yaml)
+        .with_context(|| format!("Cannot parse metadata file: {meta_file}"))?;
+
+    // Validate compression type
+    CompressionType::from_str(&import_meta.compression).map_err(|_| {
+        anyhow!(
+            "Invalid compression type '{}' in metadata file",
+            import_meta.compression
+        )
+    })?;
+
+    debug!(
+        "IMPORT: Parsed meta: ts={}, compression={}, tags={:?}",
+        import_meta.ts, import_meta.compression, import_meta.tags
+    );
+
+    // Create item with original timestamp
+    let item = db::insert_item_with_ts(conn, import_meta.ts, &import_meta.compression)?;
+    let item_id = item.id.context("New item missing ID")?;
+
+    debug!(
+        "IMPORT: Created item {} with compression {}",
+        item_id, import_meta.compression
+    );
+
+    // Set tags
+    if !import_meta.tags.is_empty() {
+        db::set_item_tags(conn, item.clone(), &import_meta.tags)?;
+        debug!("IMPORT: Set {} tags", import_meta.tags.len());
+    }
+
+    // Write data to storage using streaming copy
+    let mut item_path = data_path;
+    item_path.push(item_id.to_string());
+
+    let data_size: i64 = if let Some(ref data_file) = settings.import_data_file {
+        // Stream from file to storage using fixed-size buffers
+        let mut reader = fs::File::open(data_file)
+            .with_context(|| format!("Cannot read data file: {}", data_file.display()))?;
+        let mut writer = fs::File::create(&item_path)
+            .with_context(|| format!("Cannot create item file: {}", item_path.display()))?;
+        let mut buf = [0u8; PIPESIZE];
+        let mut total = 0i64;
+        loop {
+            let n = reader.read(&mut buf)?;
+            if n == 0 {
+                break;
+            }
+            writer.write_all(&buf[..n])?;
+            total += n as i64;
+        }
+        total
+    } else {
+        // Stream from stdin to storage
+        let mut writer = fs::File::create(&item_path)
+            .with_context(|| format!("Cannot create item file: {}", item_path.display()))?;
+        let mut stdin = std::io::stdin().lock();
+        let mut buf = [0u8; PIPESIZE];
+        let mut total = 0i64;
+        loop {
+            let n = stdin.read(&mut buf)?;
+            if n == 0 {
+                break;
+            }
+            writer.write_all(&buf[..n])?;
+            total += n as i64;
+        }
+        total
+    };
+
+    if data_size == 0 {
+        cmd.error(
+            clap::error::ErrorKind::InvalidValue,
+            "No data provided (empty file or stdin)",
+        )
+        .exit();
+    }
+
+    debug!(
+        "IMPORT: Wrote {} bytes to {}",
+        data_size,
+        item_path.display()
+    );
+
+    // Set metadata
+    for (key, value) in &import_meta.metadata {
+        db::query_upsert_meta(
+            conn,
+            db::Meta {
+                id: item_id,
+                name: key.clone(),
+                value: value.clone(),
+            },
+        )?;
+    }
+    if !import_meta.metadata.is_empty() {
+        debug!(
+            "IMPORT: Set {} metadata entries",
+            import_meta.metadata.len()
+        );
+    }
+
+    // Update item sizes (use imported size if available, otherwise data length)
+    let size_to_record = import_meta.uncompressed_size.unwrap_or(data_size);
+    let mut updated_item = item;
+    updated_item.uncompressed_size = Some(size_to_record);
+    updated_item.compressed_size = Some(std::fs::metadata(&item_path)?.len() as i64);
+    updated_item.closed = true;
+    db::update_item(conn, updated_item)?;
+
+    if !settings.quiet {
+        println!(
+            "KEEP: Imported item {} tags: {:?}",
+            item_id, import_meta.tags
+        );
+    }
+
+    Ok(())
+}

src/modes/info.rs

@@ -1,58 +1,95 @@
-use crate::db::Item;
-use crate::modes::common::{format_size, get_output_format, OutputFormat};
-use anyhow::anyhow;
-use serde_json;
-use serde_yaml;
-use serde::{Deserialize, Serialize};
+use crate::config;
+use crate::modes::common::{DisplayItemInfo, OutputFormat, format_size, render_item_info_table};
+use crate::services::types::ItemWithMeta;
+use anyhow::{Context, Result, anyhow};
 use clap::Command;
 use clap::error::ErrorKind;
+use serde::{Deserialize, Serialize};
 use std::path::PathBuf;
-use std::str::FromStr;
 
-use crate::compression_engine::CompressionType;
-use crate::db::{get_item, get_item_last, get_item_matching};
-use crate::modes::common::get_format_box_chars_no_border_line_separator;
+use crate::services::item_service::ItemService;
 use chrono::prelude::*;
-use is_terminal::IsTerminal;
-use prettytable::format;
-use prettytable::{Attr, Cell, Row, Table};
 
+/// Displays detailed information about an item or the last item if no ID/tags specified.
+///
+/// Supports table, JSON, or YAML output formats. Validates input (at most one ID, no mixing IDs/tags).
+/// Uses ItemService to fetch the item and displays via helpers.
+///
+/// # Arguments
+///
+/// * `cmd` - Mutable Clap command for error handling and exiting on invalid args.
+/// * `settings` - Application settings for output formatting and human-readable sizes.
+/// * `ids` - Mutable vector of item IDs (at most one; cleared if tags used).
+/// * `tags` - Mutable vector of tags (mutually exclusive with IDs).
+/// * `conn` - Mutable database connection for querying items.
+/// * `data_path` - Path to data directory for file metadata.
+///
+/// # Returns
+///
+/// `Ok(())` on success, or `Err(anyhow::Error)` if item not found or DB query fails.
+///
+/// # Errors
+///
+/// * Clap errors if invalid args (e.g., multiple IDs).
+/// * Anyhow error if no matching item found.
+///
+/// # Examples
+///
+/// ```ignore
+/// // Example usage requires Command, Settings, Connection, and PathBuf instances
+/// mode_info(&mut cmd, &settings, &mut vec![123], &mut vec![], &mut conn, data_path)?;
+/// ```
 pub fn mode_info(
     cmd: &mut Command,
-    args: &crate::Args,
-    ids: &mut Vec<i64>,
-    tags: &mut Vec<String>,
+    settings: &config::Settings,
+    ids: &mut [i64],
+    tags: &mut [String],
     conn: &mut rusqlite::Connection,
     data_path: PathBuf,
-) -> anyhow::Result<()> {
+) -> Result<()> {
+    // For --info, we can use either IDs or tags, but not both
     if !ids.is_empty() && !tags.is_empty() {
-        cmd.error(ErrorKind::InvalidValue, "Both ID and tags given, you must supply exactly one ID or atleast one tag when using --info").exit();
+        cmd.error(
+            ErrorKind::InvalidValue,
+            "Both ID and tags given, you must supply either IDs or tags when using --info",
+        )
+        .exit();
     } else if ids.len() > 1 {
-        cmd.error(ErrorKind::InvalidValue, "More than one ID given, you must supply exactly one ID or atleast one tag when using --info").exit();
+        cmd.error(
+            ErrorKind::InvalidValue,
+            "More than one ID given, you must supply exactly one ID when using --info",
+        )
+        .exit();
     }
+    // If both are empty, find_item will find the last item
 
-    let mut meta: std::collections::HashMap<String, String> = std::collections::HashMap::new();
-    for item in args.item.meta.iter() {
-        let item = item.clone();
-        meta.insert(item.key, item.value);
-    }
+    let item_service = ItemService::new(data_path.clone());
+    let item_with_meta = item_service
+        .find_item(conn, ids, tags, &settings.meta_filter())
+        .map_err(|e| anyhow!("Unable to find matching item in database: {}", e))?;
 
-    let item_maybe = match tags.is_empty() && meta.is_empty() {
-        true => match ids.iter().next() {
-            Some(item_id) => get_item(conn, *item_id)?,
-            None => get_item_last(conn)?,
-        },
-        false => get_item_matching(conn, tags, &meta)?,
-    };
-
-    match item_maybe {
-        Some(item) => show_item(item, args, conn, data_path),
-        None => Err(anyhow!("Unable to find matching item in database")),
-    }
+    show_item(item_with_meta, settings, data_path)
 }
 
-#[derive(Serialize, Deserialize)]
-struct ItemInfo {
+#[derive(Debug, Serialize, Deserialize)]
+/// Structured representation of item information for JSON/YAML output.
+///
+/// This struct serializes item details including ID, timestamp, sizes, compression, tags, and metadata
+/// for non-table output formats.
+///
+/// # Fields
+///
+/// * `id` - The unique item ID.
+/// * `timestamp` - Formatted timestamp string.
+/// * `path` - Full file path to the item.
+/// * `stream_size` - Original uncompressed size in bytes (optional).
+/// * `stream_size_formatted` - Human-readable stream size.
+/// * `compression` - Compression type used.
+/// * `file_size` - Compressed file size in bytes (optional).
+/// * `file_size_formatted` - Human-readable file size.
+/// * `tags` - List of associated tags.
+/// * `meta` - Metadata key-value pairs.
+pub struct ItemInfo {
     id: i64,
     timestamp: String,
     path: String,
@@ -65,138 +102,143 @@ struct ItemInfo {
     meta: std::collections::HashMap<String, String>,
 }
 
+/// Displays item information in table format or delegates to structured output.
+///
+/// Builds a comfy-table for tabular display or calls structured helper for JSON/YAML.
+/// Handles file size via metadata and formats tags/meta accordingly.
+///
+/// # Arguments
+///
+/// * `item_with_meta` - Item with associated metadata and tags.
+/// * `settings` - Application settings for formatting (e.g., human-readable sizes).
+/// * `data_path` - Path to data directory for calculating compressed file size.
+///
+/// # Returns
+///
+/// `Ok(())` on success, or `Err(anyhow::Error)` if path resolution fails.
+///
+/// # Errors
+///
+/// * Anyhow error if item path cannot be stringified.
+///
+/// # Examples
+///
+/// ```ignore
+/// // Example usage requires ItemWithMeta, Settings, and PathBuf instances
+/// show_item(item_with_meta, &settings, data_path)?;
+/// ```
 fn show_item(
-    item: Item, // Using the provided struct definition
-    args: &crate::Args,
-    conn: &mut rusqlite::Connection,
+    item_with_meta: ItemWithMeta,
+    settings: &config::Settings,
     data_path: PathBuf,
-) -> anyhow::Result<()> {
-    let item_id = item.id.unwrap(); // Consider using if let or expect for Option
-
-    let item_tags: Vec<String> = crate::db::get_item_tags(conn, &item)?
-        .into_iter()
-        .map(|x| x.name)
-        .collect();
-
-    let output_format = get_output_format(args);
+) -> Result<()> {
+    let output_format = crate::modes::common::settings_output_format(settings);
 
     if output_format != OutputFormat::Table {
-        return show_item_structured(item, args, conn, data_path, output_format);
+        return show_item_structured(item_with_meta, settings, data_path, output_format);
     }
 
-    let mut table = Table::new();
-    if std::io::stdout().is_terminal() {
-        table.set_format(get_format_box_chars_no_border_line_separator());
-    } else {
-        table.set_format(*format::consts::FORMAT_NO_BORDER_LINE_SEPARATOR);
-    }
+    let item_tags = item_with_meta.tag_names();
+    let item = item_with_meta.item;
+    let item_id = item.id.context("Item missing ID")?;
 
-    table.add_row(Row::new(vec![
-        Cell::new("ID").with_style(Attr::Bold),
-        Cell::new(&item_id.to_string()),
-    ]));
+    let mut item_path_buf = data_path.clone();
+    item_path_buf.push(item_id.to_string());
 
-    let ts_cell = Cell::new(&item.ts.with_timezone(&Local).format("%F %T %Z").to_string());
-    table.add_row(Row::new(vec![
-        Cell::new("Timestamp").with_style(Attr::Bold),
-        ts_cell,
-    ]));
-
-    let mut item_path_buf = data_path.clone(); // Renamed to avoid conflict if item_path is used later
-    item_path_buf.push(item.id.unwrap().to_string()); // Again, consider safer unwrap
-
-    table.add_row(Row::new(vec![
-        Cell::new("Path").with_style(Attr::Bold),
-        Cell::new(item_path_buf.to_str().expect("Unable to get item path")),
-    ]));
-
-    let size_cell = match item.size {
-        Some(size) => Cell::new(format_size(size as u64, args.options.human_readable).as_str()),
-        None => Cell::new("Missing")
-            .with_style(Attr::ForegroundColor(prettytable::color::RED))
-            .with_style(Attr::Bold),
+    let size_str = match item.uncompressed_size {
+        Some(size) => format_size(size as u64, settings.human_readable),
+        None => "Missing".to_string(),
     };
-    table.add_row(Row::new(vec![
-        Cell::new("Stream Size").with_style(Attr::Bold),
-        size_cell,
-    ]));
 
-    // compression_type is CompressionType due to '?'
-    let compression_type_val = CompressionType::from_str(&item.compression)
-        .map_err(|e| anyhow!("Failed to parse compression type: {}", e))?;
-
-    table.add_row(Row::new(vec![
-        Cell::new("Compression").with_style(Attr::Bold),
-        Cell::new(&compression_type_val.to_string()),
-    ]));
-
-    let file_size_cell = match item_path_buf.metadata() {
-        Ok(metadata) => {
-            Cell::new(format_size(metadata.len(), args.options.human_readable).as_str())
-        }
-        Err(_) => Cell::new("Missing")
-            .with_style(Attr::ForegroundColor(prettytable::color::RED))
-            .with_style(Attr::Bold),
+    let file_size_str = match item_path_buf.metadata() {
+        Ok(metadata) => format_size(metadata.len(), settings.human_readable),
+        Err(_) => "Missing".to_string(),
     };
-    table.add_row(Row::new(vec![
-        Cell::new("File Size").with_style(Attr::Bold),
-        file_size_cell,
-    ]));
 
-    table.add_row(Row::new(vec![
-        Cell::new("Tags").with_style(Attr::Bold),
-        Cell::new(&item_tags.join(" ")),
-    ]));
+    let metadata: Vec<(String, String)> = item_with_meta
+        .meta
+        .iter()
+        .map(|m| (m.name.clone(), m.value.clone()))
+        .collect();
 
-    for meta in crate::db::get_item_meta(conn, &item)? {
-        let meta_name = format!("Meta: {}", &meta.name);
-        table.add_row(Row::new(vec![
-            Cell::new(meta_name.as_str()).with_style(Attr::Bold),
-            Cell::new(&meta.value),
-        ]));
-    }
+    let display = DisplayItemInfo {
+        id: item_id,
+        timestamp: item.ts.with_timezone(&Local).format("%F %T %Z").to_string(),
+        path: item_path_buf
+            .to_str()
+            .ok_or_else(|| anyhow::anyhow!("non-UTF-8 item path"))?
+            .to_string(),
+        stream_size: size_str,
+        compression: item.compression.clone(),
+        file_size: file_size_str,
+        tags: item_tags,
+        metadata,
+    };
 
-    table.printstd();
+    render_item_info_table(&display, &settings.table_config);
     Ok(())
 }
 
+/// Displays item information in structured JSON or YAML format.
+///
+/// Serializes ItemInfo and prints pretty-formatted output. Handles file metadata for sizes.
+///
+/// # Arguments
+///
+/// * `item_with_meta` - Item with metadata and tags.
+/// * `settings` - Settings for size formatting (human-readable).
+/// * `data_path` - Data path for compressed file size calculation.
+/// * `output_format` - JSON or YAML (Table is unreachable here).
+///
+/// # Returns
+///
+/// `Ok(())` on success, or `Err(anyhow::Error)` if serialization or path fails.
+///
+/// # Errors
+///
+/// * Serde errors during JSON/YAML serialization.
+/// * Anyhow error if file metadata unavailable.
+///
+/// # Examples
+///
+/// ```ignore
+/// // Example usage requires ItemWithMeta, Settings, PathBuf, and OutputFormat instances
+/// show_item_structured(item_with_meta, &settings, data_path, OutputFormat::Json)?;
+/// ```
 fn show_item_structured(
-    item: Item,
-    args: &crate::Args,
-    conn: &mut rusqlite::Connection,
+    item_with_meta: ItemWithMeta,
+    settings: &config::Settings,
     data_path: PathBuf,
     output_format: OutputFormat,
-) -> anyhow::Result<()> {
-    let item_id = item.id.unwrap();
-    let item_tags: Vec<String> = crate::db::get_item_tags(conn, &item)?
-        .into_iter()
-        .map(|x| x.name)
-        .collect();
+) -> Result<()> {
+    let item_tags = item_with_meta.tag_names();
+    let meta_map = item_with_meta.meta_as_map();
+    let item = item_with_meta.item;
+    let item_id = item.id.context("Item missing ID")?;
 
     let mut item_path_buf = data_path.clone();
     item_path_buf.push(item_id.to_string());
 
     let file_size = item_path_buf.metadata().map(|m| m.len()).ok();
     let file_size_formatted = match file_size {
-        Some(size) => format_size(size, args.options.human_readable),
+        Some(size) => format_size(size, settings.human_readable),
         None => "Missing".to_string(),
     };
 
-    let stream_size_formatted = match item.size {
-        Some(size) => format_size(size as u64, args.options.human_readable),
+    let stream_size_formatted = match item.uncompressed_size {
+        Some(size) => format_size(size as u64, settings.human_readable),
         None => "Missing".to_string(),
     };
 
-    let mut meta_map = std::collections::HashMap::new();
-    for meta in crate::db::get_item_meta(conn, &item)? {
-        meta_map.insert(meta.name, meta.value);
-    }
-
     let item_info = ItemInfo {
         id: item_id,
-        timestamp: item.ts.with_timezone(&chrono::Local).format("%F %T %Z").to_string(),
+        timestamp: item
+            .ts
+            .with_timezone(&chrono::Local)
+            .format("%F %T %Z")
+            .to_string(),
         path: item_path_buf.to_str().unwrap_or("").to_string(),
-        stream_size: item.size.map(|s| s as u64),
+        stream_size: item.uncompressed_size.map(|s| s as u64),
         stream_size_formatted,
         compression: item.compression,
         file_size,
@@ -205,15 +247,7 @@ fn show_item_structured(
         meta: meta_map,
     };
 
-    match output_format {
-        OutputFormat::Json => {
-            println!("{}", serde_json::to_string_pretty(&item_info)?);
-        }
-        OutputFormat::Yaml => {
-            println!("{}", serde_yaml::to_string(&item_info)?);
-        }
-        OutputFormat::Table => unreachable!(),
-    }
+    crate::modes::common::print_serialized(&item_info, &output_format)?;
 
     Ok(())
 }

src/modes/list.rs

@@ -1,228 +1,301 @@
-use crate::db::{get_items, get_items_matching};
+/// List mode implementation.
+///
+/// This module provides the functionality to list stored items with customizable
+/// formatting, filtering by tags, and support for different output formats
+/// including table, JSON, and YAML.
+use crate::config;
 use crate::modes::common::ColumnType;
-use crate::modes::common::{size_column, string_column, get_output_format, OutputFormat};
+use crate::modes::common::{OutputFormat, apply_color, apply_table_attribute, format_size};
+use crate::services::item_service::ItemService;
+use crate::services::types::ItemWithMeta;
+use anyhow::{Context, Result};
+use comfy_table::CellAlignment;
+use comfy_table::{Attribute, Cell, Color, Row};
 use serde::{Deserialize, Serialize};
 use serde_json;
 use serde_yaml;
-use anyhow::anyhow;
-use log::debug;
-use prettytable::color;
-use prettytable::row;
-use prettytable::format::Alignment;
-use prettytable::{Attr, Cell, Row, Table};
 
+/// Structure representing a list item for structured output formats.
+///
+/// This struct holds all the information needed to serialize an item for JSON or
+/// YAML output in list mode.
 #[derive(Serialize, Deserialize)]
 struct ListItem {
+    /// Item ID.
+    ///
+    /// The unique identifier for the item.
     id: Option<i64>,
+    /// Timestamp.
+    ///
+    /// The formatted timestamp string for the item.
     time: String,
+    /// Size in bytes.
+    ///
+    /// The raw size of the item content.
     size: Option<u64>,
+    /// Formatted size.
+    ///
+    /// Human-readable size string.
     size_formatted: String,
+    /// Compression type.
+    ///
+    /// The compression algorithm used for the item.
     compression: String,
+    /// File size in bytes.
+    ///
+    /// The size of the stored file on disk.
     file_size: Option<u64>,
+    /// Formatted file size.
+    ///
+    /// Human-readable file size string.
     file_size_formatted: String,
+    /// File path.
+    ///
+    /// The full path to the item's storage file.
     file_path: String,
+    /// Tags.
+    ///
+    /// Vector of tag names associated with the item.
     tags: Vec<String>,
+    /// Metadata.
+    ///
+    /// HashMap of metadata key-value pairs.
     meta: std::collections::HashMap<String, String>,
 }
 
+/// Main list mode function.
+///
+/// This function handles the listing of items based on tags, applying formatting
+/// and output options from settings. It supports table, JSON, and YAML output formats.
+///
+/// # Arguments
+///
+/// * `cmd` - Mutable reference to the Clap command for error handling.
+/// * `settings` - Reference to application settings.
+/// * `ids` - Mutable vector of item IDs (should be empty for list mode).
+/// * `tags` - Reference to vector of tags for filtering.
+/// * `conn` - Mutable reference to database connection.
+/// * `data_path` - Path to the data directory.
+///
+/// # Returns
+///
+/// * `Result<()>` - Success or error if listing fails.
 pub fn mode_list(
-    cmd: &mut clap::Command,
-    args: &crate::Args,
-    ids: &mut Vec<i64>,
-    tags: &Vec<String>,
+    _cmd: &mut clap::Command,
+    settings: &config::Settings,
+    ids: &mut [i64],
+    tags: &[String],
     conn: &mut rusqlite::Connection,
     data_path: std::path::PathBuf,
-) -> anyhow::Result<()> {
-    if !ids.is_empty() {
-        cmd.error(
-            clap::error::ErrorKind::InvalidValue,
-            "ID given, you can only supply tags when using --list",
-        )
-        .exit();
+) -> Result<()> {
+    let item_service = ItemService::new(data_path.clone());
+    let items_with_meta = item_service.get_items(conn, ids, tags, &settings.meta_filter())?;
+
+    if settings.ids_only {
+        for item_with_meta in &items_with_meta {
+            if let Some(id) = item_with_meta.item.id {
+                println!("{id}");
+            }
+        }
+        return Ok(());
     }
 
-    let mut meta: std::collections::HashMap<String, String> = std::collections::HashMap::new();
-    for item in args.item.meta.iter() {
-        let item = item.clone();
-        meta.insert(item.key, item.value);
-    }
-
-    let items = match tags.is_empty() && meta.is_empty() {
-        true => get_items(conn)?,
-        false => get_items_matching(conn, tags, &meta)?,
-    };
-
-    debug!("MAIN: Items: {:?}", items);
-
-    // Collect all item IDs for batch queries
-    let item_ids: Vec<i64> = items.iter().map(|item| item.id.unwrap()).collect();
-    
-    // Fetch all tags for all items in a single query
-    let all_tags = crate::db::get_tags_for_items(conn, &item_ids)?;
-    let mut tags_by_item: std::collections::HashMap<i64, Vec<String>> =
-        std::collections::HashMap::new();
-    
-    // Convert Tag structs to just names
-    for (item_id, tags) in all_tags {
-        let tag_names: Vec<String> = tags.into_iter().map(|tag| tag.name).collect();
-        tags_by_item.insert(item_id, tag_names);
-    }
-    
-    // Fetch all metadata for all items in a single query
-    let meta_by_item = crate::db::get_meta_for_items(conn, &item_ids)?;
-
-    let output_format = get_output_format(args);
+    let output_format = crate::modes::common::settings_output_format(settings);
 
     if output_format != OutputFormat::Table {
-        return show_list_structured(items, tags_by_item, meta_by_item, data_path, args, output_format);
+        return show_list_structured(items_with_meta, data_path, settings, output_format);
     }
 
-    let mut table = Table::new();
-    table.set_format(*prettytable::format::consts::FORMAT_CLEAN);
+    let mut table = crate::modes::common::create_table_with_config(&settings.table_config);
 
-    let list_format = args.options.list_format.split(",");
-
-    let mut title_row = row!();
-
-    for column in list_format.clone() {
-        let mut column_format = column.split(":");
-        let column_name = column_format.next().expect("Unable to parse column name");
-        let column_type = ColumnType::from_str(column_name)
-            .map_err(|_| anyhow!("Unknown column {:?}", column_name))?;
-
-        if column_type == ColumnType::Meta {
-            let meta_name = column_format
-                .next()
-                .expect("Unable to parse metadata name for meta column");
-            title_row.add_cell(Cell::new(meta_name).with_style(Attr::Bold));
-        } else {
-            title_row.add_cell(Cell::new(&column_type.to_string()).with_style(Attr::Bold));
-        }
+    // Create header row
+    let mut header_cells = Vec::new();
+    for column in &settings.list_format {
+        header_cells.push(Cell::new(&column.label).add_attribute(Attribute::Bold));
     }
+    table.set_header(header_cells);
 
-    table.set_titles(title_row);
+    for item_with_meta in items_with_meta {
+        let tags = item_with_meta.tag_names();
+        let meta = item_with_meta.meta_as_map();
+        let item = item_with_meta.item;
 
-    for item in items {
-        let item_id = item.id.unwrap();
-        let tags = tags_by_item.get(&item_id).unwrap();
-        let meta = meta_by_item.get(&item_id).unwrap();
         let mut item_path = data_path.clone();
-        item_path.push(item.id.unwrap().to_string());
+        item_path.push(item.id.context("Item missing ID")?.to_string());
 
-        let mut table_row = Row::new(vec![]);
+        let mut table_row = Row::new();
+
+        for column in &settings.list_format {
+            let column_type = column
+                .name
+                .parse::<ColumnType>()
+                .with_context(|| format!("Unknown column type {:?} in list format", column.name))?;
 
-        for column in list_format.clone() {
-            let mut column_format = column.split(":");
-            let column_name = column_format.next().expect("Unable to parse column name");
-            let column_type = ColumnType::from_str(column_name)
-                .unwrap_or_else(|_| panic!("Unknown column {:?}", column_name));
             let mut meta_name: Option<&str> = None;
 
-            if column_type == ColumnType::Meta {
-                meta_name = column_format.next();
+            if let ColumnType::Meta = column_type {
+                let parts: Vec<&str> = column.name.split(':').collect();
+                if parts.len() > 1 {
+                    meta_name = Some(parts[1]);
+                }
             }
 
-            let column_width: usize = match column_format.next() {
-                Some(len) => len.parse().unwrap_or(0),
-                None => 0,
-            };
-
-            let cell = match column_type {
-                ColumnType::Id => Cell::new_align(
-                    &string_column(item.id.unwrap_or(0).to_string(), column_width),
-                    Alignment::RIGHT,
-                ),
-                ColumnType::Time => Cell::new(&string_column(
-                    item.ts
+            let cell_content = match column_type {
+                ColumnType::Id => item.id.unwrap_or(0).to_string(),
+                ColumnType::Time => item
+                    .ts
                     .with_timezone(&chrono::Local)
                     .format("%F %T")
                     .to_string(),
-                    column_width,
-                )),
-                ColumnType::Size => match item.size {
-                    Some(size) => Cell::new_align(
-                        &size_column(size as u64, args.options.human_readable, column_width),
-                        Alignment::RIGHT,
-                    ),
+                ColumnType::Size => match item.uncompressed_size {
+                    Some(size) => format_size(size as u64, settings.human_readable),
                     None => match item_path.metadata() {
-                        Ok(_) => Cell::new_align("Unknown", Alignment::RIGHT)
-                            .with_style(Attr::ForegroundColor(color::YELLOW))
-                            .with_style(Attr::Bold),
-                        Err(_) => Cell::new_align("Missing", Alignment::RIGHT)
-                            .with_style(Attr::ForegroundColor(color::RED))
-                            .with_style(Attr::Bold),
+                        Ok(_) => "Unknown".to_string(),
+                        Err(e) => {
+                            log::warn!("File missing or inaccessible: {}", e);
+                            "Missing".to_string()
+                        }
                     },
                 },
-                ColumnType::Compression => {
-                    Cell::new(&string_column(item.compression.to_string(), column_width))
-                },
+                ColumnType::Compression => item.compression.to_string(),
                 ColumnType::FileSize => match item_path.metadata() {
-                    Ok(metadata) => Cell::new_align(
-                        &size_column(metadata.len(), args.options.human_readable, column_width),
-                        Alignment::RIGHT,
-                    ),
-                    Err(_) => Cell::new_align("Missing", Alignment::RIGHT)
-                        .with_style(Attr::ForegroundColor(color::RED))
-                        .with_style(Attr::Bold),
+                    Ok(metadata) => format_size(metadata.len(), settings.human_readable),
+                    Err(e) => {
+                        log::warn!("File missing or inaccessible: {}", e);
+                        "Missing".to_string()
+                    }
                 },
-                ColumnType::FilePath => Cell::new(&string_column(
-                    item_path.clone().into_os_string().into_string().unwrap(),
-                    column_width,
-                )),
-                ColumnType::Tags => Cell::new(&string_column(tags.join(" "), column_width)),
+                ColumnType::FilePath => item_path
+                    .clone()
+                    .into_os_string()
+                    .into_string()
+                    .unwrap_or_else(|os| os.to_string_lossy().into_owned()),
+                ColumnType::Tags => tags.join(" "),
                 ColumnType::Meta => match meta_name {
                     Some(meta_name) => match meta.get(meta_name) {
-                        Some(meta_value) => {
-                            Cell::new(&string_column(meta_value.to_string(), column_width))
+                        Some(meta_value) => meta_value.to_string(),
+                        None => "".to_string(),
+                    },
+                    None => "".to_string(),
+                },
+            };
+
+            // Truncate content to max 3 lines
+            let mut cell_lines: Vec<String> =
+                cell_content.split('\n').map(|s| s.to_string()).collect();
+            if cell_lines.len() > 3 {
+                cell_lines.truncate(3);
+                // Add ellipsis to the last line if we truncated
+                if let Some(last_line) = cell_lines.last_mut() {
+                    if last_line.len() > 3 {
+                        last_line.truncate(last_line.len() - 3);
                     }
-                        None => Cell::new(""),
-                    },
-                    None => Cell::new(""),
-                },
+                    last_line.push_str("...");
+                }
+            }
+            let truncated_content = cell_lines.join("\n");
+
+            let mut cell = Cell::new(truncated_content);
+
+            // Apply column-specific styling
+            if let Some(fg_color) = &column.fg_color {
+                cell = apply_color(cell, fg_color, true);
+            }
+
+            if let Some(bg_color) = &column.bg_color {
+                cell = apply_color(cell, bg_color, false);
+            }
+
+            for attribute in &column.attributes {
+                cell = apply_table_attribute(cell, attribute);
+            }
+
+            // Apply padding if specified
+            if let Some((_left_padding, _right_padding)) = column.padding {
+                // Note: comfy-table doesn't directly support padding, so we'd need to handle this
+                // by adding spaces to the content, or use a different approach
+            }
+
+            // Apply styling for specific cases
+            match column_type {
+                ColumnType::Size => {
+                    if item.uncompressed_size.is_none() {
+                        if item_path.metadata().is_ok() {
+                            cell = cell
+                                .fg(comfy_table::Color::Yellow)
+                                .add_attribute(Attribute::Bold);
+                        } else {
+                            cell = cell
+                                .fg(comfy_table::Color::Red)
+                                .add_attribute(Attribute::Bold);
+                        }
+                    }
+                }
+                ColumnType::FileSize => {
+                    if item_path.metadata().is_err() {
+                        cell = cell
+                            .fg(comfy_table::Color::Red)
+                            .add_attribute(Attribute::Bold);
+                    }
+                }
+                _ => {}
+            }
+
+            // Apply alignment
+            cell = match column.align {
+                crate::config::ColumnAlignment::Right => cell.set_alignment(CellAlignment::Right),
+                crate::config::ColumnAlignment::Left => cell.set_alignment(CellAlignment::Left),
+                crate::config::ColumnAlignment::Center => cell.set_alignment(CellAlignment::Center),
             };
             table_row.add_cell(cell);
         }
         table.add_row(table_row);
     }
 
-    table.printstd();
+    println!(
+        "{}",
+        crate::modes::common::trim_lines_end(&table.trim_fmt())
+    );
 
     Ok(())
 }
 
 fn show_list_structured(
-    items: Vec<crate::db::Item>,
-    tags_by_item: std::collections::HashMap<i64, Vec<String>>,
-    meta_by_item: std::collections::HashMap<i64, std::collections::HashMap<String, String>>,
+    items_with_meta: Vec<ItemWithMeta>,
     data_path: std::path::PathBuf,
-    args: &crate::Args,
+    settings: &config::Settings,
     output_format: OutputFormat,
-) -> anyhow::Result<()> {
+) -> Result<()> {
     let mut list_items = Vec::new();
 
-    for item in items {
-        let item_id = item.id.unwrap();
-        let tags = tags_by_item.get(&item_id).cloned().unwrap_or_default();
-        let meta = meta_by_item.get(&item_id).cloned().unwrap_or_default();
+    for item_with_meta in items_with_meta {
+        let tags = item_with_meta.tag_names();
+        let meta = item_with_meta.meta_as_map();
+        let item = item_with_meta.item;
+        let item_id = item.id.context("Item missing ID")?;
 
         let mut item_path = data_path.clone();
         item_path.push(item_id.to_string());
 
         let file_size = item_path.metadata().map(|m| m.len()).ok();
         let file_size_formatted = match file_size {
-            Some(size) => crate::modes::common::format_size(size, args.options.human_readable),
+            Some(size) => crate::modes::common::format_size(size, settings.human_readable),
             None => "Missing".to_string(),
         };
 
-        let size_formatted = match item.size {
-            Some(size) => crate::modes::common::format_size(size as u64, args.options.human_readable),
+        let size_formatted = match item.uncompressed_size {
+            Some(size) => crate::modes::common::format_size(size as u64, settings.human_readable),
             None => "Unknown".to_string(),
         };
 
         let list_item = ListItem {
             id: item.id,
-            time: item.ts.with_timezone(&chrono::Local).format("%F %T").to_string(),
-            size: item.size.map(|s| s as u64),
+            time: item
+                .ts
+                .with_timezone(&chrono::Local)
+                .format("%F %T")
+                .to_string(),
+            size: item.uncompressed_size.map(|s| s as u64),
             size_formatted,
             compression: item.compression,
             file_size,
@@ -235,15 +308,7 @@ fn show_list_structured(
         list_items.push(list_item);
     }
 
-    match output_format {
-        OutputFormat::Json => {
-            println!("{}", serde_json::to_string_pretty(&list_items)?);
-        }
-        OutputFormat::Yaml => {
-            println!("{}", serde_yaml::to_string(&list_items)?);
-        }
-        OutputFormat::Table => unreachable!(),
-    }
+    crate::modes::common::print_serialized(&list_items, &output_format)?;
 
     Ok(())
 }

src/modes/mod.rs

@@ -1,10 +1,64 @@
+#[cfg(feature = "server")]
+pub mod server;
+
+#[cfg(feature = "client")]
+pub mod client;
+
+/// Common utilities for all modes, including column types and output formatting.
 pub mod common;
+
 pub mod delete;
 pub mod diff;
+pub mod export;
+pub mod generate_config;
 pub mod get;
+pub mod import;
 pub mod info;
 pub mod list;
 pub mod save;
-pub mod server;
 pub mod status;
+pub mod status_plugins;
 pub mod update;
+
+/// Column types, output formats, and formatting utilities shared across modes.
+pub use common::{ColumnType, OutputFormat, format_size, settings_output_format};
+
+/// Deletes items from the database by ID.
+pub use delete::mode_delete;
+
+/// Compares two items and shows differences.
+pub use diff::mode_diff;
+
+/// Exports an item to data and metadata files.
+pub use export::mode_export;
+
+/// Generates a default configuration file.
+pub use generate_config::mode_generate_config;
+
+/// Retrieves and outputs item content.
+pub use get::mode_get;
+
+/// Imports an item from metadata and data files.
+pub use import::mode_import;
+
+/// Displays detailed information about items.
+pub use info::mode_info;
+
+/// Lists items with optional filtering.
+pub use list::mode_list;
+
+/// Saves new item content with optional tags and metadata.
+pub use save::mode_save;
+
+#[cfg(feature = "server")]
+/// Starts the HTTP server for REST API access.
+pub use server::mode_server;
+
+/// Shows status of directories and compression support.
+pub use status::mode_status;
+
+/// Lists available plugins and their configurations.
+pub use status_plugins::mode_status_plugins;
+
+/// Updates an item's tags and metadata by ID.
+pub use update::mode_update;

src/modes/save.rs

@@ -1,12 +1,24 @@
-use anyhow::{anyhow, Result};
+use anyhow::Result;
 use clap::Command;
-use log::debug;
-use std::io::{Read, Write, IsTerminal};
+use std::io::{Read, Write};
 
-// Import the missing functions from common module
-use crate::modes::common::{cmd_args_digest_type, cmd_args_compression_type, cmd_args_meta_plugin_types};
+use crate::config;
+use crate::services::item_service::ItemService;
 
-fn validate_save_args(cmd: &mut Command, ids: &Vec<i64>) {
+/// Validates save mode arguments and exits with error if invalid.
+///
+/// This function checks that no item IDs are provided for save mode,
+/// as save operations create new items rather than modifying existing ones.
+///
+/// # Arguments
+///
+/// * `cmd` - Mutable reference to the Clap command for error reporting.
+/// * `ids` - Reference to the vector of item IDs (should be empty for save mode).
+///
+/// # Panics
+///
+/// Exits the program via Clap error if IDs are provided.
+fn validate_save_args(cmd: &mut Command, ids: &[i64]) {
     if !ids.is_empty() {
         cmd.error(
             clap::error::ErrorKind::InvalidValue,
@@ -16,260 +28,107 @@ fn validate_save_args(cmd: &mut Command, ids: &Vec<i64>) {
     }
 }
 
-fn initialize_tags(tags: &mut Vec<String>) {
-    if tags.is_empty() {
-        tags.push("none".to_string());
+/// A tee reader that duplicates input to both a reader and a writer as it reads.
+///
+/// This struct implements the `Read` trait and forwards all read operations to
+/// an underlying reader while simultaneously writing the same data to a writer.
+/// It's useful for saving content to a file while also echoing it to stdout.
+///
+/// # Fields
+///
+/// * `reader` - The underlying reader providing the data source.
+/// * `writer` - The writer receiving copies of all read data.
+struct TeeReader<R: Read, W: Write> {
+    reader: R,
+    writer: W,
+}
+
+impl<R: Read, W: Write> Read for TeeReader<R, W> {
+    /// Reads data from the underlying reader and duplicates it to the writer.
+    ///
+    /// This implementation reads from the inner reader and then writes the same
+    /// bytes to the writer. If the read returns 0 bytes (EOF), it returns 0.
+    ///
+    /// # Arguments
+    ///
+    /// * `buf` - Buffer to fill with data from the reader.
+    ///
+    /// # Returns
+    ///
+    /// * `io::Result<usize>` - Number of bytes read, or an I/O error.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error if the underlying read or write operations fail.
+    ///
+    /// # Examples
+    ///
+    /// ```ignore
+    /// let mut tee = TeeReader {
+    ///     reader: std::io::Cursor::new(b"Hello, world!"),
+    ///     writer: std::io::sink(),
+    /// };
+    /// let mut buf = [0; 5];
+    /// let n = tee.read(&mut buf).unwrap();
+    /// assert_eq!(n, 5);
+    /// assert_eq!(&buf[..n], b"Hello");
+    /// ```
+    fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
+        let n = self.reader.read(buf)?;
+        if n > 0 {
+            self.writer.write_all(&buf[..n])?;
+        }
+        Ok(n)
     }
 }
 
-fn setup_compression_and_plugins(
-    cmd: &mut Command,
-    args: &crate::Args,
-) -> (crate::compression_engine::CompressionType, Box<dyn crate::compression_engine::CompressionEngine>, Vec<Box<dyn crate::meta_plugin::MetaPlugin>>) {
-    let digest_type = cmd_args_digest_type(cmd, &args);
-    debug!("MAIN: Digest type: {:?}", digest_type);
-    
-    let compression_type = cmd_args_compression_type(cmd, &args);
-    debug!("MAIN: Compression type: {:?}", compression_type);
-    let compression_engine =
-        crate::compression_engine::get_compression_engine(compression_type.clone()).expect("Unable to get compression engine");
-
-    // Start with meta plugin types from command line
-    let mut meta_plugin_types: Vec<crate::meta_plugin::MetaPluginType> = cmd_args_meta_plugin_types(cmd, &args);
-    debug!("MAIN: Meta plugin types: {:?}", meta_plugin_types);
-    
-    // Convert digest type to meta plugin type and add to the list if needed
-    let digest_meta_plugin_type = match digest_type {
-        crate::meta_plugin::MetaPluginType::DigestSha256 => Some(crate::meta_plugin::MetaPluginType::DigestSha256),
-        crate::meta_plugin::MetaPluginType::DigestMd5 => Some(crate::meta_plugin::MetaPluginType::DigestMd5),
-        _ => None,
-    };
-    
-    // Add digest meta plugin to the list if needed
-    if let Some(digest_plugin_type) = digest_meta_plugin_type {
-        if !meta_plugin_types.contains(&digest_plugin_type) {
-            meta_plugin_types.push(digest_plugin_type);
-        }
-    }
-
-    // Initialize meta_plugins with MetaPlugin instances for each MetaPluginType
-    let mut meta_plugins: Vec<Box<dyn crate::meta_plugin::MetaPlugin>> = meta_plugin_types
-        .iter()
-        .map(|meta_plugin_type| crate::meta_plugin::get_meta_plugin(meta_plugin_type.clone()))
-        .collect();
-    
-    // Check for unsupported meta plugins, warn the user, and remove them from the list
-    let mut i = 0;
-    meta_plugins.retain(|meta_plugin| {
-        let is_supported = meta_plugin.is_supported();
-        if !is_supported {
-            // We need to get the meta name for the warning message
-            // Since we can't mutably borrow meta_plugin here, we create a temporary one
-            let meta_plugin_type = meta_plugin_types[i].clone();
-            let mut temp_plugin = crate::meta_plugin::get_meta_plugin(meta_plugin_type);
-            eprintln!("Warning: Meta plugin '{}' is enabled but not supported on this system", temp_plugin.meta_name());
-        }
-        i += 1;
-        is_supported
-    });
-
-    (compression_type, compression_engine, meta_plugins)
-}
-
-fn create_and_log_item(
-    conn: &mut rusqlite::Connection,
-    args: &crate::Args,
-    tags: &Vec<String>,
-    compression_type: &crate::compression_engine::CompressionType,
-) -> Result<crate::db::Item, anyhow::Error> {
-    let mut item = crate::db::Item {
-        id: None,
-        ts: chrono::Utc::now(),
-        size: None,
-        compression: compression_type.to_string(),
-    };
-
-    let id = crate::db::insert_item(conn, item.clone())?;
-    item.id = Some(id);
-    debug!("MAIN: Added item {:?}", item.clone());
-
-    if !args.options.quiet {
-        if std::io::stderr().is_terminal() {
-            let mut t = term::stderr().unwrap();
-            t.reset().unwrap_or(());
-            t.attr(term::Attr::Bold).unwrap_or(());
-            write!(t, "KEEP:").unwrap_or(());
-            t.reset().unwrap_or(());
-            write!(t, " New item ").unwrap_or(());
-            t.attr(term::Attr::Bold).unwrap_or(());
-            write!(t, "{id}")?;
-            t.reset().unwrap_or(());
-            write!(t, "  tags: ")?;
-            t.attr(term::Attr::Bold).unwrap_or(());
-            write!(t, "{}", tags.join(" "))?;
-            t.reset().unwrap_or(());
-            writeln!(t)?;
-            std::io::stderr().flush()?;
-        } else {
-            let mut t = std::io::stderr();
-            writeln!(t, "KEEP: New item: {}  tags: {:?}", id, tags)?;
-        }
-    }
-
-    Ok(item)
-}
-
-fn setup_item_metadata(
-    conn: &mut rusqlite::Connection,
-    _args: &crate::Args,
-    item: &crate::db::Item,
-    tags: &Vec<String>,
-) -> Result<(), anyhow::Error> {
-    crate::db::set_item_tags(conn, item.clone(), tags)?;
-    Ok(())
-}
-
-fn collect_item_meta(args: &crate::Args) -> std::collections::HashMap<String, String> {
-    let mut item_meta: std::collections::HashMap<String, String> = crate::modes::common::get_meta_from_env();
-
-    if let Ok(hostname) = gethostname::gethostname().into_string() {
-        if !item_meta.contains_key("hostname") {
-            item_meta.insert("hostname".to_string(), hostname);
-        }
-    }
-
-    for item in args.item.meta.iter() {
-        let item = item.clone();
-        item_meta.insert(item.key, item.value);
-    }
-
-    item_meta
-}
-
-fn process_input_stream(
-    compression_engine: &Box<dyn crate::compression_engine::CompressionEngine>,
-    data_path: &std::path::PathBuf,
-    item_id: i64,
-    meta_plugins: &mut Vec<Box<dyn crate::meta_plugin::MetaPlugin>>,
-) -> Result<(Box<dyn std::io::Write>, crate::db::Item), anyhow::Error> {
-    let mut item = crate::db::Item {
-        id: Some(item_id),
-        ts: chrono::Utc::now(),
-        size: None,
-        compression: String::new(), // Will be set later
-    };
-
-    let mut item_path = data_path.clone();
-    item_path.push(item_id.to_string());
-
-    let mut stdin = std::io::stdin().lock();
-    let mut stdout = std::io::stdout().lock();
-    let mut buffer = [0; libc::BUFSIZ as usize];
-
-    let mut item_out: Box<dyn std::io::Write> =
-        compression_engine
-            .create(item_path.clone())
-            .map_err(|e| anyhow!("Unable to write file {:?}: {}", item_path, e))?;
-
-    debug!("MAIN: Starting IO loop");
-    loop {
-        let n = stdin.read(&mut buffer[..libc::BUFSIZ as usize])?;
-        item.size = match item.size {
-            None => Some(n as i64),
-            Some(prev_n) => Some(prev_n + n as i64),
-        };
-
-        if n == 0 {
-            debug!("MAIN: EOF on STDIN");
-            break;
-        }
-
-        debug!("MAIN: Loop - {:?} bytes", item.size);
-
-        stdout.write_all(&buffer[..n])?;
-        item_out.write_all(&buffer[..n])?;
-
-        for meta_plugin in meta_plugins.iter_mut() {
-            meta_plugin.update(&buffer[..n]);
-        }
-    }
-    debug!("MAIN: Ending IO loop after {:?} bytes", item.size);
-
-    stdout.flush()?;
-    item_out.flush()?;
-
-    Ok((item_out, item))
-}
-
-fn finalize_meta_plugins(
-    conn: &rusqlite::Connection,
-    meta_plugins: &mut Vec<Box<dyn crate::meta_plugin::MetaPlugin>>,
-    item: &crate::db::Item,
-) -> Result<(), anyhow::Error> {
-    for meta_plugin in meta_plugins.iter_mut() {
-        let meta_name = meta_plugin.meta_name();
-
-        match meta_plugin.finalize() {
-            Ok(meta_value) => {
-                let meta = crate::db::Meta {
-                    id: item.id.ok_or_else(|| anyhow!("Item missing ID"))?,
-                    name: meta_name.clone(),
-                    value: meta_value,
-                };
-                if let Err(e) = crate::db::store_meta(conn, meta) {
-                    eprintln!("Warning: Failed to store meta value for {}: {}", meta_name, e);
-                }
-            }
-            Err(e) => {
-                eprintln!("Warning: Failed to finalize meta plugin {}: {}", meta_name, e);
-            }
-        }
-    }
-    Ok(())
-}
-
+/// Main save mode function.
+///
+/// This function handles the save operation by reading from stdin, duplicating
+/// the input to stdout (for real-time display), and saving the content to the
+/// item service. It validates arguments, creates the tee reader, and processes
+/// the save operation.
+///
+/// # Arguments
+///
+/// * `cmd` - Mutable reference to the Clap command for error handling.
+/// * `settings` - Application settings containing configuration.
+/// * `ids` - Mutable vector of item IDs (should be empty for save mode).
+/// * `tags` - Mutable vector of tags to associate with the new item.
+/// * `conn` - Mutable reference to the database connection.
+/// * `data_path` - Path to the data storage directory.
+///
+/// # Returns
+///
+/// * `Result<(), anyhow::Error>` - Success or error if save fails.
+///
+/// # Examples
+///
+/// ```ignore
+/// // In CLI context, this would be called internally
+/// mode_save(&mut cmd, &settings, &mut vec![], &mut vec!["important".to_string()], &mut conn, data_path)?;
+/// ```
 pub fn mode_save(
     cmd: &mut Command,
-    args: &crate::Args,
-    ids: &mut Vec<i64>,
+    settings: &config::Settings,
+    ids: &mut [i64],
     tags: &mut Vec<String>,
     conn: &mut rusqlite::Connection,
     data_path: std::path::PathBuf,
 ) -> Result<(), anyhow::Error> {
     validate_save_args(cmd, ids);
-    initialize_tags(tags);
 
-    let (compression_type, compression_engine, mut meta_plugins) = setup_compression_and_plugins(cmd, args);
+    let item_service = ItemService::new(data_path);
 
-    let mut item = create_and_log_item(conn, args, tags, &compression_type)?;
-    setup_item_metadata(conn, args, &item, tags)?; // Pass mutable reference
+    let stdin = std::io::stdin();
+    let stdout = std::io::stdout();
 
-    // Save as much as possible in case something breaks - don't use transactions
-    // This allows partial saves to succeed even if some metadata operations fail
-    let item_meta = collect_item_meta(args);
-    let item_id = item.id.ok_or_else(|| anyhow!("Item missing ID"))?;
-
-    for kv in item_meta.iter() {
-        let meta = crate::db::Meta {
-            id: item_id,
-            name: kv.0.to_string(),
-            value: kv.1.to_string(),
+    let tee_reader = TeeReader {
+        reader: stdin.lock(),
+        writer: stdout.lock(),
     };
-        crate::db::store_meta(conn, meta)?;
-    }
 
-    let (_item_out, processed_item) = process_input_stream(
-        &compression_engine,
-        &data_path,
-        item_id,
-        &mut meta_plugins,
-    )?;
-    
-    item.size = processed_item.size;
-    item.compression = compression_type.to_string();
-
-    finalize_meta_plugins(conn, &mut meta_plugins, &item)?;
-    crate::db::update_item(conn, item.clone())?;
+    item_service.save_item(tee_reader, cmd, settings, tags, conn)?;
 
     Ok(())
 }

src/modes/server.rs

@@ -1,101 +0,0 @@
-use anyhow::Result;
-use axum::{
-    routing::get,
-    Router,
-};
-use clap::Command;
-use log::{debug, info, warn};
-use std::net::SocketAddr;
-use std::path::PathBuf;
-use std::sync::Arc;
-use tokio::sync::Mutex;
-use tower_http::cors::CorsLayer;
-use tower::ServiceBuilder;
-use tower_http::trace::TraceLayer;
-
-use crate::Args;
-
-mod common;
-mod status;
-mod items;
-mod content;
-mod docs;
-
-pub use common::{ServerConfig, AppState, logging_middleware};
-use status::handle_status;
-use items::{handle_list_items, handle_get_item, handle_put_item, handle_delete_item};
-use content::{handle_get_content_latest, handle_get_content};
-use docs::{handle_openapi, handle_swagger_ui};
-
-pub fn mode_server(
-    _cmd: &mut Command,
-    args: &Args,
-    conn: &mut rusqlite::Connection,
-    data_path: PathBuf,
-) -> Result<()> {
-    let server_address = args.mode.server.as_ref().unwrap();
-    
-    let config = ServerConfig {
-        address: server_address.clone(),
-        password: args.options.server_password.clone(),
-    };
-    
-    // We need to move the connection into the async runtime
-    let rt = tokio::runtime::Runtime::new()?;
-    // Take ownership of the connection and move it into the async runtime
-    let owned_conn = std::mem::replace(conn, rusqlite::Connection::open_in_memory()?);
-    rt.block_on(run_server(config, owned_conn, data_path, args))
-}
-
-async fn run_server(
-    config: ServerConfig,
-    conn: rusqlite::Connection,
-    data_dir: PathBuf,
-    args: &Args,
-) -> Result<()> {
-    debug!("Starting REST HTTP server on {}", config.address);
-    
-    // Use the existing database connection
-    let db_conn = Arc::new(Mutex::new(conn));
-    
-    let state = AppState {
-        db: db_conn,
-        data_dir: data_dir.clone(),
-        password: config.password.clone(),
-        args: Arc::new(args.clone()),
-    };
-    
-    let app = Router::new()
-        .route("/status", get(handle_status))
-        .route("/item/", get(handle_list_items).put(handle_put_item))
-        .route("/item/:id", get(handle_get_item).delete(handle_delete_item))
-        .route("/content", get(handle_get_content_latest))
-        .route("/content/:id", get(handle_get_content))
-        .route("/openapi.json", get(handle_openapi))
-        .route("/swagger/", get(handle_swagger_ui))
-        .layer(axum::middleware::from_fn(logging_middleware))
-        .layer(
-            ServiceBuilder::new()
-                .layer(TraceLayer::new_for_http())
-                .layer(CorsLayer::permissive())
-        )
-        .with_state(state);
-    
-    let addr: SocketAddr = if config.address.starts_with('/') || config.address.starts_with("./") {
-        // Unix socket - not supported by axum directly, fall back to TCP
-        warn!("Unix sockets not yet implemented, falling back to TCP on 127.0.0.1:8080");
-        "127.0.0.1:8080".parse()?
-    } else {
-        config.address.parse()?
-    };
-    
-    info!("SERVER: HTTP server listening on {}", addr);
-    
-    let listener = tokio::net::TcpListener::bind(addr).await?;
-    axum::serve(
-        listener, 
-        app.into_make_service_with_connect_info::<SocketAddr>()
-    ).await?;
-    
-    Ok(())
-}

src/modes/server/api/common.rs

@@ -0,0 +1,37 @@
+use axum::{
+    http::{StatusCode, header},
+    response::Response,
+};
+use log;
+use serde::Serialize;
+
+pub struct ResponseBuilder;
+
+impl ResponseBuilder {
+    pub fn json<T: Serialize>(data: T) -> Result<Response, StatusCode> {
+        let json = serde_json::to_vec(&data).map_err(|e| {
+            log::warn!("Failed to serialize response: {e}");
+            StatusCode::INTERNAL_SERVER_ERROR
+        })?;
+
+        Response::builder()
+            .header(header::CONTENT_TYPE, "application/json")
+            .header(header::CONTENT_LENGTH, json.len().to_string())
+            .body(axum::body::Body::from(json))
+            .map_err(|e| {
+                log::warn!("Failed to build response: {e}");
+                StatusCode::INTERNAL_SERVER_ERROR
+            })
+    }
+
+    pub fn binary(content: &[u8], mime_type: &str) -> Result<Response, StatusCode> {
+        Response::builder()
+            .header(header::CONTENT_TYPE, mime_type)
+            .header(header::CONTENT_LENGTH, content.len().to_string())
+            .body(axum::body::Body::from(content.to_vec()))
+            .map_err(|e| {
+                log::warn!("Failed to build response: {e}");
+                StatusCode::INTERNAL_SERVER_ERROR
+            })
+    }
+}

src/modes/server/api/mod.rs

@@ -0,0 +1,102 @@
+pub mod common;
+pub mod item;
+pub mod status;
+
+use axum::{
+    Router,
+    routing::{delete, get, post},
+};
+
+use crate::modes::server::common::AppState;
+use utoipa::OpenApi;
+
+#[cfg(feature = "swagger")]
+use utoipa_swagger_ui::SwaggerUi;
+
+#[derive(OpenApi)]
+#[openapi(
+    info(
+        title = "Keep API",
+        version = "0.1.0",
+        description = "REST API for Keep - a tool to manage temporary files with automatic compression and metadata generation",
+        contact(
+            name = "Keep Project",
+        )
+    ),
+    paths(
+        status::handle_status,
+        item::handle_list_items,
+        item::handle_post_item,
+        item::handle_get_item_latest_meta,
+        item::handle_get_item_latest_content,
+        item::handle_get_item_meta,
+        item::handle_get_item_content,
+    ),
+    components(
+        schemas(
+            crate::modes::server::common::ItemInfo,
+            crate::modes::server::common::ItemContentInfo,
+            crate::modes::server::common::ItemInfoListResponse,
+            crate::modes::server::common::ItemInfoResponse,
+            crate::modes::server::common::ItemContentInfoResponse,
+            crate::modes::server::common::MetadataResponse,
+            crate::modes::server::common::StatusInfoResponse,
+            crate::common::status::StatusInfo,
+            crate::modes::server::common::ItemQuery,
+            crate::modes::server::common::ItemContentQuery,
+        )
+    ),
+    tags(
+        (name = "status", description = "System status and health check endpoints"),
+        (name = "item", description = "Item management endpoints for storing, retrieving, and managing content with metadata"),
+    ),
+    servers(
+        (url = "/", description = "Local server")
+    )
+)]
+#[allow(dead_code)]
+struct ApiDoc;
+
+pub fn add_routes(router: Router<AppState>) -> Router<AppState> {
+    router
+        // Status endpoints
+        .route("/api/status", get(status::handle_status))
+        .route("/api/plugins/status", get(status::handle_plugins_status))
+        // Item endpoints
+        .route(
+            "/api/item/",
+            get(item::handle_list_items).post(item::handle_post_item),
+        )
+        .route(
+            "/api/item/latest/meta",
+            get(item::handle_get_item_latest_meta),
+        )
+        .route(
+            "/api/item/latest/content",
+            get(item::handle_get_item_latest_content),
+        )
+        .route(
+            "/api/item/{item_id}/meta",
+            get(item::handle_get_item_meta).post(item::handle_post_item_meta),
+        )
+        .route(
+            "/api/item/{item_id}/content",
+            get(item::handle_get_item_content),
+        )
+        .route("/api/item/{item_id}", delete(item::handle_delete_item))
+        .route("/api/item/{item_id}/info", get(item::handle_get_item_info))
+        .route("/api/item/{item_id}/update", post(item::handle_update_item))
+        .route("/api/diff", get(item::handle_diff_items))
+        .route("/api/export", get(item::handle_export_items))
+        .route("/api/import", post(item::handle_import_items))
+}
+
+#[cfg(feature = "swagger")]
+pub fn add_docs_routes(router: Router<AppState>) -> Router<AppState> {
+    router.merge(SwaggerUi::new("/swagger").url("/openapi.json", ApiDoc::openapi()))
+}
+
+#[cfg(not(feature = "swagger"))]
+pub fn add_docs_routes(router: Router<AppState>) -> Router<AppState> {
+    router
+}

src/modes/server/api/status.rs

@@ -0,0 +1,129 @@
+use axum::{extract::State, http::StatusCode, response::Json};
+
+use crate::modes::server::common::{ApiResponse, AppState, StatusInfoResponse};
+
+async fn generate_status(
+    state: &AppState,
+) -> Result<crate::common::status::StatusInfo, StatusCode> {
+    let db_path = state
+        .db
+        .lock()
+        .await
+        .path()
+        .unwrap_or("unknown")
+        .to_string();
+
+    let status_service = crate::services::status_service::StatusService::new();
+    let mut cmd = state.cmd.lock().await;
+    status_service
+        .generate_status(
+            &mut cmd,
+            &state.settings,
+            state.data_dir.clone(),
+            db_path.into(),
+        )
+        .map_err(|e| {
+            log::warn!("Failed to generate status: {e}");
+            StatusCode::INTERNAL_SERVER_ERROR
+        })
+}
+
+#[utoipa::path(
+    get,
+    path = "/api/status",
+    operation_id = "keep_status",
+    summary = "Get system status",
+    description = "Retrieve system status including database info, storage paths, compression engines, and metadata plugins.",
+    responses(
+        (status = 200, description = "System status retrieved", body = StatusInfoResponse),
+        (status = 401, description = "Unauthorized"),
+        (status = 500, description = "Internal server error")
+    ),
+    security(
+        ("bearerAuth" = [])
+    ),
+    tag = "status"
+)]
+/// Axum handler for the /api/status GET endpoint.
+///
+/// Generates and returns comprehensive system status using the StatusService.
+/// Includes paths, plugins, compression info, and configuration details.
+///
+/// # Arguments
+///
+/// * `State(state)` - The shared AppState containing settings, DB, and paths.
+///
+/// # Returns
+///
+/// * `Ok(Json<StatusInfoResponse>)` - Success response with status data.
+/// * `Err(StatusCode)` - HTTP error status (e.g., 500 for internal errors; 401 if auth fails elsewhere).
+///
+/// # Errors
+///
+/// Returns StatusCode::INTERNAL_SERVER_ERROR if status generation panics or fails (current impl assumes success).
+/// Auth errors are handled by middleware before reaching this handler.
+///
+/// # Examples
+///
+/// ```ignore
+/// // In an Axum app:
+/// async fn app() -> Result<Json<StatusInfoResponse>, StatusCode> {
+///     handle_status(State(app_state)).await
+/// }
+/// ```
+pub async fn handle_status(
+    State(state): State<AppState>,
+) -> Result<Json<StatusInfoResponse>, StatusCode> {
+    let status_info = generate_status(&state).await?;
+
+    let response = StatusInfoResponse {
+        success: true,
+        data: Some(status_info),
+        error: None,
+    };
+
+    Ok(Json(response))
+}
+
+#[derive(Debug, serde::Serialize, serde::Deserialize, utoipa::ToSchema)]
+pub struct PluginsStatusResponse {
+    pub meta_plugins: std::collections::HashMap<String, crate::common::status::MetaPluginInfo>,
+    pub filter_plugins: Vec<crate::common::status::FilterPluginInfo>,
+    pub compression: Vec<crate::common::status::CompressionInfo>,
+}
+
+#[utoipa::path(
+    get,
+    path = "/api/plugins/status",
+    operation_id = "keep_plugins_status",
+    summary = "Get plugins status",
+    description = "Retrieve detailed status of all available plugins including meta, filter, and compression plugins.",
+    responses(
+        (status = 200, description = "Plugins status retrieved", body = ApiResponse<PluginsStatusResponse>),
+        (status = 401, description = "Unauthorized"),
+        (status = 500, description = "Internal server error")
+    ),
+    security(
+        ("bearerAuth" = [])
+    ),
+    tag = "status"
+)]
+pub async fn handle_plugins_status(
+    State(state): State<AppState>,
+) -> Result<Json<crate::modes::server::common::ApiResponse<PluginsStatusResponse>>, StatusCode> {
+    let status_info = generate_status(&state).await?;
+
+    let response_data = PluginsStatusResponse {
+        meta_plugins: status_info.meta_plugins,
+        filter_plugins: status_info.filter_plugins,
+        compression: status_info.compression,
+    };
+
+    let response = crate::modes::server::common::ApiResponse::<PluginsStatusResponse> {
+        success: true,
+        data: Some(response_data),
+        error: None,
+    };
+
+    Ok(Json(response))
+}

src/modes/server/auth.rs

@@ -0,0 +1,118 @@
+use axum::http::Method;
+use jsonwebtoken::{DecodingKey, TokenData, Validation, decode};
+use log::debug;
+use serde::Deserialize;
+
+/// JWT claims for permission-based access control.
+///
+/// External token generators should include these claims in the JWT payload.
+/// The server validates the signature and checks permissions for each request.
+///
+/// # Example token payload
+///
+/// ```json
+/// {
+///   "sub": "my-client",
+///   "exp": 1735689600,
+///   "read": true,
+///   "write": true,
+///   "delete": false
+/// }
+/// ```
+#[derive(Debug, Deserialize)]
+pub struct Claims {
+    /// Subject (client identifier).
+    pub sub: String,
+    /// Expiration time (Unix timestamp).
+    pub exp: usize,
+    /// Read permission (GET requests).
+    #[serde(default)]
+    pub read: bool,
+    /// Write permission (POST/PUT requests).
+    #[serde(default)]
+    pub write: bool,
+    /// Delete permission (DELETE requests).
+    #[serde(default)]
+    pub delete: bool,
+}
+
+/// Returns the required permission for an HTTP method.
+///
+/// # Mapping
+///
+/// - GET, HEAD → "read"
+/// - POST, PUT, PATCH → "write"
+/// - DELETE → "delete"
+///
+/// # Arguments
+///
+/// * `method` - The HTTP method of the incoming request.
+///
+/// # Returns
+///
+/// A string slice representing the required permission.
+pub fn required_permission(method: &Method) -> &'static str {
+    if method == Method::GET || method == Method::HEAD {
+        "read"
+    } else if method == Method::DELETE {
+        "delete"
+    } else {
+        "write"
+    }
+}
+
+/// Checks if the JWT claims grant the required permission.
+///
+/// # Arguments
+///
+/// * `claims` - The validated JWT claims.
+/// * `permission` - The required permission string ("read", "write", or "delete").
+///
+/// # Returns
+///
+/// `true` if the claims grant the permission, `false` otherwise.
+pub fn check_permission(claims: &Claims, permission: &str) -> bool {
+    match permission {
+        "read" => claims.read,
+        "write" => claims.write,
+        "delete" => claims.delete,
+        _ => false,
+    }
+}
+
+/// Validates a JWT token and returns the claims.
+///
+/// Uses HMAC-SHA256 signature verification with the provided secret.
+///
+/// # Arguments
+///
+/// * `token` - The JWT token string (without "Bearer " prefix).
+/// * `secret` - The secret key used to verify the signature.
+///
+/// # Returns
+///
+/// * `Ok(Claims)` - The validated claims if the token is valid.
+/// * `Err(String)` - A human-readable error message if validation fails.
+pub fn validate_jwt(token: &str, secret: &str) -> Result<Claims, String> {
+    let mut validation = Validation::new(jsonwebtoken::Algorithm::HS256);
+    validation.algorithms = vec![jsonwebtoken::Algorithm::HS256];
+    validation.set_required_spec_claims(&["exp", "sub"]);
+
+    let token_data: TokenData<Claims> = decode::<Claims>(
+        token,
+        &DecodingKey::from_secret(secret.as_bytes()),
+        &validation,
+    )
+    .map_err(|e| {
+        debug!("JWT validation failed: {e}");
+        match e.kind() {
+            jsonwebtoken::errors::ErrorKind::ExpiredSignature => "Token expired".to_string(),
+            jsonwebtoken::errors::ErrorKind::InvalidSignature => "Invalid token".to_string(),
+            jsonwebtoken::errors::ErrorKind::InvalidToken => "Malformed token".to_string(),
+            jsonwebtoken::errors::ErrorKind::ImmatureSignature => "Token not yet valid".to_string(),
+            _ => "Invalid token".to_string(),
+        }
+    })?;
+
+    Ok(token_data.claims)
+}

src/modes/server/content.rs

@@ -1,181 +0,0 @@
-use anyhow::{Result, anyhow};
-use axum::{
-    extract::{ConnectInfo, Path, Query, State},
-    http::{HeaderMap, StatusCode},
-    response::Json,
-};
-use log::warn;
-use serde_json::json;
-use std::collections::HashMap;
-use std::io::Read;
-use std::net::SocketAddr;
-use std::path::PathBuf;
-use std::str::FromStr;
-
-use crate::compression_engine::{CompressionType, get_compression_engine};
-use crate::db;
-use super::common::{AppState, ApiResponse, TagsQuery, check_auth};
-
-pub async fn handle_get_content_latest(
-    State(state): State<AppState>,
-    Query(params): Query<TagsQuery>,
-    headers: HeaderMap,
-    ConnectInfo(addr): ConnectInfo<SocketAddr>,
-) -> Result<Json<ApiResponse<String>>, StatusCode> {
-    if !check_auth(&headers, &state.password) {
-        warn!("Unauthorized request to /content from {}", addr);
-        return Err(StatusCode::UNAUTHORIZED);
-    }
-    
-    let mut conn = state.db.lock().await;
-    
-    let item = if let Some(tags_str) = params.tags {
-        let tags: Vec<String> = tags_str.split(',').map(|t| t.trim().to_string()).collect();
-        db::get_item_matching(&mut *conn, &tags, &HashMap::new())
-            .map_err(|e| {
-                warn!("Failed to get item matching tags {:?} for content: {}", tags, e);
-                StatusCode::INTERNAL_SERVER_ERROR
-            })?
-    } else {
-        db::get_item_last(&mut *conn).map_err(|e| {
-            warn!("Failed to get last item for content: {}", e);
-            StatusCode::INTERNAL_SERVER_ERROR
-        })?
-    };
-    
-    if let Some(item) = item {
-        match get_item_content(&item, &state.data_dir).await {
-            Ok(content) => {
-                let response = ApiResponse {
-                    success: true,
-                    data: Some(content),
-                    error: None,
-                };
-                Ok(Json(response))
-            }
-            Err(e) => {
-                warn!("Failed to get content for item {}: {}", item.id.unwrap_or(0), e);
-                let response = ApiResponse::<String> {
-                    success: false,
-                    data: None,
-                    error: Some(format!("Failed to retrieve content: {}", e)),
-                };
-                Ok(Json(response))
-            }
-        }
-    } else {
-        Err(StatusCode::NOT_FOUND)
-    }
-}
-
-pub async fn handle_get_content(
-    State(state): State<AppState>,
-    Path(item_id): Path<String>,
-    headers: HeaderMap,
-    ConnectInfo(addr): ConnectInfo<SocketAddr>,
-) -> Result<Json<ApiResponse<String>>, StatusCode> {
-    if !check_auth(&headers, &state.password) {
-        warn!("Unauthorized request to /content/{} from {}", item_id, addr);
-        return Err(StatusCode::UNAUTHORIZED);
-    }
-    
-    if let Ok(id) = item_id.parse::<i64>() {
-        // Validate that item ID is positive to prevent path traversal issues
-        if id <= 0 {
-            warn!("Invalid item ID {} from {}", id, addr);
-            return Err(StatusCode::BAD_REQUEST);
-        }
-        
-        let mut conn = state.db.lock().await;
-        
-        if let Some(item) = db::get_item(&mut *conn, id).map_err(|e| {
-            warn!("Failed to get item {} for content: {}", id, e);
-            StatusCode::INTERNAL_SERVER_ERROR
-        })? {
-            match get_item_content(&item, &state.data_dir).await {
-                Ok(content) => {
-                    let response = ApiResponse {
-                        success: true,
-                        data: Some(content),
-                        error: None,
-                    };
-                    Ok(Json(response))
-                }
-                Err(e) => {
-                    warn!("Failed to get content for item {}: {}", id, e);
-                    let response = ApiResponse::<String> {
-                        success: false,
-                        data: None,
-                        error: Some(format!("Failed to retrieve content: {}", e)),
-                    };
-                    Ok(Json(response))
-                }
-            }
-        } else {
-            Err(StatusCode::NOT_FOUND)
-        }
-    } else {
-        Err(StatusCode::BAD_REQUEST)
-    }
-}
-
-async fn get_item_content(item: &db::Item, data_dir: &PathBuf) -> Result<String> {
-    let item_id = item.id.ok_or_else(|| anyhow!("Item missing ID"))?;
-    
-    // Validate that item ID is positive to prevent path traversal issues
-    if item_id <= 0 {
-        return Err(anyhow!("Invalid item ID: {}", item_id));
-    }
-    
-    let mut item_path = data_dir.clone();
-    item_path.push(item_id.to_string());
-    
-    let compression_type = CompressionType::from_str(&item.compression)?;
-    let compression_engine = get_compression_engine(compression_type)?;
-    
-    // Read the content using the compression engine
-    let mut reader = compression_engine.open(item_path)?;
-    let mut content = String::new();
-    reader.read_to_string(&mut content)?;
-    
-    Ok(content)
-}
-
-pub fn get_content_openapi_spec() -> serde_json::Value {
-    json!({
-        "/content": {
-            "get": {
-                "summary": "Get content of latest item",
-                "parameters": [
-                    {
-                        "name": "tags",
-                        "in": "query",
-                        "schema": {"type": "string"},
-                        "description": "Comma-separated list of tags to filter by"
-                    }
-                ],
-                "responses": {
-                    "200": {"description": "Item content"},
-                    "404": {"description": "No items found"}
-                }
-            }
-        },
-        "/content/{id}": {
-            "get": {
-                "summary": "Get content by item ID",
-                "parameters": [
-                    {
-                        "name": "id",
-                        "in": "path",
-                        "required": true,
-                        "schema": {"type": "integer"}
-                    }
-                ],
-                "responses": {
-                    "200": {"description": "Item content"},
-                    "404": {"description": "Item not found"}
-                }
-            }
-        }
-    })
-}

src/modes/server/docs.rs

@@ -1,110 +0,0 @@
-use axum::response::{Html, Json};
-use serde_json::json;
-
-use super::status::get_status_openapi_spec;
-use super::items::get_items_openapi_spec;
-use super::content::get_content_openapi_spec;
-
-pub async fn handle_openapi() -> Json<serde_json::Value> {
-    let mut paths = json!({});
-    
-    // Merge all endpoint specifications
-    let status_paths = get_status_openapi_spec();
-    let items_paths = get_items_openapi_spec();
-    let content_paths = get_content_openapi_spec();
-    
-    // Merge the path objects
-    if let serde_json::Value::Object(ref mut paths_map) = paths {
-        if let serde_json::Value::Object(status_map) = status_paths {
-            for (key, value) in status_map {
-                paths_map.insert(key, value);
-            }
-        }
-        if let serde_json::Value::Object(items_map) = items_paths {
-            for (key, value) in items_map {
-                paths_map.insert(key, value);
-            }
-        }
-        if let serde_json::Value::Object(content_map) = content_paths {
-            for (key, value) in content_map {
-                paths_map.insert(key, value);
-            }
-        }
-    }
-    
-    let openapi_spec = json!({
-        "openapi": "3.0.0",
-        "info": {
-            "title": "Keep API",
-            "version": "1.0.0",
-            "description": "REST API for the Keep data storage system"
-        },
-        "servers": [
-            {
-                "url": "/",
-                "description": "Local server"
-            }
-        ],
-        "components": {
-            "securitySchemes": {
-                "bearerAuth": {
-                    "type": "http",
-                    "scheme": "bearer"
-                }
-            },
-            "schemas": {
-                "ItemInfo": {
-                    "type": "object",
-                    "properties": {
-                        "id": {"type": "integer"},
-                        "ts": {"type": "string", "format": "date-time"},
-                        "size": {"type": "integer", "nullable": true},
-                        "compression": {"type": "string"},
-                        "tags": {"type": "array", "items": {"type": "string"}},
-                        "metadata": {"type": "object"}
-                    }
-                },
-                "StatusInfo": {
-                    "type": "object",
-                    "properties": {
-                        "version": {"type": "string"},
-                        "database_path": {"type": "string"},
-                        "data_directory": {"type": "string"},
-                        "compression_engines": {"type": "array", "items": {"type": "string"}},
-                        "meta_plugins": {"type": "array", "items": {"type": "string"}}
-                    }
-                }
-            }
-        },
-        "security": [{"bearerAuth": []}],
-        "paths": paths
-    });
-    
-    Json(openapi_spec)
-}
-
-pub async fn handle_swagger_ui() -> Html<&'static str> {
-    let html = r#"<!DOCTYPE html>
-<html>
-<head>
-    <title>Keep API Documentation</title>
-    <link rel="stylesheet" type="text/css" href="https://unpkg.com/swagger-ui-dist@3.52.5/swagger-ui.css" />
-</head>
-<body>
-    <div id="swagger-ui"></div>
-    <script src="https://unpkg.com/swagger-ui-dist@3.52.5/swagger-ui-bundle.js"></script>
-    <script>
-        SwaggerUIBundle({
-            url: '/openapi.json',
-            dom_id: '#swagger-ui',
-            presets: [
-                SwaggerUIBundle.presets.apis,
-                SwaggerUIBundle.presets.standalone
-            ]
-        });
-    </script>
-</body>
-</html>"#;
-    
-    Html(html)
-}