Compare commits

...

8 Commits

Author SHA1 Message Date
8379ae2136 refactor: rename plugin features with type prefix for consistency
- Plugin features now use type_ prefix (meta_magic, filter_grep, etc.)
- Added meta_all_musl and filter_all_musl for MUSL-compatible builds
- grep filter plugin made optional via filter_grep feature flag
- Removed regex crate from grep-related code, uses strip_prefix instead
- Updated CHANGELOG.md with breaking change documentation
2026-03-21 17:36:29 -03:00
12de215527 feat: feature-gate CLI args by server/client features
- CLI now shows only relevant options: --server and --server-* args
  hidden when built without 'server' feature; --client-* args hidden
  without 'client' feature. Run --help only displays applicable options.
- Removed verbose 'conflicts_with_all' from all mode args — clap's
  implicit group("mode") already enforces mutual exclusivity.
- 'server' feature now includes TLS/HTTPS by default (axum-server);
  'tls' feature removed. rustls already available via client/ureq.
- Gated KeepModes::Server, server mode detection, and server-password
  validation in main.rs.
- Gated server arg reads in config.rs.
- Removed redundant #[cfg(feature = "tls")] guards from server/mod.rs.
- Gated resolve_item_id/resolve_item_ids helpers in common.rs.
- All 4 feature combinations (server+client, server-only, client-only,
  neither) compile and pass tests.
2026-03-21 16:26:27 -03:00
e2cb36d2a8 feat(server): add file_size to API ItemInfo response 2026-03-21 14:03:58 -03:00
0004324301 perf: pre-allocate status info collections with known capacities 2026-03-21 13:54:37 -03:00
b3edfe7de6 chore: code review cleanup — fixes, deps, docs
Fixed:
- CLI help typo: "metatdata" -> "metadata"
- Filter buffer OOM: check size before loading into memory

Changed:
- #[inline] on HTML escape helpers for hot path performance
- Replaced once_cell and lazy_static with std::sync::LazyLock
- Removed unused once_cell and lazy_static crate dependencies

Refactored:
- Added module-level doc to services/ module

Documentation:
- README.md: zstd is native not external, "none" -> "raw"
- DESIGN.md: current schema and meta plugins section
- CHANGELOG.md: Unreleased section populated
2026-03-21 11:44:37 -03:00
ab2fb07505 docs: add changelog update instructions to AGENTS.md 2026-03-21 10:56:43 -03:00
547f0b5d11 docs: add CHANGELOG.md following Keep a Changelog format 2026-03-21 10:55:16 -03:00
30d7836bcf refactor: deduplicate ItemInfo, improve error handling, fix pre-existing bugs
- Move ItemInfo to services/types.rs for sharing between client and server
- Replace .expect() in compression_service with proper error handling
- Add CoreError::PayloadTooLarge variant for semantic error handling
- Export CoreError from lib.rs for library users
- Unify get_item_meta_name/value to take &str instead of String
- Extract item_path() helper in ItemService to reduce duplication
- Add warning logs for silent errors in list.rs
- Fix pre-existing borrow errors: tx moved in export handler,
  item_with_meta partial move in TryFrom implementation
- Fix unused data_dir variables in server code
2026-03-21 10:43:26 -03:00
35 changed files with 529 additions and 350 deletions

View File

@@ -53,3 +53,13 @@ TERM=dumb cargo build --features server # With server feature
- Use `html_escape` crate for all user-controlled data in HTML pages - Use `html_escape` crate for all user-controlled data in HTML pages
- `esc()` for text content, `esc_attr()` for HTML attributes - `esc()` for text content, `esc_attr()` for HTML attributes
- Security headers middleware: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin` - Security headers middleware: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin`
## Changelog
The project uses [Keep a Changelog](https://keepachangelog.com/). The changelog lives at `CHANGELOG.md` in the project root.
- **Always update `CHANGELOG.md`** when making changes that affect users (new features, breaking changes, bug fixes, etc.)
- Add entries under the `[Unreleased]` section using these categories: `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, `Security`
- Keep descriptions concise and user-focused — what changed from the user's perspective, not implementation details
- Commit changelog updates in the same commit as the feature/fix they document
- Before releasing a new version, move `[Unreleased]` entries to a versioned section (e.g., `[0.2.0] - YYYY-MM-DD`) and add a new empty `[Unreleased]` above it

107
CHANGELOG.md Normal file
View File

@@ -0,0 +1,107 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- New `filter_grep` feature to optionally include the grep filter plugin (regex-based line filtering). Disabling this feature removes the `regex` crate and its ~800 KiB dependency stack from the binary.
- New `meta_all_musl` feature for all MUSL-compatible meta plugins (excludes `meta_magic` which requires libmagic)
- New `filter_all_musl` feature for all MUSL-compatible filter plugins
- Database index on `items(ts)` column for faster ORDER BY sorting
- Server API `ItemInfo` now includes `file_size` — actual filesystem-reported size of the item data file
### Changed
- CLI args now feature-gated: `--server` and related options hidden when built without `server` feature; `--client-*` options hidden when built without `client` feature. Run `--help` only shows relevant options.
- `server` Cargo feature now includes TLS support by default (`axum-server`); `tls` feature removed
- Clap `conflicts_with_all` removed from all mode args — exclusivity now handled by implicit `group("mode")`
- Filter plugins check size before loading content into memory (prevents OOM on large inputs)
- Status page pre-allocates collections with known capacities (meta plugins, compression info)
- `#[inline]` on HTML escape helper functions (`esc`, `esc_attr`) for hot path performance
- Removed `once_cell` crate (replaced with `std::sync::LazyLock` from Rust 1.80)
- Removed `lazy_static` crate (replaced with `std::sync::LazyLock`)
### Breaking
- Plugin feature flags renamed with type prefix for consistency:
- `magic``meta_magic`
- `infer``meta_infer`
- `tree_magic_mini``meta_tree_magic_mini`
- `tokens``meta_tokens`
- `grep``filter_grep`
- `all-meta-plugins``meta_all`
- `all-filter-plugins``filter_all`
### Fixed
- CLI help text typo: "metatdata" → "metadata" in `--get` and `--info` descriptions
### Refactored
- Added module-level documentation to `services/` module
### Documentation
- README.md: Fixed compression table — zstd is native (not external), "none" renamed to "raw"
- DESIGN.md: Updated schema to reflect current `items` table columns and meta plugin inventory
## [0.1.0] - 2026-03-21
### Added
- Streaming tar-based export (`--export`) producing `.keep.tar` archives without loading entire files into memory
- Streaming tar-based import (`--import`) extracting `.keep.tar` archives with new IDs
- Server endpoints `GET /api/export` and `POST /api/import`
- ID-based filtering for `--list` (`keep -l 1 2 3` lists specific items by ID)
- Server API accepts optional `ids` query parameter on `GET /api/item/`
- `--ids-only` flag for `--list` mode for scripting
- `infer` and `tree_magic_mini` meta plugins for MIME type detection
- Native `zstd` compression plugin as default
- Configurable compression via `--compression` flag
- Export/import modes with format detection (JSON, YAML, binary)
- `XDG_CONFIG_HOME` support for default config file location
- `XDG_DATA_HOME` support for default storage location
- Tilde (`~`) expansion in config file paths
### Changed
- `CompressionType::None` renamed to `CompressionType::Raw` (with `"none"` as alias for backward compatibility)
- `items.size` column renamed to `items.uncompressed_size`
- Added `items.compressed_size` column tracking compressed file size on disk
- Added `items.closed` column tracking whether an item is fully written
- Default `list_format` in config now matches CLI default (7 vs 5 columns)
- All filter plugins share deduplicated option implementations
### Refactored
- Extracted `spawn_body_reader()` and `check_binary_content()` helpers for streaming uploads
- Extracted `yaml_value_to_string()` helper for meta plugins
- Extracted `item_path()` helper in `ItemService` to reduce path duplication
- Unified `get_item_meta_name`/`value` to take `&str` instead of `String`
- Shared `ItemInfo` struct between client and server
- Compression service now returns `Result` types instead of panicking via `.expect()`
- `ApiResponse::ok()` and `ApiResponse::empty()` constructors
- `meta_filter()` helper on `Settings` for consistent filtering
- Added `tag_names()` method on `ItemWithMeta`
- `filter_clone_box!` macro for filter plugin cloning
### Fixed
- Panic guards in diff, compression engine, and spawned threads
- Pre-existing borrow errors in export handler and `TryFrom` implementation
- TOCTOU race in `stream_raw_content_response`
- Swallowed write errors in meta plugins (digest, magic_file, exec)
- Truncated uploads (413) now properly store compressed data
- `term::stderr().unwrap()` panic in `item_service`
- `.unwrap()` panics in compression engine `Read`/`Write` impls
- Client API errors now propagate to user instead of being swallowed
- Import endpoint returns 413 on `max_body_size` instead of truncating
- `keep --list` uses `list_format` from config in all modes
- All tables respect `table_config` from settings
- `DisplayListItem` struct removed (was unused)
- `#[serde(alias = "size")]` on `ImportMeta` for backward compatibility

2
Cargo.lock generated
View File

@@ -1727,7 +1727,6 @@ dependencies = [
"inventory", "inventory",
"is-terminal", "is-terminal",
"jsonwebtoken", "jsonwebtoken",
"lazy_static",
"libc", "libc",
"local-ip-address", "local-ip-address",
"log", "log",
@@ -1735,7 +1734,6 @@ dependencies = [
"magic", "magic",
"md5", "md5",
"nix", "nix",
"once_cell",
"os_pipe", "os_pipe",
"pest", "pest",
"pest_derive", "pest_derive",

View File

@@ -35,7 +35,6 @@ hyper = { version = "1.0", features = ["full"] }
http-body-util = "0.1" http-body-util = "0.1"
inventory = "0.3" inventory = "0.3"
is-terminal = "0.4" is-terminal = "0.4"
lazy_static = "1.5"
libc = "0.2" libc = "0.2"
local-ip-address = "0.6" local-ip-address = "0.6"
log = "0.4" log = "0.4"
@@ -45,10 +44,9 @@ magic = { version = "0.13", optional = true }
infer = { version = "0.19", optional = true } infer = { version = "0.19", optional = true }
tree_magic_mini = { version = "3.2", optional = true } tree_magic_mini = { version = "3.2", optional = true }
nix = { version = "0.30", features = ["fs", "process"] } nix = { version = "0.30", features = ["fs", "process"] }
once_cell = "1.21"
comfy-table = "7.2" comfy-table = "7.2"
pwhash = "1.0" pwhash = "1.0"
regex = "1.10" regex = { version = "1.10", optional = true }
ringbuf = "0.4" ringbuf = "0.4"
rusqlite = { version = "0.37", features = ["bundled", "array", "chrono"] } rusqlite = { version = "0.37", features = ["bundled", "array", "chrono"] }
rusqlite_migration = "2.3" rusqlite_migration = "2.3"
@@ -87,19 +85,20 @@ tiktoken-rs = { version = "0.9", optional = true }
tempfile = "3.3" tempfile = "3.3"
[features] [features]
# Default features include core compression engines and swagger UI # Default features include core compression engines plugins that support MUSL
default = [ default = [
"client", "client",
"gzip", "gzip",
"infer", "filter_grep",
"meta_infer",
"lz4", "lz4",
"tokens", "meta_tokens",
"tree_magic_mini", "meta_tree_magic_mini",
"zstd" "zstd"
] ]
# Server feature (includes axum and related dependencies) # Server feature (includes axum and TLS/HTTPS via axum-server; rustls already available via client/ureq)
server = ["dep:axum", "dep:tower", "dep:tower-http", "dep:utoipa", "dep:jsonwebtoken"] server = ["dep:axum", "dep:tower", "dep:tower-http", "dep:utoipa", "dep:jsonwebtoken", "dep:axum-server"]
# Compression features # Compression features
gzip = ["flate2"] gzip = ["flate2"]
@@ -108,14 +107,18 @@ bzip2 = []
xz = [] xz = []
zstd = ["dep:zstd"] zstd = ["dep:zstd"]
# Plugin features (meta and filter) # Meta plugin features
all-meta-plugins = ["dep:magic", "dep:infer", "dep:tree_magic_mini"] meta_magic = ["dep:magic"]
all-filter-plugins = [] meta_infer = ["dep:infer"]
meta_tree_magic_mini = ["dep:tree_magic_mini"]
meta_tokens = ["dep:tiktoken-rs"]
meta_all = ["meta_magic", "meta_infer", "meta_tree_magic_mini", "meta_tokens"]
meta_all_musl = ["meta_infer", "meta_tree_magic_mini", "meta_tokens"]
# Individual plugin features # Filter plugin features
magic = ["dep:magic"] filter_grep = ["dep:regex"]
infer = ["dep:infer"] filter_all = ["filter_grep"]
tree_magic_mini = ["dep:tree_magic_mini"] filter_all_musl = ["filter_grep"]
# Swagger UI feature # Swagger UI feature
swagger = ["dep:utoipa-swagger-ui"] swagger = ["dep:utoipa-swagger-ui"]
@@ -123,11 +126,5 @@ swagger = ["dep:utoipa-swagger-ui"]
# Client feature (HTTP client for remote server) # Client feature (HTTP client for remote server)
client = ["dep:ureq", "dep:os_pipe"] client = ["dep:ureq", "dep:os_pipe"]
# TLS feature (HTTPS server support)
tls = ["dep:axum-server"]
# Token counting feature (LLM token support via tiktoken)
tokens = ["dep:tiktoken-rs"]
[dev-dependencies] [dev-dependencies]
rand = "0.9" rand = "0.9"

View File

@@ -117,7 +117,7 @@
## Data Storage ## Data Storage
### Database Schema ### Database Schema
- `items` table: id (primary key), ts (timestamp), size (optional), compression - `items` table: id (primary key), ts (timestamp), uncompressed_size (optional), compressed_size (optional), closed (boolean), compression
- `tags` table: id (foreign key to items), name (tag name) - `tags` table: id (foreign key to items), name (tag name)
- `metas` table: id (foreign key to items), name (meta key), value (meta value) - `metas` table: id (foreign key to items), name (meta key), value (meta value)
- Indexes on tag names and meta names for faster queries - Indexes on tag names and meta names for faster queries
@@ -178,26 +178,25 @@
- None (no compression) - None (no compression)
## Supported Meta Plugins ## Supported Meta Plugins
- FileMagic - File type detection using file command
- FileMime - MIME type detection using file command Meta plugins collect metadata during item save. Each plugin produces one or more key-value pairs:
- FileEncoding - File encoding detection using file command
- LineCount - Line count using wc command - `magic_file` - File type detection using libmagic (when `magic` feature enabled)
- WordCount - Word count using wc command - `infer` - MIME type detection using infer crate (when `infer` feature enabled)
- Cwd - Current working directory - `tree_magic_mini` - MIME type detection using tree_magic_mini (when `tree_magic_mini` feature enabled)
- Binary - Binary file detection - `tokens` - LLM token counting using tiktoken (when `tokens` feature enabled)
- Uid - Current user ID - `text` - Text analysis: line count, word count, char count, line average length
- User - Current username - `digest` - SHA-256 and MD5 checksums
- Gid - Current group ID - `hostname` - System hostname (full and short)
- Group - Current group name - `cwd` - Current working directory
- Shell - Shell path from SHELL environment variable - `user` - Current username and UID
- ShellPid - Shell process ID from PPID environment variable - `shell` - Shell path from SHELL environment variable
- KeepPid - Keep process ID - `shell_pid` - Shell process ID from PPID
- DigestSha256 - SHA-256 digest - `keep_pid` - Keep process ID
- DigestMd5 - MD5 digest using md5sum command - `env` - Arbitrary environment variables (via `KEEP_META_ENV_*` prefix)
- ReadTime - Time taken to read data - `exec` - Execute external commands for custom metadata
- ReadRate - Rate of data reading - `read_time` - Time taken to read content
- Hostname - System hostname - `read_rate` - Content read rate (bytes/second)
- FullHostname - Fully qualified domain name
## Testing Strategy ## Testing Strategy
- Unit tests for each module in `src/tests/` - Unit tests for each module in `src/tests/`

View File

@@ -345,8 +345,8 @@ Items are compressed automatically on save. Default: LZ4.
| `gzip` | Internal | Fast | Good | | `gzip` | Internal | Fast | Good |
| `bzip2` | External | Slow | Better | | `bzip2` | External | Slow | Better |
| `xz` | External | Slowest | Best | | `xz` | External | Slowest | Best |
| `zstd` | External | Fast | Good | | `zstd` | Internal | Fast | Good |
| `none` | Internal | N/A | N/A | | `raw` | Internal | N/A | N/A |
```sh ```sh
# Specify compression per item # Specify compression per item

View File

@@ -24,81 +24,80 @@ pub struct Args {
/// Struct for mode-specific arguments, defining CLI flags for different operations. /// Struct for mode-specific arguments, defining CLI flags for different operations.
#[derive(Parser, Debug, Clone)] #[derive(Parser, Debug, Clone)]
pub struct ModeArgs { pub struct ModeArgs {
#[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["get", "diff", "list", "delete", "info", "update", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short, long)]
#[arg(help("Save an item using any tags or metadata provided"))] #[arg(help("Save an item using any tags or metadata provided"))]
pub save: bool, pub save: bool,
#[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "diff", "list", "delete", "info", "update", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short, long)]
#[arg(help( #[arg(help("Get an item either by its ID or by a combination of matching tags and metadata"))]
"Get an item either by it's ID or by a combination of matching tags and metatdata"
))]
pub get: bool, pub get: bool,
#[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "list", "delete", "info", "update", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), long)]
#[arg(help("Show a diff between two items by ID"))] #[arg(help("Show a diff between two items by ID"))]
pub diff: bool, pub diff: bool,
#[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "delete", "info", "update", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short, long)]
#[arg(help("List items, filtering on tags or metadata if given"))] #[arg(help("List items, filtering on tags or metadata if given"))]
pub list: bool, pub list: bool,
#[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "list", "info", "update", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short, long)]
#[arg(help("Delete items either by ID or by matching tags"))] #[arg(help("Delete items either by ID or by matching tags"))]
#[arg(requires = "ids_or_tags")] #[arg(requires = "ids_or_tags")]
pub delete: bool, pub delete: bool,
#[arg(group("mode"), help_heading("Mode Options"), short, long, conflicts_with_all(["save", "get", "diff", "list", "delete", "update", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short, long)]
#[arg(help( #[arg(help("Get an item either by its ID or by a combination of matching tags and metadata"))]
"Get an item either by it's ID or by a combination of matching tags and metatdata"
))]
pub info: bool, pub info: bool,
#[arg(group("mode"), help_heading("Mode Options"), short('u'), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "status", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short('u'), long)]
#[arg(help("Update an item's tags and metadata by ID"))] #[arg(help("Update an item's tags and metadata by ID"))]
pub update: bool, pub update: bool,
#[arg(group("mode"), help_heading("Mode Options"), short('S'), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "server", "status_plugins", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), short('S'), long)]
#[arg(help("Show status of directories and supported compression algorithms"))] #[arg(help("Show status of directories and supported compression algorithms"))]
pub status: bool, pub status: bool,
#[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "status", "server", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), long)]
#[arg(help("Show available plugins and their configurations"))] #[arg(help("Show available plugins and their configurations"))]
pub status_plugins: bool, pub status_plugins: bool,
#[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "status", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), long)]
#[arg(help("Export items to a .keep.tar archive (requires IDs or tags)"))] #[arg(help("Export items to a .keep.tar archive (requires IDs or tags)"))]
pub export: bool, pub export: bool,
#[arg(group("mode"), help_heading("Mode Options"), long, value_name("FILE"), conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "status", "export"]))] #[arg(group("mode"), help_heading("Mode Options"), long, value_name("FILE"))]
#[arg(help("Import items from a .keep.tar archive or legacy .meta.yml file"))] #[arg(help("Import items from a .keep.tar archive or legacy .meta.yml file"))]
pub import: Option<String>, pub import: Option<String>,
#[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "status"]))] #[cfg(feature = "server")]
#[arg(group("mode"), help_heading("Mode Options"), long)]
#[arg(help("Start REST HTTP server"))] #[arg(help("Start REST HTTP server"))]
pub server: bool, pub server: bool,
#[arg(group("mode"), help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "status", "server", "export", "import"]))] #[arg(group("mode"), help_heading("Mode Options"), long)]
#[arg(help("Generate default configuration and output to stdout"))] #[arg(help("Generate default configuration and output to stdout"))]
pub generate_config: bool, pub generate_config: bool,
#[arg(help_heading("Mode Options"), long, conflicts_with_all(["save", "get", "diff", "list", "delete", "info", "update", "status", "server", "generate_config", "export", "import"]))] #[arg(help_heading("Mode Options"), long)]
#[arg(help("Generate shell completion script"))] #[arg(help("Generate shell completion script"))]
pub generate_completion: Option<Shell>, pub generate_completion: Option<Shell>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_ADDRESS"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_ADDRESS"))]
#[arg(help("Server address to bind to"))] #[arg(help("Server address to bind to"))]
pub server_address: Option<String>, pub server_address: Option<String>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PORT"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PORT"))]
#[arg(help("Server port to bind to"))] #[arg(help("Server port to bind to"))]
pub server_port: Option<u16>, pub server_port: Option<u16>,
#[cfg(feature = "tls")] #[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_CERT"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_CERT"))]
#[arg(help("Path to TLS certificate file (PEM) for HTTPS"))] #[arg(help("Path to TLS certificate file (PEM) for HTTPS"))]
pub server_cert: Option<PathBuf>, pub server_cert: Option<PathBuf>,
#[cfg(feature = "tls")] #[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_KEY"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_KEY"))]
#[arg(help("Path to TLS private key file (PEM) for HTTPS"))] #[arg(help("Path to TLS private key file (PEM) for HTTPS"))]
pub server_key: Option<PathBuf>, pub server_key: Option<PathBuf>,
@@ -253,24 +252,29 @@ pub struct OptionsArgs {
#[arg(help("Output format (only works with --info, --status, --list)"))] #[arg(help("Output format (only works with --info, --status, --list)"))]
pub output_format: Option<String>, pub output_format: Option<String>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PASSWORD"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PASSWORD"))]
#[arg(help("Password for server authentication (requires --server)"))] #[arg(help("Password for server authentication (requires --server)"))]
pub server_password: Option<String>, pub server_password: Option<String>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PASSWORD_HASH"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PASSWORD_HASH"))]
#[arg(help("Password hash for server authentication (requires --server)"))] #[arg(help("Password hash for server authentication (requires --server)"))]
pub server_password_hash: Option<String>, pub server_password_hash: Option<String>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_USERNAME"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_USERNAME"))]
#[arg(help( #[arg(help(
"Username for server Basic authentication (requires --server, defaults to 'keep')" "Username for server Basic authentication (requires --server, defaults to 'keep')"
))] ))]
pub server_username: Option<String>, pub server_username: Option<String>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_JWT_SECRET"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_JWT_SECRET"))]
#[arg(help("JWT secret for token-based authentication (requires --server)"))] #[arg(help("JWT secret for token-based authentication (requires --server)"))]
pub server_jwt_secret: Option<String>, pub server_jwt_secret: Option<String>,
#[cfg(feature = "server")]
#[arg( #[arg(
help_heading("Server Options"), help_heading("Server Options"),
long, long,
@@ -279,6 +283,7 @@ pub struct OptionsArgs {
#[arg(help("Path to file containing JWT secret (requires --server)"))] #[arg(help("Path to file containing JWT secret (requires --server)"))]
pub server_jwt_secret_file: Option<PathBuf>, pub server_jwt_secret_file: Option<PathBuf>,
#[cfg(feature = "server")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_MAX_BODY_SIZE"))] #[arg(help_heading("Server Options"), long, env("KEEP_SERVER_MAX_BODY_SIZE"))]
#[arg(help("Maximum request body size in bytes (requires --server, default: unlimited)"))] #[arg(help("Maximum request body size in bytes (requires --server, default: unlimited)"))]
pub server_max_body_size: Option<u64>, pub server_max_body_size: Option<u64>,

View File

@@ -1,22 +1,9 @@
use crate::services::error::CoreError; use crate::services::{ItemInfo, error::CoreError};
use base64::Engine; use base64::Engine;
use serde::de::DeserializeOwned; use serde::de::DeserializeOwned;
use std::collections::HashMap; use std::collections::HashMap;
use std::io::Read; use std::io::Read;
/// Item information returned from the server API.
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
pub struct ItemInfo {
pub id: i64,
pub ts: String,
pub uncompressed_size: Option<i64>,
pub compressed_size: Option<i64>,
pub closed: bool,
pub compression: String,
pub tags: Vec<String>,
pub metadata: HashMap<String, String>,
}
/// Percent-encode a value for use in a URL query string. /// Percent-encode a value for use in a URL query string.
fn url_encode(s: &str) -> String { fn url_encode(s: &str) -> String {
let mut result = String::with_capacity(s.len() * 3); let mut result = String::with_capacity(s.len() * 3);

View File

@@ -89,7 +89,7 @@ pub fn generate_status_info(
}; };
let _default_type = crate::compression_engine::default_compression_type(); let _default_type = crate::compression_engine::default_compression_type();
let mut compression_info = Vec::new(); let mut compression_info = Vec::with_capacity(CompressionType::iter().count());
// Sort compression types by their string representation // Sort compression types by their string representation
let mut sorted_compression_types: Vec<CompressionType> = CompressionType::iter().collect(); let mut sorted_compression_types: Vec<CompressionType> = CompressionType::iter().collect();
@@ -141,7 +141,8 @@ pub fn generate_status_info(
}); });
} }
let mut meta_plugins_map = std::collections::HashMap::new(); let mut meta_plugins_map =
std::collections::HashMap::with_capacity(MetaPluginType::iter().count());
let mut enabled_meta_plugins_vec = Vec::new(); let mut enabled_meta_plugins_vec = Vec::new();
// Sort meta plugin types by their string representation to avoid creating plugins just for sorting // Sort meta plugin types by their string representation to avoid creating plugins just for sorting

View File

@@ -7,8 +7,6 @@ use strum::{Display, EnumIter, EnumString};
use log::*; use log::*;
use lazy_static::lazy_static;
extern crate enum_map; extern crate enum_map;
use enum_map::enum_map; use enum_map::enum_map;
use enum_map::{Enum, EnumMap}; use enum_map::{Enum, EnumMap};
@@ -180,10 +178,9 @@ impl Clone for Box<dyn CompressionEngine> {
} }
} }
lazy_static! { fn init_compression_engines() -> EnumMap<CompressionType, Box<dyn CompressionEngine>> {
static ref COMPRESSION_ENGINES: EnumMap<CompressionType, Box<dyn CompressionEngine>> = { #[allow(unused_mut)]
#[allow(unused_mut)] // mut needed when gzip/lz4 features are enabled let mut em: EnumMap<CompressionType, Box<dyn CompressionEngine>> = enum_map! {
let mut em = enum_map! {
CompressionType::LZ4 => Box::new(crate::compression_engine::program::CompressionEngineProgram::new( CompressionType::LZ4 => Box::new(crate::compression_engine::program::CompressionEngineProgram::new(
"lz4", "lz4",
vec!["-c"], vec!["-c"],
@@ -234,9 +231,12 @@ lazy_static! {
} }
em em
};
} }
static COMPRESSION_ENGINES: std::sync::LazyLock<
EnumMap<CompressionType, Box<dyn CompressionEngine>>,
> = std::sync::LazyLock::new(init_compression_engines);
pub fn default_compression_type() -> CompressionType { pub fn default_compression_type() -> CompressionType {
CompressionType::LZ4 CompressionType::LZ4
} }

View File

@@ -301,42 +301,48 @@ impl Settings {
config_builder = config_builder.set_override("force", true)?; config_builder = config_builder.set_override("force", true)?;
} }
#[cfg(feature = "server")]
if let Some(server_password) = &args.options.server_password { if let Some(server_password) = &args.options.server_password {
config_builder = config_builder =
config_builder.set_override("server.password", server_password.as_str())?; config_builder.set_override("server.password", server_password.as_str())?;
} }
#[cfg(feature = "server")]
if let Some(server_password_hash) = &args.options.server_password_hash { if let Some(server_password_hash) = &args.options.server_password_hash {
config_builder = config_builder config_builder = config_builder
.set_override("server.password_hash", server_password_hash.as_str())?; .set_override("server.password_hash", server_password_hash.as_str())?;
} }
#[cfg(feature = "server")]
if let Some(server_username) = &args.options.server_username { if let Some(server_username) = &args.options.server_username {
config_builder = config_builder =
config_builder.set_override("server.username", server_username.as_str())?; config_builder.set_override("server.username", server_username.as_str())?;
} }
#[cfg(feature = "server")]
if let Some(server_address) = &args.mode.server_address { if let Some(server_address) = &args.mode.server_address {
config_builder = config_builder =
config_builder.set_override("server.address", server_address.as_str())?; config_builder.set_override("server.address", server_address.as_str())?;
} }
#[cfg(feature = "server")]
if let Some(server_port) = args.mode.server_port { if let Some(server_port) = args.mode.server_port {
config_builder = config_builder.set_override("server.port", server_port)?; config_builder = config_builder.set_override("server.port", server_port)?;
} }
#[cfg(feature = "tls")] #[cfg(feature = "server")]
if let Some(server_cert) = &args.mode.server_cert { if let Some(server_cert) = &args.mode.server_cert {
config_builder = config_builder config_builder = config_builder
.set_override("server.cert_file", server_cert.to_string_lossy().as_ref())?; .set_override("server.cert_file", server_cert.to_string_lossy().as_ref())?;
} }
#[cfg(feature = "tls")] #[cfg(feature = "server")]
if let Some(server_key) = &args.mode.server_key { if let Some(server_key) = &args.mode.server_key {
config_builder = config_builder config_builder = config_builder
.set_override("server.key_file", server_key.to_string_lossy().as_ref())?; .set_override("server.key_file", server_key.to_string_lossy().as_ref())?;
} }
#[cfg(feature = "server")]
if let Some(max_body_size) = args.options.server_max_body_size { if let Some(max_body_size) = args.options.server_max_body_size {
config_builder = config_builder.set_override("server.max_body_size", max_body_size)?; config_builder = config_builder.set_override("server.max_body_size", max_body_size)?;
} }

View File

@@ -1,6 +1,5 @@
use anyhow::{Context, Error, Result, anyhow}; use anyhow::{Context, Error, Result, anyhow};
use chrono::prelude::*; use chrono::prelude::*;
use lazy_static::lazy_static;
use log::*; use log::*;
use rusqlite::{Connection, OpenFlags, Row, params}; use rusqlite::{Connection, OpenFlags, Row, params};
use rusqlite_migration::{M, Migrations}; use rusqlite_migration::{M, Migrations};
@@ -47,25 +46,21 @@ let id = db::insert_item(&conn, item)?;
``` ```
*/ */
lazy_static! { static MIGRATIONS: std::sync::LazyLock<Migrations<'static>> = std::sync::LazyLock::new(|| {
// Database schema migrations for the Keep application. Migrations::new(vec![
//
// Defines the sequence of migrations to create and update the schema.
// Applied automatically when opening a database connection.
static ref MIGRATIONS: Migrations<'static> = Migrations::new(vec![
M::up( M::up(
"CREATE TABLE items( "CREATE TABLE items(
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
ts TEXT NOT NULL, ts TEXT NOT NULL,
size INTEGER NULL, size INTEGER NULL,
compression TEXT NOT NULL)" compression TEXT NOT NULL)",
), ),
M::up( M::up(
"CREATE TABLE tags ( "CREATE TABLE tags (
id INTEGER NOT NULL, id INTEGER NOT NULL,
name TEXT NOT NULL, name TEXT NOT NULL,
FOREIGN KEY(id) REFERENCES items(id) ON DELETE CASCADE, FOREIGN KEY(id) REFERENCES items(id) ON DELETE CASCADE,
PRIMARY KEY(id, name));" PRIMARY KEY(id, name));",
), ),
M::up( M::up(
"CREATE TABLE metas ( "CREATE TABLE metas (
@@ -73,16 +68,17 @@ lazy_static! {
name TEXT NOT NULL, name TEXT NOT NULL,
value TEXT NOT NULL, value TEXT NOT NULL,
FOREIGN KEY(id) REFERENCES items(id) ON DELETE CASCADE, FOREIGN KEY(id) REFERENCES items(id) ON DELETE CASCADE,
PRIMARY KEY(id, name));" PRIMARY KEY(id, name));",
), ),
M::up("CREATE INDEX idx_tags_name ON tags(name)"), M::up("CREATE INDEX idx_tags_name ON tags(name)"),
M::up("CREATE INDEX idx_metas_name ON metas(name)"), M::up("CREATE INDEX idx_metas_name ON metas(name)"),
M::up("CREATE INDEX idx_items_ts ON items(ts)"),
M::up("UPDATE items SET compression = 'raw' WHERE compression = 'none'"), M::up("UPDATE items SET compression = 'raw' WHERE compression = 'none'"),
M::up("ALTER TABLE items RENAME COLUMN size TO uncompressed_size"), M::up("ALTER TABLE items RENAME COLUMN size TO uncompressed_size"),
M::up("ALTER TABLE items ADD COLUMN compressed_size INTEGER NULL"), M::up("ALTER TABLE items ADD COLUMN compressed_size INTEGER NULL"),
M::up("ALTER TABLE items ADD COLUMN closed BOOLEAN NOT NULL DEFAULT 1"), M::up("ALTER TABLE items ADD COLUMN closed BOOLEAN NOT NULL DEFAULT 1"),
]); ])
} });
/// Represents an item stored in the database. /// Represents an item stored in the database.
/// ///
@@ -1355,11 +1351,11 @@ pub fn get_item_meta(conn: &Connection, item: &Item) -> Result<Vec<Meta>> {
/// let db_path = _tmp.path().join("keep.db"); /// let db_path = _tmp.path().join("keep.db");
/// let conn = db::open(db_path)?; /// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), uncompressed_size: None, compressed_size: None, closed: false, compression: "lz4".to_string() }; /// let item = Item { id: Some(1), ts: Utc::now(), uncompressed_size: None, compressed_size: None, closed: false, compression: "lz4".to_string() };
/// let meta = db::get_item_meta_name(&conn, &item, "mime_type".to_string())?; /// let meta = db::get_item_meta_name(&conn, &item, "mime_type")?;
/// # Ok(()) /// # Ok(())
/// # } /// # }
/// ``` /// ```
pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Result<Option<Meta>> { pub fn get_item_meta_name(conn: &Connection, item: &Item, name: &str) -> Result<Option<Meta>> {
debug!("DB: Getting item meta name: {item:?} {name:?}"); debug!("DB: Getting item meta name: {item:?} {name:?}");
let mut statement = conn let mut statement = conn
.prepare_cached("SELECT id, name, value FROM metas WHERE id=?1 AND name=?2") .prepare_cached("SELECT id, name, value FROM metas WHERE id=?1 AND name=?2")
@@ -1407,11 +1403,11 @@ pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Resul
/// let db_path = _tmp.path().join("keep.db"); /// let db_path = _tmp.path().join("keep.db");
/// let conn = db::open(db_path)?; /// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), uncompressed_size: None, compressed_size: None, closed: false, compression: "lz4".to_string() }; /// let item = Item { id: Some(1), ts: Utc::now(), uncompressed_size: None, compressed_size: None, closed: false, compression: "lz4".to_string() };
/// let value = db::get_item_meta_value(&conn, &item, "source".to_string())?; /// let value = db::get_item_meta_value(&conn, &item, "source")?;
/// # Ok(()) /// # Ok(())
/// # } /// # }
/// ``` /// ```
pub fn get_item_meta_value(conn: &Connection, item: &Item, name: String) -> Result<Option<String>> { pub fn get_item_meta_value(conn: &Connection, item: &Item, name: &str) -> Result<Option<String>> {
debug!("DB: Getting item meta value: {item:?} {name:?}"); debug!("DB: Getting item meta value: {item:?} {name:?}");
let mut statement = conn let mut statement = conn
.prepare_cached("SELECT value FROM metas WHERE id=?1 AND name=?2") .prepare_cached("SELECT value FROM metas WHERE id=?1 AND name=?2")

View File

@@ -2,6 +2,7 @@ use std::io::{Read, Result, Write};
use std::str::FromStr; use std::str::FromStr;
use strum::EnumString; use strum::EnumString;
#[cfg(feature = "filter_grep")]
pub mod grep; pub mod grep;
/// Filter plugin module for processing input streams. /// Filter plugin module for processing input streams.
/// ///
@@ -16,7 +17,7 @@ pub mod grep;
/// ``` /// ```
/// # use std::io::{Read, Write}; /// # use std::io::{Read, Write};
/// # use keep::filter_plugin::parse_filter_string; /// # use keep::filter_plugin::parse_filter_string;
/// let mut chain = parse_filter_string("head_lines(10)|grep(pattern=error)")?; /// let mut chain = parse_filter_string("head_lines(10)|tail_lines(5)")?;
/// # let mut reader: &mut dyn Read = &mut std::io::empty(); /// # let mut reader: &mut dyn Read = &mut std::io::empty();
/// # let mut writer: Vec<u8> = Vec::new(); /// # let mut writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut reader, &mut writer)?; /// # chain.filter(&mut reader, &mut writer)?;
@@ -26,12 +27,13 @@ pub mod head;
pub mod skip; pub mod skip;
pub mod strip_ansi; pub mod strip_ansi;
pub mod tail; pub mod tail;
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
pub mod tokens; pub mod tokens;
pub mod utils; pub mod utils;
use std::collections::HashMap; use std::collections::HashMap;
#[cfg(feature = "filter_grep")]
pub use grep::GrepFilter; pub use grep::GrepFilter;
pub use head::{HeadBytesFilter, HeadLinesFilter}; pub use head::{HeadBytesFilter, HeadLinesFilter};
pub use skip::{SkipBytesFilter, SkipLinesFilter}; pub use skip::{SkipBytesFilter, SkipLinesFilter};
@@ -199,13 +201,14 @@ pub enum FilterType {
TailLines, TailLines,
SkipBytes, SkipBytes,
SkipLines, SkipLines,
#[cfg(feature = "filter_grep")]
Grep, Grep,
StripAnsi, StripAnsi,
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
HeadTokens, HeadTokens,
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
SkipTokens, SkipTokens,
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
TailTokens, TailTokens,
} }
@@ -213,6 +216,44 @@ pub enum FilterType {
/// Prevents OOM on large files by rejecting inputs that exceed this limit. /// Prevents OOM on large files by rejecting inputs that exceed this limit.
const MAX_FILTER_BUFFER_SIZE: usize = 256 * 1024 * 1024; const MAX_FILTER_BUFFER_SIZE: usize = 256 * 1024 * 1024;
struct BoundedVecWriter {
data: Vec<u8>,
limit: usize,
}
impl BoundedVecWriter {
fn new(limit: usize) -> Self {
Self {
data: Vec::new(),
limit,
}
}
fn into_inner(self) -> Vec<u8> {
self.data
}
}
impl std::io::Write for BoundedVecWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
if self.data.len() + buf.len() > self.limit {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
format!(
"Input size exceeds maximum filter buffer size ({} bytes)",
MAX_FILTER_BUFFER_SIZE
),
));
}
self.data.write_all(buf)?;
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// A chain of filter plugins applied sequentially. /// A chain of filter plugins applied sequentially.
/// ///
/// Chains multiple filters, applying them in order to the input stream. /// Chains multiple filters, applying them in order to the input stream.
@@ -318,9 +359,8 @@ impl FilterChain {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::{FilterChain, GrepFilter}; /// # use keep::filter_plugin::FilterChain;
/// let mut chain = FilterChain::new(); /// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(GrepFilter::new("error".to_string()).unwrap()));
/// ``` /// ```
pub fn add_plugin(&mut self, plugin: Box<dyn FilterPlugin>) { pub fn add_plugin(&mut self, plugin: Box<dyn FilterPlugin>) {
self.plugins.push(plugin); self.plugins.push(plugin);
@@ -360,21 +400,10 @@ impl FilterChain {
} }
// For multiple plugins, we need to chain them together // For multiple plugins, we need to chain them together
// We'll use a temporary buffer to hold intermediate results // We'll use a bounded buffer to hold intermediate results
let mut current_data = Vec::new(); let mut bounded_writer = BoundedVecWriter::new(MAX_FILTER_BUFFER_SIZE);
std::io::copy(reader, &mut current_data)?; std::io::copy(reader, &mut bounded_writer)?;
let mut current_data = bounded_writer.into_inner();
if current_data.len() > MAX_FILTER_BUFFER_SIZE {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
format!(
"Input size ({} bytes) exceeds maximum filter buffer size ({} bytes). \
Consider using fewer filter plugins or smaller inputs.",
current_data.len(),
MAX_FILTER_BUFFER_SIZE
),
));
}
// Store the plugins length to avoid borrowing issues // Store the plugins length to avoid borrowing issues
let plugins_len = self.plugins.len(); let plugins_len = self.plugins.len();
@@ -508,6 +537,7 @@ fn create_filter_with_options(
// Get the default options for this filter type by creating a temporary instance // Get the default options for this filter type by creating a temporary instance
// To do this, we need to create a default instance of the appropriate filter // To do this, we need to create a default instance of the appropriate filter
let option_defs = match filter_type { let option_defs = match filter_type {
#[cfg(feature = "filter_grep")]
FilterType::Grep => grep::GrepFilter::new("".to_string())?.options(), FilterType::Grep => grep::GrepFilter::new("".to_string())?.options(),
FilterType::HeadBytes => head::HeadBytesFilter::new(0).options(), FilterType::HeadBytes => head::HeadBytesFilter::new(0).options(),
FilterType::HeadLines => head::HeadLinesFilter::new(0).options(), FilterType::HeadLines => head::HeadLinesFilter::new(0).options(),
@@ -516,11 +546,11 @@ fn create_filter_with_options(
FilterType::SkipBytes => skip::SkipBytesFilter::new(0).options(), FilterType::SkipBytes => skip::SkipBytesFilter::new(0).options(),
FilterType::SkipLines => skip::SkipLinesFilter::new(0).options(), FilterType::SkipLines => skip::SkipLinesFilter::new(0).options(),
FilterType::StripAnsi => strip_ansi::StripAnsiFilter::new().options(), FilterType::StripAnsi => strip_ansi::StripAnsiFilter::new().options(),
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
FilterType::HeadTokens => tokens::HeadTokensFilter::new(0).options(), FilterType::HeadTokens => tokens::HeadTokensFilter::new(0).options(),
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
FilterType::SkipTokens => tokens::SkipTokensFilter::new(0).options(), FilterType::SkipTokens => tokens::SkipTokensFilter::new(0).options(),
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
FilterType::TailTokens => tokens::TailTokensFilter::new(0).options(), FilterType::TailTokens => tokens::TailTokensFilter::new(0).options(),
}; };
@@ -590,6 +620,7 @@ fn create_specific_filter(
options: &HashMap<String, serde_json::Value>, options: &HashMap<String, serde_json::Value>,
) -> Result<Box<dyn FilterPlugin>> { ) -> Result<Box<dyn FilterPlugin>> {
match filter_type { match filter_type {
#[cfg(feature = "filter_grep")]
FilterType::Grep => { FilterType::Grep => {
let pattern = options let pattern = options
.get("pattern") .get("pattern")
@@ -690,7 +721,7 @@ fn create_specific_filter(
} }
Ok(Box::new(strip_ansi::StripAnsiFilter::new())) Ok(Box::new(strip_ansi::StripAnsiFilter::new()))
} }
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
FilterType::HeadTokens => { FilterType::HeadTokens => {
let count = options let count = options
.get("count") .get("count")
@@ -708,7 +739,7 @@ fn create_specific_filter(
f.encoding = encoding; f.encoding = encoding;
Ok(Box::new(f)) Ok(Box::new(f))
} }
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
FilterType::SkipTokens => { FilterType::SkipTokens => {
let count = options let count = options
.get("count") .get("count")
@@ -726,7 +757,7 @@ fn create_specific_filter(
f.encoding = encoding; f.encoding = encoding;
Ok(Box::new(f)) Ok(Box::new(f))
} }
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
FilterType::TailTokens => { FilterType::TailTokens => {
let count = options let count = options
.get("count") .get("count")
@@ -747,7 +778,7 @@ fn create_specific_filter(
} }
} }
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
fn parse_encoding_option( fn parse_encoding_option(
options: &std::collections::HashMap<String, serde_json::Value>, options: &std::collections::HashMap<String, serde_json::Value>,
) -> (crate::tokenizer::TokenEncoding, crate::tokenizer::Tokenizer) { ) -> (crate::tokenizer::TokenEncoding, crate::tokenizer::Tokenizer) {

View File

@@ -45,19 +45,23 @@ pub mod services;
#[cfg(feature = "client")] #[cfg(feature = "client")]
pub mod client; pub mod client;
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
pub mod tokenizer; pub mod tokenizer;
// Re-export Args struct for library usage // Re-export Args struct for library usage
pub use args::Args; pub use args::Args;
// Re-export PIPESIZE constant // Re-export PIPESIZE constant
pub use common::PIPESIZE; pub use common::PIPESIZE;
pub use services::CoreError;
// Import all filter plugins to ensure they register themselves // Import all filter plugins to ensure they register themselves
#[allow(unused_imports)] #[allow(unused_imports)]
use filter_plugin::{grep, head, skip, strip_ansi, tail}; #[cfg(feature = "filter_grep")]
use filter_plugin::grep;
#[allow(unused_imports)]
use filter_plugin::{head, skip, strip_ansi, tail};
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
#[allow(unused_imports)] #[allow(unused_imports)]
use filter_plugin::tokens as token_filters; use filter_plugin::tokens as token_filters;
@@ -65,19 +69,19 @@ use crate::meta_plugin::{
cwd, digest, env, exec, hostname, keep_pid, read_rate, read_time, shell, shell_pid, user, cwd, digest, env, exec, hostname, keep_pid, read_rate, read_time, shell, shell_pid, user,
}; };
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
#[allow(unused_imports)] #[allow(unused_imports)]
use crate::meta_plugin::magic_file; use crate::meta_plugin::magic_file;
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
#[allow(unused_imports)] #[allow(unused_imports)]
use crate::meta_plugin::tokens; use crate::meta_plugin::tokens;
#[cfg(feature = "infer")] #[cfg(feature = "meta_infer")]
#[allow(unused_imports)] #[allow(unused_imports)]
use crate::meta_plugin::infer_plugin; use crate::meta_plugin::infer_plugin;
#[cfg(feature = "tree_magic_mini")] #[cfg(feature = "meta_tree_magic_mini")]
#[allow(unused_imports)] #[allow(unused_imports)]
use crate::meta_plugin::tree_magic_mini; use crate::meta_plugin::tree_magic_mini;

View File

@@ -122,6 +122,7 @@ fn main() -> Result<(), Error> {
Import, Import,
Status, Status,
StatusPlugins, StatusPlugins,
#[cfg(feature = "server")]
Server, Server,
GenerateConfig, GenerateConfig,
} }
@@ -150,9 +151,14 @@ fn main() -> Result<(), Error> {
mode = KeepModes::Status; mode = KeepModes::Status;
} else if args.mode.status_plugins { } else if args.mode.status_plugins {
mode = KeepModes::StatusPlugins; mode = KeepModes::StatusPlugins;
} else if args.mode.server { }
#[cfg(feature = "server")]
{
if args.mode.server {
mode = KeepModes::Server; mode = KeepModes::Server;
} else if args.mode.generate_config { }
}
if args.mode.generate_config {
mode = KeepModes::GenerateConfig; mode = KeepModes::GenerateConfig;
} }
@@ -188,6 +194,7 @@ fn main() -> Result<(), Error> {
} }
// Validate server password usage // Validate server password usage
#[cfg(feature = "server")]
if settings.server_password().is_some() && mode != KeepModes::Server { if settings.server_password().is_some() && mode != KeepModes::Server {
cmd.error( cmd.error(
ErrorKind::InvalidValue, ErrorKind::InvalidValue,
@@ -355,19 +362,8 @@ fn main() -> Result<(), Error> {
KeepModes::StatusPlugins => { KeepModes::StatusPlugins => {
modes::status_plugins::mode_status_plugins(&mut cmd, &settings, data_path, db_path) modes::status_plugins::mode_status_plugins(&mut cmd, &settings, data_path, db_path)
} }
KeepModes::Server => {
#[cfg(feature = "server")] #[cfg(feature = "server")]
{ KeepModes::Server => modes::server::mode_server(&mut cmd, &settings, &mut conn, data_path),
modes::server::mode_server(&mut cmd, &settings, &mut conn, data_path)
}
#[cfg(not(feature = "server"))]
{
cmd.error(
ErrorKind::MissingRequiredArgument,
"This binary was not compiled with server support. Recompile with --features server"
).exit();
}
}
KeepModes::GenerateConfig => { KeepModes::GenerateConfig => {
modes::generate_config::mode_generate_config(&mut cmd, &settings) modes::generate_config::mode_generate_config(&mut cmd, &settings)
} }

View File

@@ -1,6 +1,6 @@
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
use magic::{Cookie, CookieFlags}; use magic::{Cookie, CookieFlags};
#[cfg(not(feature = "magic"))] #[cfg(not(feature = "meta_magic"))]
use std::process::{Command, Stdio}; use std::process::{Command, Stdio};
use std::io::{self, Write}; use std::io::{self, Write};
@@ -16,12 +16,12 @@ use crate::meta_plugin::{
// separate cookies can be used from different threads concurrently without // separate cookies can be used from different threads concurrently without
// synchronization. Using thread_local! avoids unsafe impl Send since the // synchronization. Using thread_local! avoids unsafe impl Send since the
// storage is inherently !Send. // storage is inherently !Send.
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
thread_local! { thread_local! {
static MAGIC_COOKIE: std::cell::RefCell<Option<Cookie>> = const { std::cell::RefCell::new(None) }; static MAGIC_COOKIE: std::cell::RefCell<Option<Cookie>> = const { std::cell::RefCell::new(None) };
} }
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
#[derive(Debug)] #[derive(Debug)]
pub struct MagicFileMetaPluginImpl { pub struct MagicFileMetaPluginImpl {
buffer: Vec<u8>, buffer: Vec<u8>,
@@ -30,7 +30,7 @@ pub struct MagicFileMetaPluginImpl {
base: BaseMetaPlugin, base: BaseMetaPlugin,
} }
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
impl MagicFileMetaPluginImpl { impl MagicFileMetaPluginImpl {
pub fn new( pub fn new(
options: Option<std::collections::HashMap<String, serde_yaml::Value>>, options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
@@ -113,7 +113,7 @@ impl MagicFileMetaPluginImpl {
} }
} }
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
impl MetaPlugin for MagicFileMetaPluginImpl { impl MetaPlugin for MagicFileMetaPluginImpl {
fn is_finalized(&self) -> bool { fn is_finalized(&self) -> bool {
self.is_finalized self.is_finalized
@@ -222,10 +222,10 @@ impl MetaPlugin for MagicFileMetaPluginImpl {
} }
} }
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
pub use MagicFileMetaPluginImpl as MagicFileMetaPlugin; pub use MagicFileMetaPluginImpl as MagicFileMetaPlugin;
#[cfg(not(feature = "magic"))] #[cfg(not(feature = "meta_magic"))]
#[derive(Debug)] #[derive(Debug)]
pub struct FallbackMagicFileMetaPlugin { pub struct FallbackMagicFileMetaPlugin {
buffer: Vec<u8>, buffer: Vec<u8>,
@@ -234,7 +234,7 @@ pub struct FallbackMagicFileMetaPlugin {
base: BaseMetaPlugin, base: BaseMetaPlugin,
} }
#[cfg(not(feature = "magic"))] #[cfg(not(feature = "meta_magic"))]
impl FallbackMagicFileMetaPlugin { impl FallbackMagicFileMetaPlugin {
pub fn new( pub fn new(
options: Option<std::collections::HashMap<String, serde_yaml::Value>>, options: Option<std::collections::HashMap<String, serde_yaml::Value>>,
@@ -336,7 +336,7 @@ impl FallbackMagicFileMetaPlugin {
} }
} }
#[cfg(not(feature = "magic"))] #[cfg(not(feature = "meta_magic"))]
impl MetaPlugin for FallbackMagicFileMetaPlugin { impl MetaPlugin for FallbackMagicFileMetaPlugin {
fn is_finalized(&self) -> bool { fn is_finalized(&self) -> bool {
self.is_finalized self.is_finalized
@@ -441,7 +441,7 @@ impl MetaPlugin for FallbackMagicFileMetaPlugin {
} }
} }
#[cfg(not(feature = "magic"))] #[cfg(not(feature = "meta_magic"))]
pub use FallbackMagicFileMetaPlugin as MagicFileMetaPlugin; pub use FallbackMagicFileMetaPlugin as MagicFileMetaPlugin;
use crate::meta_plugin::register_meta_plugin; use crate::meta_plugin::register_meta_plugin;

View File

@@ -1,5 +1,4 @@
use log::{debug, warn}; use log::{debug, warn};
use once_cell::sync::Lazy;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashMap; use std::collections::HashMap;
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
@@ -9,7 +8,7 @@ pub mod digest;
pub mod env; pub mod env;
pub mod exec; pub mod exec;
pub mod hostname; pub mod hostname;
#[cfg(feature = "infer")] #[cfg(feature = "meta_infer")]
pub mod infer_plugin; pub mod infer_plugin;
pub mod keep_pid; pub mod keep_pid;
pub mod magic_file; pub mod magic_file;
@@ -18,32 +17,32 @@ pub mod read_time;
pub mod shell; pub mod shell;
pub mod shell_pid; pub mod shell_pid;
pub mod text; pub mod text;
#[cfg(feature = "tokens")] #[cfg(feature = "meta_tokens")]
pub mod tokens; pub mod tokens;
#[cfg(feature = "tree_magic_mini")] #[cfg(feature = "meta_tree_magic_mini")]
pub mod tree_magic_mini; pub mod tree_magic_mini;
pub mod user; pub mod user;
pub use digest::DigestMetaPlugin; pub use digest::DigestMetaPlugin;
pub use exec::MetaPluginExec; pub use exec::MetaPluginExec;
#[cfg(feature = "magic")] #[cfg(feature = "meta_magic")]
pub use magic_file::MagicFileMetaPlugin; pub use magic_file::MagicFileMetaPlugin;
// pub use text::TextMetaPlugin; // Removed duplicate // pub use text::TextMetaPlugin; // Removed duplicate
pub use cwd::CwdMetaPlugin; pub use cwd::CwdMetaPlugin;
pub use env::EnvMetaPlugin; pub use env::EnvMetaPlugin;
pub use hostname::HostnameMetaPlugin; pub use hostname::HostnameMetaPlugin;
#[cfg(feature = "infer")] #[cfg(feature = "meta_infer")]
pub use infer_plugin::InferMetaPlugin; pub use infer_plugin::InferMetaPlugin;
pub use keep_pid::KeepPidMetaPlugin; pub use keep_pid::KeepPidMetaPlugin;
pub use read_rate::ReadRateMetaPlugin; pub use read_rate::ReadRateMetaPlugin;
pub use read_time::ReadTimeMetaPlugin; pub use read_time::ReadTimeMetaPlugin;
pub use shell::ShellMetaPlugin; pub use shell::ShellMetaPlugin;
pub use shell_pid::ShellPidMetaPlugin; pub use shell_pid::ShellPidMetaPlugin;
#[cfg(feature = "tree_magic_mini")] #[cfg(feature = "meta_tree_magic_mini")]
pub use tree_magic_mini::TreeMagicMiniMetaPlugin; pub use tree_magic_mini::TreeMagicMiniMetaPlugin;
pub use user::UserMetaPlugin; pub use user::UserMetaPlugin;
#[cfg(not(feature = "magic"))] #[cfg(not(feature = "meta_magic"))]
pub use magic_file::FallbackMagicFileMetaPlugin as MagicFileMetaPlugin; pub use magic_file::FallbackMagicFileMetaPlugin as MagicFileMetaPlugin;
type PluginConstructor = fn( type PluginConstructor = fn(
@@ -444,9 +443,9 @@ where
/// ///
/// An empty `HashMap` (default implementation). /// An empty `HashMap` (default implementation).
fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> { fn outputs(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
use once_cell::sync::Lazy; use std::sync::LazyLock;
static EMPTY: Lazy<std::collections::HashMap<String, serde_yaml::Value>> = static EMPTY: LazyLock<std::collections::HashMap<String, serde_yaml::Value>> =
Lazy::new(std::collections::HashMap::new); LazyLock::new(std::collections::HashMap::new);
&EMPTY &EMPTY
} }
@@ -471,9 +470,9 @@ where
/// ///
/// An empty `HashMap` (default implementation). /// An empty `HashMap` (default implementation).
fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> { fn options(&self) -> &std::collections::HashMap<String, serde_yaml::Value> {
use once_cell::sync::Lazy; use std::sync::LazyLock;
static EMPTY: Lazy<std::collections::HashMap<String, serde_yaml::Value>> = static EMPTY: LazyLock<std::collections::HashMap<String, serde_yaml::Value>> =
Lazy::new(std::collections::HashMap::new); LazyLock::new(std::collections::HashMap::new);
&EMPTY &EMPTY
} }
@@ -602,8 +601,9 @@ where
} }
/// Global registry for meta plugins. /// Global registry for meta plugins.
static META_PLUGIN_REGISTRY: Lazy<Mutex<HashMap<MetaPluginType, PluginConstructor>>> = static META_PLUGIN_REGISTRY: std::sync::LazyLock<
Lazy::new(|| Mutex::new(HashMap::new())); Mutex<HashMap<MetaPluginType, PluginConstructor>>,
> = std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
/// Register a meta plugin with the global registry. /// Register a meta plugin with the global registry.
/// ///

View File

@@ -39,7 +39,7 @@ pub fn mode(
// Decompress through streaming readers // Decompress through streaming readers
let mut decompressed_reader: Box<dyn Read> = let mut decompressed_reader: Box<dyn Read> =
CompressionService::decompressing_reader(reader, &compression_type); CompressionService::decompressing_reader(reader, &compression_type)?;
// Binary detection: sample first chunk // Binary detection: sample first chunk
let mut sample_buf = [0u8; crate::common::PIPESIZE]; let mut sample_buf = [0u8; crate::common::PIPESIZE];

View File

@@ -29,7 +29,7 @@ pub fn mode(
} else { } else {
cmd.error( cmd.error(
clap::error::ErrorKind::InvalidValue, clap::error::ErrorKind::InvalidValue,
format!("Unsupported import format: {import_path} (expected .keep.tar or .meta.yml)"), format!("Unsupported import format: {}", import_path),
) )
.exit(); .exit();
} }

View File

@@ -1,8 +1,9 @@
use crate::client::{ItemInfo, KeepClient}; use crate::client::KeepClient;
use crate::compression_engine::CompressionType; use crate::compression_engine::CompressionType;
use crate::config::Settings; use crate::config::Settings;
use crate::meta_plugin::SaveMetaFn; use crate::meta_plugin::SaveMetaFn;
use crate::modes::common::settings_compression_type; use crate::modes::common::settings_compression_type;
use crate::services::ItemInfo;
use crate::services::compression_service::CompressionService; use crate::services::compression_service::CompressionService;
use crate::services::meta_service::MetaService; use crate::services::meta_service::MetaService;
use anyhow::Result; use anyhow::Result;
@@ -75,7 +76,7 @@ pub fn mode(
// Wrap pipe writer with appropriate compression // Wrap pipe writer with appropriate compression
let mut compressor: Box<dyn Write> = let mut compressor: Box<dyn Write> =
CompressionService::compressing_writer(Box::new(pipe_writer), &compression_type_clone); CompressionService::compressing_writer(Box::new(pipe_writer), &compression_type_clone)?;
loop { loop {
let n = stdin_lock.read(&mut buffer)?; let n = stdin_lock.read(&mut buffer)?;

View File

@@ -21,9 +21,7 @@ use chrono::{DateTime, Utc};
use clap::Command; use clap::Command;
use clap::error::ErrorKind; use clap::error::ErrorKind;
use comfy_table::{Attribute, Cell, ContentArrangement, Table}; use comfy_table::{Attribute, Cell, ContentArrangement, Table};
use lazy_static::lazy_static;
use log::debug; use log::debug;
use regex::Regex;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashMap; use std::collections::HashMap;
use std::env; use std::env;
@@ -57,21 +55,18 @@ pub enum OutputFormat {
Yaml, Yaml,
} }
lazy_static! { pub const IMPORT_FORMAT_ERROR: &str =
static ref KEEP_META_RE: Regex = Regex::new(r"^KEEP_META_(.+)$").unwrap(); "Unsupported import format: {} (expected .keep.tar or .meta.yml)";
}
pub fn get_meta_from_env() -> HashMap<String, String> { pub fn get_meta_from_env() -> HashMap<String, String> {
debug!("COMMON: Getting meta from KEEP_META_*"); debug!("COMMON: Getting meta from KEEP_META_*");
let mut meta_env: HashMap<String, String> = HashMap::new(); let mut meta_env: HashMap<String, String> = HashMap::new();
const PREFIX: &str = "KEEP_META_";
for (key, value) in env::vars() { for (key, value) in env::vars() {
if let Some(meta_name_caps) = KEEP_META_RE.captures(key.as_str()) { if let Some(name) = key.strip_prefix(PREFIX) {
let name = meta_name_caps.get(1).map(|m| m.as_str().to_string()); if !name.is_empty() && name != "PLUGINS" {
if let Some(name) = name {
if name != "PLUGINS" {
debug!("COMMON: Found meta: {}={}", name, value); debug!("COMMON: Found meta: {}={}", name, value);
meta_env.insert(name, value); meta_env.insert(name.to_string(), value);
}
} }
} }
} }
@@ -640,6 +635,7 @@ pub struct ImportMeta {
/// ///
/// Returns the first ID if provided, the newest item matching tags, /// Returns the first ID if provided, the newest item matching tags,
/// or the newest item overall if neither is specified. /// or the newest item overall if neither is specified.
#[cfg(feature = "client")]
pub fn resolve_item_id( pub fn resolve_item_id(
client: &crate::client::KeepClient, client: &crate::client::KeepClient,
ids: &[i64], ids: &[i64],
@@ -663,6 +659,7 @@ pub fn resolve_item_id(
} }
/// Resolve item IDs from explicit IDs or tags (multi-item variant). /// Resolve item IDs from explicit IDs or tags (multi-item variant).
#[cfg(feature = "client")]
pub fn resolve_item_ids( pub fn resolve_item_ids(
client: &crate::client::KeepClient, client: &crate::client::KeepClient,
ids: &[i64], ids: &[i64],

View File

@@ -52,7 +52,7 @@ pub fn mode_import(
} else { } else {
cmd.error( cmd.error(
clap::error::ErrorKind::InvalidValue, clap::error::ErrorKind::InvalidValue,
format!("Unsupported import format: {import_path} (expected .keep.tar or .meta.yml)"), format!("Unsupported import format: {}", import_path),
) )
.exit(); .exit();
} }

View File

@@ -151,13 +151,19 @@ pub fn mode_list(
Some(size) => format_size(size as u64, settings.human_readable), Some(size) => format_size(size as u64, settings.human_readable),
None => match item_path.metadata() { None => match item_path.metadata() {
Ok(_) => "Unknown".to_string(), Ok(_) => "Unknown".to_string(),
Err(_) => "Missing".to_string(), Err(e) => {
log::warn!("File missing or inaccessible: {}", e);
"Missing".to_string()
}
}, },
}, },
ColumnType::Compression => item.compression.to_string(), ColumnType::Compression => item.compression.to_string(),
ColumnType::FileSize => match item_path.metadata() { ColumnType::FileSize => match item_path.metadata() {
Ok(metadata) => format_size(metadata.len(), settings.human_readable), Ok(metadata) => format_size(metadata.len(), settings.human_readable),
Err(_) => "Missing".to_string(), Err(e) => {
log::warn!("File missing or inaccessible: {}", e);
"Missing".to_string()
}
}, },
ColumnType::FilePath => item_path ColumnType::FilePath => item_path
.clone() .clone()

View File

@@ -178,6 +178,7 @@ pub async fn handle_list_items(
let item_infos: Vec<ItemInfo> = items_with_meta let item_infos: Vec<ItemInfo> = items_with_meta
.into_iter() .into_iter()
.filter_map(|iwm| ItemInfo::try_from(iwm).ok()) .filter_map(|iwm| ItemInfo::try_from(iwm).ok())
.map(|info| info.with_file_size(&state.data_dir))
.collect(); .collect();
ResponseBuilder::json(ApiResponse::ok(item_infos)) ResponseBuilder::json(ApiResponse::ok(item_infos))
@@ -227,7 +228,7 @@ async fn handle_as_meta_response(
/// offset/length applied at the stream level — never loads the full item into memory. /// offset/length applied at the stream level — never loads the full item into memory.
async fn handle_as_meta_response_with_metadata( async fn handle_as_meta_response_with_metadata(
db: &Arc<tokio::sync::Mutex<rusqlite::Connection>>, db: &Arc<tokio::sync::Mutex<rusqlite::Connection>>,
data_dir: &std::path::Path, _data_dir: &std::path::Path,
item_service: &Arc<ItemService>, item_service: &Arc<ItemService>,
item_id: i64, item_id: i64,
metadata: &HashMap<String, String>, metadata: &HashMap<String, String>,
@@ -339,6 +340,7 @@ pub async fn handle_post_item(
let db = state.db.clone(); let db = state.db.clone();
let item_service = state.item_service.clone(); let item_service = state.item_service.clone();
let settings = state.settings.clone(); let settings = state.settings.clone();
let data_dir = state.data_dir.clone();
// Parse tags from query parameter // Parse tags from query parameter
let tags: Vec<String> = params let tags: Vec<String> = params
@@ -472,7 +474,9 @@ pub async fn handle_post_item(
return Err(StatusCode::PAYLOAD_TOO_LARGE); return Err(StatusCode::PAYLOAD_TOO_LARGE);
} }
let item_info = ItemInfo::try_from(item_with_meta).map_err(|e| { let item_info = ItemInfo::try_from(item_with_meta)
.map(|info| info.with_file_size(&data_dir))
.map_err(|e| {
warn!("Item conversion failed: {e}"); warn!("Item conversion failed: {e}");
StatusCode::INTERNAL_SERVER_ERROR StatusCode::INTERNAL_SERVER_ERROR
})?; })?;
@@ -1092,6 +1096,7 @@ pub async fn handle_delete_item(
compression: deleted_item.compression, compression: deleted_item.compression,
tags: vec![], tags: vec![],
metadata: HashMap::new(), metadata: HashMap::new(),
file_size: None,
}; };
Ok(Json(ApiResponse::ok(item_info))) Ok(Json(ApiResponse::ok(item_info)))
@@ -1124,6 +1129,7 @@ pub async fn handle_get_item_info(
let db = state.db.clone(); let db = state.db.clone();
let item_service = state.item_service.clone(); let item_service = state.item_service.clone();
let data_dir = state.data_dir.clone();
let item_with_meta = task::spawn_blocking(move || { let item_with_meta = task::spawn_blocking(move || {
let conn = db.blocking_lock(); let conn = db.blocking_lock();
@@ -1136,7 +1142,9 @@ pub async fn handle_get_item_info(
})? })?
.map_err(handle_item_error)?; .map_err(handle_item_error)?;
let item_info = ItemInfo::try_from(item_with_meta).map_err(|e| { let item_info = ItemInfo::try_from(item_with_meta)
.map(|info| info.with_file_size(&data_dir))
.map_err(|e| {
warn!("Item conversion failed: {e}"); warn!("Item conversion failed: {e}");
StatusCode::INTERNAL_SERVER_ERROR StatusCode::INTERNAL_SERVER_ERROR
})?; })?;
@@ -1352,6 +1360,7 @@ pub async fn handle_update_item(
let db = state.db.clone(); let db = state.db.clone();
let item_service = state.item_service.clone(); let item_service = state.item_service.clone();
let settings = state.settings.clone(); let settings = state.settings.clone();
let data_dir = state.data_dir.clone();
let size_param = params.uncompressed_size; let size_param = params.uncompressed_size;
let item_info = task::spawn_blocking(move || { let item_info = task::spawn_blocking(move || {
@@ -1369,7 +1378,9 @@ pub async fn handle_update_item(
return Err(StatusCode::INTERNAL_SERVER_ERROR); return Err(StatusCode::INTERNAL_SERVER_ERROR);
} }
match item_service.get_item(&conn, item_id) { match item_service.get_item(&conn, item_id) {
Ok(iwm) => ItemInfo::try_from(iwm).map_err(|e| { Ok(iwm) => ItemInfo::try_from(iwm)
.map(|info| info.with_file_size(&data_dir))
.map_err(|e| {
warn!("Item conversion failed: {e}"); warn!("Item conversion failed: {e}");
StatusCode::INTERNAL_SERVER_ERROR StatusCode::INTERNAL_SERVER_ERROR
}), }),
@@ -1392,7 +1403,9 @@ pub async fn handle_update_item(
); );
match result { match result {
Ok(item_with_meta) => ItemInfo::try_from(item_with_meta).map_err(|e| { Ok(item_with_meta) => ItemInfo::try_from(item_with_meta)
.map(|info| info.with_file_size(&data_dir))
.map_err(|e| {
warn!("Item conversion failed: {e}"); warn!("Item conversion failed: {e}");
StatusCode::INTERNAL_SERVER_ERROR StatusCode::INTERNAL_SERVER_ERROR
}), }),
@@ -1451,7 +1464,7 @@ pub async fn handle_export_items(
let db = state.db.clone(); let db = state.db.clone();
let item_service = state.item_service.clone(); let item_service = state.item_service.clone();
let data_dir = state.data_dir.clone(); let _data_dir = state.data_dir.clone();
// Resolve items in blocking task // Resolve items in blocking task
let items_with_meta = task::spawn_blocking(move || { let items_with_meta = task::spawn_blocking(move || {
@@ -1520,6 +1533,7 @@ pub async fn handle_export_items(
} }
} }
let tx_err = tx.clone();
let writer = ChannelWriter { tx }; let writer = ChannelWriter { tx };
if let Err(e) = crate::export_tar::write_export_tar( if let Err(e) = crate::export_tar::write_export_tar(
@@ -1532,7 +1546,7 @@ pub async fn handle_export_items(
&conn, &conn,
) { ) {
warn!("Export tar write failed: {e}"); warn!("Export tar write failed: {e}");
let _ = tx.blocking_send(Err(std::io::Error::other(format!("Export failed: {e}")))); let _ = tx_err.blocking_send(Err(std::io::Error::other(format!("Export failed: {e}"))));
} }
// Channel drops here, signaling EOF to the stream // Channel drops here, signaling EOF to the stream
}); });
@@ -1602,11 +1616,7 @@ pub async fn handle_import_items(
.map_err(|e| { .map_err(|e| {
warn!("Failed to import tar: {e}"); warn!("Failed to import tar: {e}");
match &e { match &e {
crate::services::error::CoreError::Io(io_err) crate::services::error::CoreError::PayloadTooLarge => StatusCode::PAYLOAD_TOO_LARGE,
if io_err.to_string() == "Payload too large" =>
{
StatusCode::PAYLOAD_TOO_LARGE
}
_ => StatusCode::INTERNAL_SERVER_ERROR, _ => StatusCode::INTERNAL_SERVER_ERROR,
} }
})?; })?;

View File

@@ -366,10 +366,13 @@ pub struct StatusInfoResponse {
/// let item_info = ItemInfo { /// let item_info = ItemInfo {
/// id: 42, /// id: 42,
/// ts: "2023-12-01T15:30:45Z".to_string(), /// ts: "2023-12-01T15:30:45Z".to_string(),
/// size: Some(1024), /// uncompressed_size: Some(1024),
/// compressed_size: Some(512),
/// closed: true,
/// compression: "gzip".to_string(), /// compression: "gzip".to_string(),
/// tags: vec!["important".to_string()], /// tags: vec!["important".to_string()],
/// metadata: HashMap::from([("mime_type".to_string(), "text/plain".to_string())]), /// metadata: HashMap::from([("mime_type".to_string(), "text/plain".to_string())]),
/// file_size: Some(512),
/// }; /// };
/// ``` /// ```
#[derive(Serialize, Deserialize, ToSchema)] #[derive(Serialize, Deserialize, ToSchema)]
@@ -413,12 +416,41 @@ pub struct ItemInfo {
/// Key-value pairs containing additional metadata about the item. /// Key-value pairs containing additional metadata about the item.
#[schema(example = json!({"mime_type": "text/plain", "mime_encoding": "utf-8", "line_count": "42"}))] #[schema(example = json!({"mime_type": "text/plain", "mime_encoding": "utf-8", "line_count": "42"}))]
pub metadata: HashMap<String, String>, pub metadata: HashMap<String, String>,
/// Actual file size in bytes.
///
/// The filesystem-reported size of the item's data file. This may differ from
/// `compressed_size` if the file was written and the database hasn't been updated.
/// None if the file cannot be read (e.g., file not found, permission denied).
#[schema(example = 512)]
pub file_size: Option<i64>,
}
impl ItemInfo {
/// Enriches this `ItemInfo` with the actual filesystem-reported size.
///
/// Reads the size of the item's data file from disk and sets `file_size`.
/// If the file cannot be read, `file_size` is left as None.
///
/// # Arguments
///
/// * `data_dir` - The data directory path containing item files.
///
/// # Returns
///
/// A new `ItemInfo` with `file_size` populated from the filesystem.
pub fn with_file_size(mut self, data_dir: &std::path::Path) -> Self {
let item_path = data_dir.join(self.id.to_string());
self.file_size = std::fs::metadata(&item_path).map(|m| m.len() as i64).ok();
self
}
} }
impl TryFrom<ItemWithMeta> for ItemInfo { impl TryFrom<ItemWithMeta> for ItemInfo {
type Error = anyhow::Error; type Error = anyhow::Error;
fn try_from(item_with_meta: ItemWithMeta) -> Result<Self, Self::Error> { fn try_from(item_with_meta: ItemWithMeta) -> Result<Self, Self::Error> {
let tags = item_with_meta.tag_names();
let metadata = item_with_meta.meta_as_map();
Ok(ItemInfo { Ok(ItemInfo {
id: item_with_meta id: item_with_meta
.item .item
@@ -429,8 +461,9 @@ impl TryFrom<ItemWithMeta> for ItemInfo {
compressed_size: item_with_meta.item.compressed_size, compressed_size: item_with_meta.item.compressed_size,
closed: item_with_meta.item.closed, closed: item_with_meta.item.closed,
compression: item_with_meta.item.compression, compression: item_with_meta.item.compression,
tags: item_with_meta.tag_names(), tags,
metadata: item_with_meta.meta_as_map(), metadata,
file_size: None,
}) })
} }
} }
@@ -499,6 +532,7 @@ pub struct TagsQuery {
/// ```rust /// ```rust
/// use keep::modes::server::common::ListItemsQuery; /// use keep::modes::server::common::ListItemsQuery;
/// let query = ListItemsQuery { /// let query = ListItemsQuery {
/// ids: None,
/// tags: Some("important".to_string()), /// tags: Some("important".to_string()),
/// order: Some("newest".to_string()), /// order: Some("newest".to_string()),
/// start: Some(0), /// start: Some(0),

View File

@@ -179,24 +179,18 @@ async fn run_server(
let addr: SocketAddr = bind_address.parse()?; let addr: SocketAddr = bind_address.parse()?;
// Warn if authentication is enabled without TLS // Warn if authentication is enabled without TLS
if config.password.is_some() || config.password_hash.is_some() || config.jwt_secret.is_some() { if (config.password.is_some() || config.password_hash.is_some() || config.jwt_secret.is_some())
#[cfg(not(feature = "tls"))] && (config.cert_file.is_none() || config.key_file.is_none())
log::warn!( {
"SECURITY: Authentication enabled but TLS support is not compiled in. Credentials will be transmitted in plain text!"
);
#[cfg(feature = "tls")]
if config.cert_file.is_none() || config.key_file.is_none() {
log::warn!( log::warn!(
"SECURITY: Authentication enabled but TLS is not configured. Credentials will be transmitted in plain text!" "SECURITY: Authentication enabled but TLS is not configured. Credentials will be transmitted in plain text!"
); );
} }
}
// Build the app into a service // Build the app into a service
let service = app.into_make_service_with_connect_info::<SocketAddr>(); let service = app.into_make_service_with_connect_info::<SocketAddr>();
// Use TLS if both cert and key files are provided // Use TLS if both cert and key files are provided
#[cfg(feature = "tls")]
if let (Some(cert_file), Some(key_file)) = (&config.cert_file, &config.key_file) { if let (Some(cert_file), Some(key_file)) = (&config.cert_file, &config.key_file) {
info!("SERVER: HTTPS server listening on {addr}"); info!("SERVER: HTTPS server listening on {addr}");

View File

@@ -13,11 +13,13 @@ use serde::Deserialize;
use std::collections::HashMap; use std::collections::HashMap;
/// Escape text content for safe HTML insertion. /// Escape text content for safe HTML insertion.
#[inline]
fn esc(s: &str) -> String { fn esc(s: &str) -> String {
encode_text(s).to_string() encode_text(s).to_string()
} }
/// Escape attribute values for safe HTML attribute insertion. /// Escape attribute values for safe HTML attribute insertion.
#[inline]
fn esc_attr(s: &str) -> String { fn esc_attr(s: &str) -> String {
encode_double_quoted_attribute(s).to_string() encode_double_quoted_attribute(s).to_string()
} }

View File

@@ -112,38 +112,27 @@ impl CompressionService {
Ok(reader) Ok(reader)
} }
/// Creates a decompressing reader wrapping the given reader.
///
/// Returns a boxed reader that decompresses on the fly based on the compression type.
/// Useful for decompressing network streams or other non-file sources.
///
/// # Arguments
///
/// * `reader` - The underlying compressed reader.
/// * `compression` - Compression type string (e.g., "gzip", "lz4").
///
/// # Returns
///
/// A boxed decompressing reader. Unknown/none types pass through unchanged.
pub fn decompressing_reader( pub fn decompressing_reader(
reader: Box<dyn Read>, reader: Box<dyn Read>,
compression: &CompressionType, compression: &CompressionType,
) -> Box<dyn Read> { ) -> Result<Box<dyn Read>, CoreError> {
match compression { match compression {
CompressionType::GZip => { CompressionType::GZip => {
use flate2::read::GzDecoder; use flate2::read::GzDecoder;
Box::new(GzDecoder::new(reader)) Ok(Box::new(GzDecoder::new(reader)))
} }
CompressionType::LZ4 => { CompressionType::LZ4 => {
use lz4_flex::frame::FrameDecoder; use lz4_flex::frame::FrameDecoder;
Box::new(FrameDecoder::new(reader)) Ok(Box::new(FrameDecoder::new(reader)))
} }
#[cfg(feature = "zstd")] #[cfg(feature = "zstd")]
CompressionType::ZStd => { CompressionType::ZStd => {
use zstd::stream::read::Decoder; use zstd::stream::read::Decoder;
Box::new(Decoder::new(reader).expect("Failed to create zstd decoder")) Ok(Box::new(Decoder::new(reader).map_err(|e| {
CoreError::Compression(format!("zstd decoder error: {}", e))
})?))
} }
_ => reader, _ => Ok(reader),
} }
} }
@@ -163,24 +152,24 @@ impl CompressionService {
pub fn compressing_writer( pub fn compressing_writer(
writer: Box<dyn Write>, writer: Box<dyn Write>,
compression: &CompressionType, compression: &CompressionType,
) -> Box<dyn Write> { ) -> Result<Box<dyn Write>, CoreError> {
match compression { match compression {
CompressionType::GZip => { CompressionType::GZip => {
use flate2::Compression; use flate2::Compression;
use flate2::write::GzEncoder; use flate2::write::GzEncoder;
Box::new(GzEncoder::new(writer, Compression::default())) Ok(Box::new(GzEncoder::new(writer, Compression::default())))
} }
CompressionType::LZ4 => Box::new(lz4_flex::frame::FrameEncoder::new(writer)), CompressionType::LZ4 => Ok(Box::new(lz4_flex::frame::FrameEncoder::new(writer))),
#[cfg(feature = "zstd")] #[cfg(feature = "zstd")]
CompressionType::ZStd => { CompressionType::ZStd => {
use zstd::stream::write::Encoder; use zstd::stream::write::Encoder;
Box::new( Ok(Box::new(
Encoder::new(writer, 3) Encoder::new(writer, 3)
.expect("Failed to create zstd encoder") .map_err(|e| CoreError::Compression(format!("zstd encoder error: {}", e)))?
.auto_finish(), .auto_finish(),
) ))
} }
_ => writer, _ => Ok(writer),
} }
} }
} }

View File

@@ -13,32 +13,27 @@ use thiserror::Error;
/// * `ItemNotFoundGeneric` - Generic item not found (no ID specified). /// * `ItemNotFoundGeneric` - Generic item not found (no ID specified).
/// * `InvalidInput(String)` - User or config input validation failure with message. /// * `InvalidInput(String)` - User or config input validation failure with message.
/// * `Compression(String)` - Compression/decompression errors with details. /// * `Compression(String)` - Compression/decompression errors with details.
/// * `PayloadTooLarge` - Request body exceeded maximum allowed size.
/// * `Other(anyhow::Error)` - Catch-all for other anyhow-wrapped errors. /// * `Other(anyhow::Error)` - Catch-all for other anyhow-wrapped errors.
/// * `Migration(rusqlite_migration::Error)` - Database migration failures. /// * `Migration(rusqlite_migration::Error)` - Database migration failures.
#[derive(Error, Debug)] #[derive(Error, Debug)]
pub enum CoreError { pub enum CoreError {
#[error("Database error: {0}")] #[error("Database error: {0}")]
/// Database operation failed.
Database(#[from] rusqlite::Error), Database(#[from] rusqlite::Error),
#[error("I/O error: {0}")] #[error("I/O error: {0}")]
/// File or stream I/O operation failed.
Io(#[from] std::io::Error), Io(#[from] std::io::Error),
#[error("Item not found with id {0}")] #[error("Item not found with id {0}")]
/// Item with the specified ID does not exist in the database.
ItemNotFound(i64), ItemNotFound(i64),
#[error("Item not found")] #[error("Item not found")]
/// Item does not exist (no specific ID).
ItemNotFoundGeneric, ItemNotFoundGeneric,
#[error("Invalid input: {0}")] #[error("Invalid input: {0}")]
/// Input validation failed.
InvalidInput(String), InvalidInput(String),
#[error("Compression error: {0}")] #[error("Compression error: {0}")]
/// Compression or decompression operation failed.
Compression(String), Compression(String),
#[error("Payload too large")]
PayloadTooLarge,
#[error(transparent)] #[error(transparent)]
/// Other unexpected error.
Other(#[from] anyhow::Error), Other(#[from] anyhow::Error),
#[error("Migration error: {0}")] #[error("Migration error: {0}")]
/// Database schema migration failed.
Migration(#[from] rusqlite_migration::Error), Migration(#[from] rusqlite_migration::Error),
} }

View File

@@ -1,5 +1,4 @@
use crate::filter_plugin::{FilterChain, parse_filter_string}; use crate::filter_plugin::{FilterChain, parse_filter_string};
use once_cell::sync::Lazy;
use std::collections::HashMap; use std::collections::HashMap;
use std::io::{Read, Result, Write}; use std::io::{Read, Result, Write};
use std::sync::Mutex; use std::sync::Mutex;
@@ -166,8 +165,8 @@ impl FilterService {
/// # Panics /// # Panics
/// ///
/// Lock acquisition failures (rare) cause panics in accessors. /// Lock acquisition failures (rare) cause panics in accessors.
static FILTER_PLUGIN_REGISTRY: Lazy<Mutex<HashMap<String, FilterConstructor>>> = static FILTER_PLUGIN_REGISTRY: std::sync::LazyLock<Mutex<HashMap<String, FilterConstructor>>> =
Lazy::new(|| Mutex::new(HashMap::new())); std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
/// Registers a filter plugin in the global registry. /// Registers a filter plugin in the global registry.
/// ///

View File

@@ -62,6 +62,12 @@ impl ItemService {
} }
} }
fn item_path(&self, item_id: i64) -> PathBuf {
let mut path = self.data_path.clone();
path.push(item_id.to_string());
path
}
/// Retrieves an item with its associated metadata and tags. /// Retrieves an item with its associated metadata and tags.
/// ///
/// Fetches the item from the database by ID and loads its tags and metadata. /// Fetches the item from the database by ID and loads its tags and metadata.
@@ -159,8 +165,7 @@ impl ItemService {
))); )));
} }
let mut item_path = self.data_path.clone(); let item_path = self.item_path(item_id);
item_path.push(item_id.to_string());
debug!("ITEM_SERVICE: Reading content from path: {item_path:?}"); debug!("ITEM_SERVICE: Reading content from path: {item_path:?}");
let content = self let content = self
@@ -304,8 +309,7 @@ impl ItemService {
))); )));
} }
let mut item_path = self.data_path.clone(); let item_path = self.item_path(item_id);
item_path.push(item_id.to_string());
let reader = self let reader = self
.compression_service .compression_service
@@ -345,8 +349,7 @@ impl ItemService {
))); )));
} }
let mut item_path = self.data_path.clone(); let item_path = self.item_path(item_id);
item_path.push(item_id.to_string());
let reader = self let reader = self
.compression_service .compression_service
@@ -540,8 +543,7 @@ impl ItemService {
let item = db::get_item(conn, id)?.ok_or(CoreError::ItemNotFound(id))?; let item = db::get_item(conn, id)?.ok_or(CoreError::ItemNotFound(id))?;
debug!("ITEM_SERVICE: Found item to delete: {item:?}"); debug!("ITEM_SERVICE: Found item to delete: {item:?}");
let mut item_path = self.data_path.clone(); let item_path = self.item_path(id);
item_path.push(id.to_string());
debug!("ITEM_SERVICE: Deleting file at path: {item_path:?}"); debug!("ITEM_SERVICE: Deleting file at path: {item_path:?}");
let deleted_item = item.clone(); let deleted_item = item.clone();
@@ -662,8 +664,7 @@ impl ItemService {
debug!("ITEM_SERVICE: Got {} meta plugins", plugins.len()); debug!("ITEM_SERVICE: Got {} meta plugins", plugins.len());
meta_service.initialize_plugins(&mut plugins); meta_service.initialize_plugins(&mut plugins);
let mut item_path = self.data_path.clone(); let item_path = self.item_path(item_id);
item_path.push(item_id.to_string());
debug!("ITEM_SERVICE: Writing item to path: {item_path:?}"); debug!("ITEM_SERVICE: Writing item to path: {item_path:?}");
let mut item_out = compression_engine.create(item_path.clone())?; let mut item_out = compression_engine.create(item_path.clone())?;
@@ -859,8 +860,7 @@ impl ItemService {
meta_service.initialize_plugins(&mut plugins); meta_service.initialize_plugins(&mut plugins);
} }
let mut item_path = self.data_path.clone(); let item_path = self.item_path(item_id);
item_path.push(item_id.to_string());
let mut item_out = compression_engine.create(item_path.clone())?; let mut item_out = compression_engine.create(item_path.clone())?;
@@ -933,8 +933,7 @@ impl ItemService {
return self.get_item(conn, item_id); return self.get_item(conn, item_id);
} }
let mut item_path = self.data_path.clone(); let item_path = self.item_path(item_id);
item_path.push(item_id.to_string());
if !item_path.exists() { if !item_path.exists() {
return Err(CoreError::ItemNotFound(item_id)); return Err(CoreError::ItemNotFound(item_id));

View File

@@ -1,3 +1,8 @@
/// Business logic services for the Keep application.
///
/// This module provides the core service layer that orchestrates item storage,
/// compression, metadata collection, and filtering. Services are used by both
/// local CLI modes and the HTTP server.
pub mod compression_service; pub mod compression_service;
pub mod error; pub mod error;
pub mod filter_service; pub mod filter_service;
@@ -13,5 +18,5 @@ pub use filter_service::{FilterService, register_filter_plugin};
pub use item_service::ItemService; pub use item_service::ItemService;
pub use meta_service::MetaService; pub use meta_service::MetaService;
pub use status_service::StatusService; pub use status_service::StatusService;
pub use types::{ItemWithContent, ItemWithMeta}; pub use types::{ItemInfo, ItemWithContent, ItemWithMeta};
pub use utils::{calc_byte_range, extract_tags, parse_comma_tags}; pub use utils::{calc_byte_range, extract_tags, parse_comma_tags};

View File

@@ -62,3 +62,15 @@ pub struct ItemWithContent {
/// The content bytes. /// The content bytes.
pub content: Vec<u8>, pub content: Vec<u8>,
} }
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ItemInfo {
pub id: i64,
pub ts: String,
pub uncompressed_size: Option<i64>,
pub compressed_size: Option<i64>,
pub closed: bool,
pub compression: String,
pub tags: Vec<String>,
pub metadata: HashMap<String, String>,
}

View File

@@ -3,10 +3,10 @@
#[cfg(test)] #[cfg(test)]
pub mod digest_tests; pub mod digest_tests;
#[cfg(feature = "infer")] #[cfg(feature = "meta_infer")]
#[cfg(test)] #[cfg(test)]
pub mod infer_tests; pub mod infer_tests;
#[cfg(feature = "tree_magic_mini")] #[cfg(feature = "meta_tree_magic_mini")]
#[cfg(test)] #[cfg(test)]
pub mod tree_magic_mini_tests; pub mod tree_magic_mini_tests;

View File

@@ -1,5 +1,4 @@
use anyhow::{Result, bail}; use anyhow::{Result, bail};
use once_cell::sync::Lazy;
/// Supported LLM token encodings. /// Supported LLM token encodings.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -48,10 +47,10 @@ impl std::fmt::Debug for Tokenizer {
} }
/// Static tokenizer instances — loaded once per process, shared across all plugins. /// Static tokenizer instances — loaded once per process, shared across all plugins.
static CL100K: Lazy<Tokenizer> = Lazy::new(|| { static CL100K: std::sync::LazyLock<Tokenizer> = std::sync::LazyLock::new(|| {
Tokenizer::new(TokenEncoding::Cl100kBase).expect("Failed to create cl100k_base tokenizer") Tokenizer::new(TokenEncoding::Cl100kBase).expect("Failed to create cl100k_base tokenizer")
}); });
static O200K: Lazy<Tokenizer> = Lazy::new(|| { static O200K: std::sync::LazyLock<Tokenizer> = std::sync::LazyLock::new(|| {
Tokenizer::new(TokenEncoding::O200kBase).expect("Failed to create o200k_base tokenizer") Tokenizer::new(TokenEncoding::O200kBase).expect("Failed to create o200k_base tokenizer")
}); });