asp/keep
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: asp:8a8a6e1c4b22956b774b9022e2471ef02497ebfd
Branches Tags
asp:master
..
compare: asp:bee980605f9cef32d95fc1081e23dbc9aa721504
Branches Tags
asp:master

6 Commits
8a8a6e1c4b ... bee980605f

Author SHA1 Message Date
Andrew Phillips
bee980605f feat: add HTTPS/TLS server support via rustls 
Add optional TLS support for the server using axum-server with the
tls-rustls feature. When --server-cert and --server-key are provided
(and tls feature is enabled), the server binds with TLS instead of
plain HTTP.

Changes:
- Add axum-server dependency with optional tls-rustls feature
- New 'tls' feature flag (independent of 'server')
- --server-cert/--server-key CLI args gated behind tls feature
- ServerConfig extended with cert_file/key_file fields
- Conditional TLS/HTTP binding in server mod.rs
- Fix PathBuf::to_str().unwrap() panic risk -> to_string_lossy()
- Update README.md and DESIGN.md with TLS documentation
 2026-03-12 22:18:42 -03:00
Andrew Phillips
237a581429 fix: add server streaming support, fix pre-existing compilation errors 
Server changes for client mode streaming:
- POST /api/item/ now streams body via async channel → ChannelReader
  → save_item_raw_streaming when compress=false or meta=false
- Add POST /api/item/{id}/meta endpoint for client-side metadata
- Add save_item_raw_streaming<R: Read> to SyncDataService
- Add add_item_meta to AsyncDataService

Fix pre-existing issues that were hidden behind swagger cfg gate:
- Remove #[cfg(feature = "swagger")] from item module so it compiles
  with just the server feature
- Fix parse_comma_tags usage (returns Vec, not Result)
- Fix TextDiff temporary value lifetime issue
- Fix io::Error::new → io::Error::other
- Fix ok_or_else → ok_or for Copy types
- Inline format args throughout server code
- Fix empty line after doc comment in pages.rs
- Add cfg_attr for unused_mut where mcp feature gates mutation
- Add type_complexity allow on create_auth_middleware
- Distinguish task error vs save error in spawn_blocking handlers

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
 2026-03-12 18:02:56 -03:00
Andrew Phillips
c5529bedbf feat: add client mode with streaming support 
Add client mode enabling the keep CLI to connect to a remote keep
server over HTTP. Local plugins (compression, meta, filters) run on
the client; the server stores/retrieves binary blobs.

Architecture:
- Client save uses 3-thread streaming pipeline: reader thread (stdin
  → tee/stdout → hash → compress), OS pipe, streamer thread (pipe →
  chunked HTTP POST). Memory usage is O(PIPESIZE) regardless of data
  size.
- Server accepts compress=false, meta=false, decompress=false query
  params for granular control of server-side processing.
- Streaming body handling on server via async channel → sync reader
  bridge (ChannelReader).

Key additions:
- src/client.rs: KeepClient with post_stream() for chunked upload
- src/modes/client/: save, get, list, info, delete, diff, status
- --client-url / KEEP_CLIENT_URL configuration
- --client-password / KEEP_CLIENT_PASSWORD for auth
- os_pipe dependency for zero-copy pipe streaming

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
 2026-03-12 18:01:36 -03:00
Andrew Phillips
d2581358e9 docs: rewrite README, add LICENSE, remove outdated files 
- Rewrite README.md with comprehensive documentation covering all
  features: compression engines, meta plugins, filter plugins, server
  mode, MCP integration, and configuration
- Add MIT LICENSE file
- Delete README.org (consolidated into README.md)
- Delete empty PLAN.md
- Update AGENTS.md with current build/test commands and conventions

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
 2026-03-12 18:01:23 -03:00
Andrew Phillips
79930f4b01 chore: remove outdated tool usage notes from AGENTS.md 2026-03-12 12:00:32 -03:00
Andrew Phillips
9b7cbd5244 fix: resolve doctest failures, database bugs, and remove dead code 
- Fix all 96 doctest failures across 20 files by adding hidden imports and
  proper test setup (68 pass, 33 intentionally ignored)
- Fix set_item_tags: wrap in transaction and replace item.id.unwrap() with
  proper error handling
- Fix get_items_matching: replace N+1 per-item meta queries with batch
  get_meta_for_items() call
- Fix get_item_matching: apply meta filtering instead of ignoring the parameter
- Remove duplicate doc comment in store_meta
- Remove dead code files: plugin.rs, plugins.rs, binary_detection.rs
  (never declared as modules)
- Apply cargo fmt formatting fixes
- Add keep.db to .gitignore
 2026-03-12 11:58:44 -03:00
58 changed files with 3101 additions and 758 deletions
Expand all files Collapse all files

1
.gitignore vendored
View File

@@ -1,3 +1,4 @@
/target
.aider*
.crush
keep.db

99
AGENTS.md
View File

@@ -1,84 +1,39 @@
# Agent Configuration
**IMPORTANT:** Prefer to use the `write_file` tool if the edit is for the majority of a file, or if you are correcting previous problems made edits from other tools.
## Tools
**IMPORTANT**: Be very careful when quoting text in tool calls to add the right amount of escaping.
### `write_file`
When editing files use the `write_file` tool to output the complete version of the corrected file.
**IMPORTANT**: You must provide the whole file to `write_file`, even the unchanged parts.
**IMPORTANT:** `xxx | keep | zzz` must be as performant as possible in all situations.
## Build/Test Commands
**IMPORTANT**: Do not run application, start the web server, or the trunk server.
**IMPORTANT:** The cargo command cannot be ran in parallel.
**IMPORTANT**: Do not run the application, start the web server, or the trunk server.
**IMPORTANT:** Cargo commands cannot be run in parallel. Prefix all commands with `TERM=dumb`.
```bash
# Check project
TERM=dumb cargo check
# Build project
TERM=dumb cargo build
# DO NOT RUN RUN APPLICATION (native)
# TERM=dumb cargo run
# Run all tests
TERM=dumb cargo test
# Run specific test (by name substring)
TERM=dumb cargo test test_function_name
# Run specific test with verbose output
TERM=dumb cargo test test_function_name -- --nocapture
# Check formatting
TERM=dumb cargo fmt --check
# Apply formatting
TERM=dumb cargo fmt
# Lint with clippy
TERM=dumb cargo clippy -- -D warnings
# Build for release
TERM=dumb cargo build --release
TERM=dumb cargo check # Fast compile check
TERM=dumb cargo build # Build project
TERM=dumb cargo test # Run all tests
TERM=dumb cargo test test_name # Run specific test by name substring
TERM=dumb cargo test -- --nocapture # Verbose test output
TERM=dumb cargo fmt --check # Check formatting
TERM=dumb cargo fmt # Apply formatting
TERM=dumb cargo clippy -- -D warnings # Lint (warnings are errors)
TERM=dumb cargo build --release # Release build
TERM=dumb cargo build --features server # With server feature
```
Prefix commands with `TERM=dumb` for consistent output.
## Code Conventions
## Code Style Guidelines
- `anyhow::Result` for error handling; `thiserror` for custom error types (`src/services/error.rs`)
- Plugin traits: `CompressionEngine`, `FilterPlugin`, `MetaPlugin`
- Dynamic trait objects use `clone_box()` for `Clone` on `Box<dyn Trait>`
- Plugin registration uses `ctor` constructors at module load time
- Filter plugins must implement `filter()`, `clone_box()`, and `options()`
- Meta plugins extend `BaseMetaPlugin` for boilerplate reduction
- Enum string representations: `#[strum(serialize_all = "snake_case")]`
- Lint rules: `deny(clippy::all)`, `deny(unsafe_code)` (except `libc::umask` in main.rs)
- Feature flags: `default = ["magic", "lz4", "gzip"]`; optional: `server`, `mcp`, `swagger`
### Imports
- Group imports in order: standard library, external crates, local modules
- Use explicit imports over glob imports (`use std::fs::File;` not `use std::fs::*;`)
## Testing
### Documentation
- Document all public APIs with rustdoc
- Use examples in documentation only when helpful
## Procedures
### Fix build problems
1. Check the project: `TERM=dumb cargo check`.
2. If there are errors or warnings, create a new sub agent (expert rust developer) that uses the `TERM=dumb cargo check` output as input, planned using strategic thinking.
a. Read all affected files
d. Plan the fixes using strategic thinking:
- Read other files if they provide context or examples
- Look up relevant API information
- Do not downgrade versions
- Preserve functionality
- Use `TERM=dumb cargo fix` if appropriate.
- Prefer the `write_file` tool if there is evidence of double escaping
- You must generate the full file contents when using `write_file` or it will be truncated.
c. Return the list of files modified
3. If any files were modified, loop back to 1.
### Fix formatting
1. Format the project the project: `TERM=dumb cargo fmt`
2. Continue with the fix build problems procedure.
- Tests in `src/tests/` mirroring `src/` structure; shared helpers in `src/tests/common/test_helpers.rs`
- Key helpers: `create_temp_dir()`, `create_temp_db()`, `test_compression_engine()`
- Test naming: `test_<feature>_<scenario>`

331
Cargo.lock generated
View File

@@ -124,6 +124,15 @@ dependencies = [
"derive_arbitrary",
]
[[package]]
name = "arc-swap"
version = "1.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f3647c145568cec02c42054e07bdf9a5a698e15b466fb2341bfc393cd24aa5"
dependencies = [
"rustversion",
]
[[package]]
name = "arraydeque"
version = "0.5.1"
@@ -175,6 +184,28 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "aws-lc-rs"
version = "1.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94bffc006df10ac2a68c83692d734a465f8ee6c5b384d8545a636f81d858f4bf"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.38.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4321e568ed89bb5a7d291a7f37997c2c0df89809d7b6d12062c81ddb54aa782e"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "axum"
version = "0.8.4"
@@ -229,6 +260,28 @@ dependencies = [
"tracing",
]
[[package]]
name = "axum-server"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1df331683d982a0b9492b38127151e6453639cd34926eb9c07d4cd8c6d22bfc"
dependencies = [
"arc-swap",
"bytes",
"either",
"fs-err",
"http",
"http-body",
"hyper",
"hyper-util",
"pin-project-lite",
"rustls",
"rustls-pki-types",
"tokio",
"tokio-rustls",
"tower-service",
]
[[package]]
name = "backtrace"
version = "0.3.75"
@@ -324,6 +377,8 @@ version = "1.2.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2352e5597e9c544d5e6d9c95190d5d27738ade584fa8db0a16e130e5c2b5296e"
dependencies = [
"jobserver",
"libc",
"shlex",
]
@@ -486,6 +541,35 @@ dependencies = [
"unicode-segmentation",
]
[[package]]
name = "cookie"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ddef33a339a91ea89fb53151bd0a4689cfce27055c291dfa69945475d22c747"
dependencies = [
"percent-encoding",
"time",
"version_check",
]
[[package]]
name = "cookie_store"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fc4bff745c9b4c7fb1e97b25d13153da2bc7796260141df62378998d070207f"
dependencies = [
"cookie",
"document-features",
"idna",
"indexmap",
"log",
"serde",
"serde_derive",
"serde_json",
"time",
"url",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
@@ -557,9 +641,9 @@ dependencies = [
[[package]]
name = "crypto-mac"
version = "0.10.1"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bff07008ec701e8028e2ceb8f83f0e4274ee62bd2dbdc4fefff2e9a91824081a"
checksum = "4857fd85a0c34b3c3297875b747c1e02e06b6a0ea32dd892d8192b9ce0813ea6"
dependencies = [
"generic-array",
"subtle",
@@ -610,6 +694,15 @@ dependencies = [
"syn 2.0.105",
]
[[package]]
name = "deranged"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
dependencies = [
"powerfmt",
]
[[package]]
name = "derive_arbitrary"
version = "1.4.2"
@@ -733,6 +826,12 @@ dependencies = [
"litrs",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "dyn-clone"
version = "1.0.20"
@@ -868,6 +967,22 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fs-err"
version = "3.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73fde052dbfc920003cfd2c8e2c6e6d4cc7c1091538c3a24226cec0665ab08c0"
dependencies = [
"autocfg",
"tokio",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "futures"
version = "0.3.31"
@@ -1147,13 +1262,14 @@ dependencies = [
[[package]]
name = "hyper"
version = "1.6.0"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc2b571658e38e0c01b1fdca3bbbe93c00d3d71693ff2770043f8c29bc7d6f80"
checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11"
dependencies = [
"atomic-waker",
"bytes",
"futures-channel",
"futures-util",
"futures-core",
"h2",
"http",
"http-body",
@@ -1161,6 +1277,7 @@ dependencies = [
"httpdate",
"itoa",
"pin-project-lite",
"pin-utils",
"smallvec",
"tokio",
"want",
@@ -1168,12 +1285,11 @@ dependencies = [
[[package]]
name = "hyper-util"
version = "0.1.16"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8d9b05277c7e8da2c93a568989bb6207bef0112e8d17df7a6eda4a3cf143bc5e"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
dependencies = [
"bytes",
"futures-core",
"http",
"http-body",
"hyper",
@@ -1373,6 +1489,16 @@ version = "1.0.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c"
[[package]]
name = "jobserver"
version = "0.1.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
dependencies = [
"getrandom 0.3.3",
"libc",
]
[[package]]
name = "js-sys"
version = "0.3.77"
@@ -1401,6 +1527,7 @@ dependencies = [
"anyhow",
"async-stream",
"axum",
"axum-server",
"base64 0.22.1",
"chrono",
"clap",
@@ -1429,6 +1556,7 @@ dependencies = [
"md5",
"nix",
"once_cell",
"os_pipe",
"pest",
"pest_derive",
"pwhash",
@@ -1455,6 +1583,7 @@ dependencies = [
"tokio-util",
"tower",
"tower-http",
"ureq",
"utoipa",
"utoipa-swagger-ui",
"uzers",
@@ -1719,6 +1848,12 @@ dependencies = [
"minimal-lexical",
]
[[package]]
name = "num-conv"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9"
[[package]]
name = "num-traits"
version = "0.2.19"
@@ -1771,6 +1906,16 @@ dependencies = [
"hashbrown 0.14.5",
]
[[package]]
name = "os_pipe"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967"
dependencies = [
"libc",
"windows-sys 0.60.2",
]
[[package]]
name = "parking_lot"
version = "0.12.4"
@@ -1883,6 +2028,12 @@ dependencies = [
"zerovec",
]
[[package]]
name = "powerfmt"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
[[package]]
name = "ppv-lite86"
version = "0.2.21"
@@ -2010,6 +2161,20 @@ version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c"
[[package]]
name = "ring"
version = "0.17.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
"cfg-if",
"getrandom 0.2.16",
"libc",
"untrusted",
"windows-sys 0.52.0",
]
[[package]]
name = "ringbuf"
version = "0.3.3"
@@ -2153,6 +2318,43 @@ dependencies = [
"windows-sys 0.60.2",
]
[[package]]
name = "rustls"
version = "0.23.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4"
dependencies = [
"aws-lc-rs",
"log",
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
[[package]]
name = "rustls-pki-types"
version = "1.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be040f8b0a225e40375822a563fa9524378b9d63112f53e19ffff34df5d33fdd"
dependencies = [
"zeroize",
]
[[package]]
name = "rustls-webpki"
version = "0.103.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
]
[[package]]
name = "rustversion"
version = "1.0.22"
@@ -2446,9 +2648,9 @@ dependencies = [
[[package]]
name = "subtle"
version = "2.4.1"
version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "syn"
@@ -2569,6 +2771,37 @@ dependencies = [
"cfg-if",
]
[[package]]
name = "time"
version = "0.3.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d"
dependencies = [
"deranged",
"itoa",
"num-conv",
"powerfmt",
"serde",
"time-core",
"time-macros",
]
[[package]]
name = "time-core"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b"
[[package]]
name = "time-macros"
version = "0.2.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3"
dependencies = [
"num-conv",
"time-core",
]
[[package]]
name = "tiny-keccak"
version = "2.0.2"
@@ -2619,6 +2852,16 @@ dependencies = [
"syn 2.0.105",
]
[[package]]
name = "tokio-rustls"
version = "0.26.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61"
dependencies = [
"rustls",
"tokio",
]
[[package]]
name = "tokio-stream"
version = "0.1.17"
@@ -2830,6 +3073,44 @@ version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
[[package]]
name = "untrusted"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
[[package]]
name = "ureq"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fdc97a28575b85cfedf2a7e7d3cc64b3e11bd8ac766666318003abbacc7a21fc"
dependencies = [
"base64 0.22.1",
"cookie_store",
"flate2",
"log",
"percent-encoding",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"ureq-proto",
"utf-8",
"webpki-roots",
]
[[package]]
name = "ureq-proto"
version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d81f9efa9df032be5934a46a068815a10a042b494b6a58cb0a1a97bb5467ed6f"
dependencies = [
"base64 0.22.1",
"http",
"httparse",
"log",
]
[[package]]
name = "url"
version = "2.5.4"
@@ -2841,6 +3122,12 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "utf-8"
version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9"
[[package]]
name = "utf8_iter"
version = "1.0.4"
@@ -3018,6 +3305,15 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "webpki-roots"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "which"
version = "8.0.0"
@@ -3119,6 +3415,15 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
@@ -3378,6 +3683,12 @@ dependencies = [
"synstructure",
]
[[package]]
name = "zeroize"
version = "1.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0"
[[package]]
name = "zerotrie"
version = "0.2.2"

9
Cargo.toml
View File

@@ -70,6 +70,9 @@ pest = "2.8.1"
pest_derive = "2.8.1"
dirs = "6.0.0"
similar = { version = "2.7.0", default-features = false, features = ["text"] }
ureq = { version = "3", features = ["json"], optional = true }
os_pipe = { version = "1", optional = true }
axum-server = { version = "0.8", features = ["tls-rustls"], optional = true }
[features]
# Default features include core compression engines and swagger UI
@@ -102,6 +105,12 @@ mcp = ["dep:rmcp"]
# Swagger UI feature
swagger = ["dep:utoipa-swagger-ui"]
# Client feature (HTTP client for remote server)
client = ["dep:ureq", "dep:os_pipe"]
# TLS feature (HTTPS server support)
tls = ["dep:axum-server"]
[dev-dependencies]
tempfile = "3.3.0"
rand = "0.8.5"

59
DESIGN.md
View File

@@ -31,7 +31,8 @@
- `modes/info.rs` - Show detailed item information
- `modes/diff.rs` - Compare two items
- `modes/status.rs` - Show system status and capabilities
- `modes/server.rs` - REST HTTP server mode with OpenAPI documentation
- `modes/server.rs` - REST HTTP/HTTPS server mode with OpenAPI documentation
- `modes/client.rs` - Client mode for remote server (streaming save, local decompression)
- `modes/common.rs` - Shared utilities for all modes
### Database Module
@@ -57,6 +58,16 @@
- `common/is_binary.rs` - Binary file detection utilities
- `common/status.rs` - Status information generation
### Client Module
- `client.rs` - HTTP client wrapper (ureq-based, supports streaming POST)
- `modes/client/save.rs` - 3-thread streaming save (stdin → tee → compress → pipe → HTTP POST)
- `modes/client/get.rs` - Get with server-side raw fetch + local decompression
- `modes/client/list.rs` - List delegation to server
- `modes/client/info.rs` - Info delegation to server
- `modes/client/delete.rs` - Delete delegation to server
- `modes/client/diff.rs` - Diff delegation to server
- `modes/client/status.rs` - Status delegation to server
### Utility Modules
- `plugins.rs` - Shared plugin utilities
- `args.rs` - CLI argument definitions
@@ -88,8 +99,14 @@
- `--quiet` - Do not show any messages
- `--output-format <table|json|yaml>` - Output format for info, status, and list modes
- `--server-password <PASSWORD>` - Password for server authentication
- `--server-cert <PATH>` - TLS certificate file (PEM) for HTTPS server
- `--server-key <PATH>` - TLS private key file (PEM) for HTTPS server
- `--force` - Force output even when binary data would be sent to a TTY
### Client Options (requires `client` feature)
- `--client-url <URL>` - Remote keep server URL
- `--client-password <PASSWORD>` - Remote server password
## Data Storage
### Database Schema
@@ -107,17 +124,31 @@
### Status Operations
- `GET /api/status` - Get system status information
- `GET /api/plugins/status` - Get plugin status information
### Item Operations
- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest|oldest`, `start=0`, `count=100`, `tags[]=tag1&tags[]=tag2`
- `POST /api/item/` - Add a new item
- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest|oldest`, `start=0`, `count=100`, `tags=tag1,tag2`
- `POST /api/item/` - Add a new item (body: raw content). Query params: `tags`, `metadata` (JSON), `compress=true|false`, `meta=true|false`
- `POST /api/item/<#>/meta` - Add metadata to an existing item (body: JSON object)
- `DELETE /api/item/<#>` - Delete an item
- `GET /api/item/latest` - Return the latest item as JSON. Optional params: `tags[]=tag1&tags[]=tag2`, `allow_binary=true|false`
- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags[]=tag1&tags[]=tag2`
- `GET /api/item/latest/content` - Return the raw content of the latest item. Optional params: `tags[]=tag1&tags[]=tag2`
- `GET /api/item/latest` - Return the latest item as JSON. Optional params: `tags=tag1,tag2`, `allow_binary=true|false`
- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags=tag1,tag2`
- `GET /api/item/latest/content` - Return the raw content of the latest item. Optional params: `tags=tag1,tag2`, `decompress=true|false`
- `GET /api/item/<#>` - Return the item as JSON. Optional params: `allow_binary=true|false`
- `GET /api/item/<#>/meta` - Return the item metadata as JSON
- `GET /api/item/<#>/content` - Return the raw content of the item
- `GET /api/item/<#>/content` - Return the raw content of the item. Optional params: `decompress=true|false`
- `GET /api/diff` - Diff two items. Params: `id_a`, `id_b`
### Server Modes
- **Plain HTTP** (default): `tokio::net::TcpListener` + `axum::serve()`
- **HTTPS** (with `tls` feature): `axum_server::bind_rustls()` with rustls when `--server-cert` and `--server-key` are provided
- Conditional selection at startup: cert+key present → HTTPS, otherwise → HTTP
### Client/Server Protocol
- Smart clients (keep CLI) set `compress=false` and `meta=false` on POST, handling compression/metadata locally
- Dumb clients (curl) use defaults (`compress=true`, `meta=true`), server handles everything
- GET responses include `X-Keep-Compression` header when `decompress=false`
- Streaming save uses chunked transfer encoding for constant memory usage
### Authentication
- Bearer token authentication: `Authorization: Bearer <password>`
@@ -173,5 +204,19 @@
- File permissions are restricted to user only (umask 077)
- Input validation for item IDs to prevent path traversal
- Authentication for server mode with bearer or basic auth
- TLS/HTTPS support via rustls when certificate and key are provided
- Proper resource cleanup using RAII patterns
- Safe handling of external processes with proper stdin/stdout management
## Feature Flags
- `server` - HTTP REST API server (axum-based)
- `tls` - HTTPS/TLS support for server (axum-server + rustls)
- `client` - HTTP client for remote server (ureq-based, includes streaming save)
- `mcp` - Model Context Protocol for AI assistant integration
- `swagger` - OpenAPI/Swagger UI documentation
- `magic` - File type detection via libmagic
- `lz4` - LZ4 compression (internal)
- `gzip` - GZip compression (internal)
- `bzip2` - BZip2 compression (external)
- `xz` - XZ compression (external)
- `zstd` - ZStd compression (external)

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Andrew Phillips
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

0
PLAN.md
View File

711
README.md
View File

@@ -1,16 +1,709 @@
# Keep - Temporary File Management with Compression and Metadata
# Keep
Keep is a command-line tool for managing temporary files with automatic compression, metadata generation, and querying capabilities. It supports various compression algorithms and metadata plugins for rich item inspection.
A command-line utility for storing and retrieving temporary data with automatic compression, metadata extraction, and querying. Pipe any output into `keep` for organized storage — no more losing data in `/tmp` files with cryptic names.
```sh
# Instead of this:
curl -s https://api.example.com/data > /tmp/api-data.json
# Do this:
curl -s https://api.example.com/data | keep --save api-data
keep --get api-data
```
## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Usage](#usage)
- [Save Mode](#save-mode)
- [Get Mode](#get-mode)
- [List Mode](#list-mode)
- [Info Mode](#info-mode)
- [Update Mode](#update-mode)
- [Delete Mode](#delete-mode)
- [Diff Mode](#diff-mode)
- [Status Mode](#status-mode)
- [Filters](#filters)
- [Compression](#compression)
- [Meta Plugins](#meta-plugins)
- [Configuration](#configuration)
- [Client/Server Mode](#clientserver-mode)
- [Server Mode](#server-mode)
- [Client Mode](#client-mode)
- [API Endpoints](#api-endpoints)
- [MCP (Model Context Protocol)](#mcp-model-context-protocol)
- [Shell Integration](#shell-integration)
- [Feature Flags](#feature-flags)
- [License](#license)
## Features
- **Store and Retrieve**: Save content with automatic compression and retrieve by ID or tags.
- **Compression Support**: Built-in support for LZ4, GZip, and more via external programs (BZip2, XZ, ZStd).
- **Metadata Plugins**: Automatic extraction of file type, digests, hostname, user info, and custom metadata.
- **Filtering**: Apply filters (head, tail, grep, etc.) when retrieving content.
- **Querying**: List, search, and diff items with flexible formatting.
- **REST API Server**: Optional HTTP server for programmatic access.
- **Modular Design**: Extensible via plugins for compression, metadata, and filtering.
- **Store and retrieve** — Save content with tags, retrieve by ID or tag
- **Automatic compression** — LZ4, GZip, BZip2, XZ, ZStd support
- **Metadata plugins** — Auto-extract file type, digests, hostname, user info, and more
- **Filters** — Apply transformations (head, tail, grep, strip ANSI) on retrieval
- **Querying** — List, search, diff items with flexible formatting
- **Client/server architecture** — Optional HTTP server with streaming support
- **MCP support** — Model Context Protocol integration for AI assistants
- **Modular design** — Extensible plugin system for compression, metadata, and filtering
## Installation
### From Source
Requires Rust and Cargo.
```sh
cargo build --release
```
### Install via Cargo
```sh
cargo install --path .
```
### Static Binary (Linux)
```sh
./build-static.bash
# Binary at bin/keep
```
### Build with Server/Client Features
```sh
# Server only
cargo build --release --features server
# Client only (for connecting to a remote keep server)
cargo build --release --features client
# Server + client + all optional features
cargo build --release --features server,tls,client,swagger,mcp
```
## Quick Start
```sh
# Save content with a tag
echo "Hello, world!" | keep --save greeting
# Retrieve by tag
keep --get greeting
# List all stored items
keep --list
# Get item details
keep --info greeting
# Delete by tag
keep --delete greeting
```
### Real-World Examples
```sh
# Save API response
curl -s https://api.github.com/repos/user/repo | keep --save repo-info
# Save test output with metadata
npm test 2>&1 | keep --save test-results --meta project=myapp --meta env=staging
# Chain commands: process and store
cat data.csv | sort | uniq | keep --save cleaned-data
# Diff two versions
keep --diff 1 5
# Get first 20 lines of an item
keep --get 1 --filters "head_lines(20)"
# List items from a specific project
keep --list --meta project=myapp
```
## Usage
### Save Mode
Save stdin content with tags and metadata.
```sh
# Save (auto-assigned ID, no tag)
echo "data" | keep --save
# Save with a tag
echo "data" | keep --save my-tag
# Save with multiple tags and metadata
cat report.pdf | keep --save report --meta project=alpha --meta env=prod
# Specify compression and digest algorithm
echo "data" | keep --save my-tag --compression gzip --digest sha256
```
Tags and metadata make items easy to find later. Tags are simple identifiers; metadata is key-value pairs.
### Get Mode
Retrieve items by ID or tags. This is the default mode when IDs are provided.
```sh
# Get by ID
keep --get 1
keep 1
# Get by tag
keep --get my-tag
keep my-tag
# Get with filters applied
keep --get 1 --filters "head_lines(10)"
# Get by metadata filter
keep --get --meta project=alpha
# Force binary output to TTY (override safety check)
keep --get 1 --force
```
### List Mode
List stored items with filtering and formatting.
```sh
# List all items
keep --list
# List by tag
keep --list my-tag
# Filter by metadata
keep --list --meta env=prod
# Custom column format
keep --list --list-format "id,time,size,tags"
# JSON output for scripting
keep --list --output-format json
# Human-readable file sizes
keep --list --human-readable
```
### Info Mode
Show detailed information about an item.
```sh
keep --info 1
keep --info my-tag
keep --info --meta key=value
```
### Update Mode
Update an item's tags and metadata.
```sh
# Replace tags
keep --update 1 new-tag
# Update metadata
keep --update 1 --meta key=newvalue
# Remove a metadata key
keep --update 1 --meta key
```
### Delete Mode
Delete items by ID.
```sh
keep --delete 1
keep --delete 1 2 3
```
### Diff Mode
Show differences between two items.
```sh
keep --diff 1 2
```
### Status Mode
Show system status and supported features.
```sh
keep --status
keep --status-plugins
keep --status --verbose
```
## Filters
Apply transformations to item content during retrieval. Filters are chained with `|`.
```sh
# First 10 lines
keep --get 1 --filters "head_lines(10)"
# Skip first 5 lines, then grep for errors
keep --get 1 --filters "skip_lines(5)|grep(pattern=error)"
# Strip ANSI escape codes
keep --get 1 --filters "strip_ansi"
# Last 100 bytes
keep --get 1 --filters "tail_bytes(100)"
# Complex chain
keep --get 1 --filters "skip_lines(10)|grep(pattern=TODO)|head_lines(5)"
```
### Available Filters
| Filter | Description | Parameters |
|--------|-------------|------------|
| `head_bytes(n)` | First n bytes | `count` |
| `head_lines(n)` | First n lines | `count` |
| `tail_bytes(n)` | Last n bytes | `count` |
| `tail_lines(n)` | Last n lines | `count` |
| `skip_bytes(n)` | Skip first n bytes | `count` |
| `skip_lines(n)` | Skip first n lines | `count` |
| `grep(pattern)` | Filter matching lines | `pattern` (regex) |
| `strip_ansi` | Remove ANSI escape codes | none |
Set `KEEP_FILTERS` to apply a default filter chain to all retrievals.
## Compression
Items are compressed automatically on save. Default: LZ4.
| Algorithm | Type | Speed | Ratio |
|-----------|------|-------|-------|
| `lz4` | Internal | Fastest | Lower |
| `gzip` | Internal | Fast | Good |
| `bzip2` | External | Slow | Better |
| `xz` | External | Slowest | Best |
| `zstd` | External | Fast | Good |
| `none` | Internal | N/A | N/A |
```sh
# Specify compression per item
echo "data" | keep --save my-tag --compression zstd
# Set default via environment
export KEEP_COMPRESSION=gzip
```
External compression programs (`bzip2`, `xz`, `zstd`) must be installed on the system.
## Meta Plugins
Metadata is automatically extracted when saving items.
| Plugin | Key | Description |
|--------|-----|-------------|
| `env` | `*` | Capture `KEEP_META_*` environment variables |
| `magic_file` | `file_type` | File type detection (requires `magic` feature) |
| `text` | `text_line_count`, `text_word_count` | Line and word counts |
| `user` | `uid`, `user`, `gid`, `group` | Current user info |
| `shell` | `shell` | Current shell path |
| `shell_pid` | `shell_pid` | Shell process ID |
| `keep_pid` | `keep_pid` | Keep process ID |
| `digest` | `digest_sha256`, `digest_md5` | Content digests |
| `read_time` | `read_time` | Time to read content |
| `read_rate` | `read_rate` | Data read rate |
| `hostname` | `hostname`, `hostname_short` | System hostname |
| `exec` | Custom | Run external commands for metadata |
| `cwd` | `cwd` | Current working directory |
```sh
# Use specific plugins
echo "data" | keep --save tag --meta-plugins "digest,text,user"
# Capture custom metadata via environment
KEEP_META_project=alpha echo "data" | keep --save tag
# Combine environment and CLI metadata
KEEP_META_build=1234 echo "data" | keep --save tag --meta env=staging
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `KEEP_DIR` | Storage directory | `~/.keep` |
| `KEEP_CONFIG` | Config file path | `~/.config/keep/config.yml` |
| `KEEP_COMPRESSION` | Compression algorithm | `lz4` |
| `KEEP_META_PLUGINS` | Meta plugins to use | `env` |
| `KEEP_FILTERS` | Default filter chain | none |
| `KEEP_LIST_FORMAT` | List column format | built-in defaults |
| `KEEP_SERVER_ADDRESS` | Server bind address | `127.0.0.1` |
| `KEEP_SERVER_PORT` | Server port | `21080` |
| `KEEP_SERVER_PASSWORD` | Server password | none |
| `KEEP_SERVER_PASSWORD_HASH` | Server password hash | none |
| `KEEP_SERVER_CERT` | TLS certificate file path (PEM) | none |
| `KEEP_SERVER_KEY` | TLS private key file path (PEM) | none |
| `KEEP_CLIENT_URL` | Remote keep server URL | none |
| `KEEP_CLIENT_PASSWORD` | Remote server password | none |
Any config setting can be overridden with `KEEP__<SETTING>` environment variables (double underscore separator).
### Configuration File
Default location: `~/.config/keep/config.yml`
Generate a default configuration:
```sh
keep --generate-config > ~/.config/keep/config.yml
```
```yaml
# Storage directory
dir: ~/.keep
# List view columns
list_format:
- name: id
label: "Item"
align: right
- name: time
label: "Time"
align: right
- name: size
label: "Size"
align: right
- name: tags
label: "Tags"
align: left
# Table styling
table_config:
style: utf8_full
content_arrangement: dynamic
# Default compression
compression_plugin:
name: gzip
# Default meta plugins
meta_plugins:
- name: env
- name: digest
options:
algorithm: sha256
# Server settings
server:
address: "127.0.0.1"
port: 21080
password: "secret"
# TLS (requires tls feature)
# cert_file: /path/to/cert.pem
# key_file: /path/to/key.pem
# Client settings
client:
url: "http://localhost:21080"
password: "secret"
human_readable: true
quiet: false
force: false
```
## Client/Server Mode
Keep supports a client/server architecture where one machine runs a keep server and other machines connect as clients. This is useful for:
- Centralizing stored data across multiple machines
- Sharing items between team members
- Offloading storage to a dedicated server
- Piping data from long-running processes without local storage
### Server Mode
Start an HTTP REST API server:
```sh
# Default: 127.0.0.1:21080
keep --server
# Custom address and port
keep --server --server-address 0.0.0.0 --server-port 8080
# With authentication
keep --server --server-password mypassword
```
#### HTTPS / TLS
Build with the `tls` feature to enable HTTPS:
```sh
cargo build --release --features server,tls
```
Provide a TLS certificate and private key (both PEM format):
```sh
# Via CLI flags
keep --server \
--server-cert /path/to/cert.pem \
--server-key /path/to/key.pem
# Via environment variables
export KEEP_SERVER_CERT=/path/to/cert.pem
export KEEP_SERVER_KEY=/path/to/key.pem
keep --server
# Via config file (config.yml)
server:
cert_file: /path/to/cert.pem
key_file: /path/to/key.pem
```
When cert and key are provided, the server listens with HTTPS. Without them, it falls back to plain HTTP. The port is controlled by `--server-port` (default: 21080).
**Self-signed certificates** (for development):
```sh
# Generate a self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
-days 365 -nodes -subj "/CN=localhost"
# Start server with self-signed cert
keep --server --server-cert cert.pem --server-key key.pem
# Connect client with HTTPS
keep --client-url https://localhost:21080 --save my-tag
```
The server accepts data from both dumb clients (raw HTTP/curl) and smart clients (the keep CLI).
#### Server Query Parameters
The server supports query parameters that control processing:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `tags` | none | Comma-separated tags |
| `metadata` | none | JSON-encoded metadata |
| `compress` | `true` | `false` = client already compressed, store as-is |
| `meta` | `true` | `false` = client handles metadata, skip server-side plugins |
| `decompress` | `true` | `false` = return raw compressed bytes on GET |
When using a smart client, these are set automatically. For curl, the server handles everything by default.
#### Example: Curl as a Dumb Client
```sh
# Save (server handles compression and metadata)
curl -X POST -d "my data" http://localhost:21080/api/item/?tags=my-tag
# Retrieve (server decompresses)
curl http://localhost:21080/api/item/1/content
# Save compressed (client handles compression, server skips)
gzip -c data.txt | curl -X POST -d @- "http://localhost:21080/api/item/?compress=false&tags=my-tag"
```
### Client Mode
The keep CLI can connect to a remote server as a smart client. Build with the `client` feature:
```sh
cargo build --release --features client
```
```sh
# Set server URL via flag or environment
keep --client-url http://server:21080 --save my-tag
export KEEP_CLIENT_URL=http://server:21080
# With authentication
keep --client-url http://server:21080 --client-password mypassword --save my-tag
export KEEP_CLIENT_PASSWORD=mypassword
```
#### How Client Mode Works
Client mode uses **local plugins** and **remote storage**:
1. **Save**: Local compression and metadata plugins run on the client; compressed data streams to the server
2. **Get**: Server sends raw compressed data; client decompresses locally and applies filters
3. **Other operations** (list, info, delete, diff): Delegated directly to the server
This means client behavior is consistent with local mode — the same compression settings and filters apply.
#### Streaming Architecture
Client save uses a 3-thread streaming pipeline for constant memory usage regardless of data size:
```
┌──────────────┐ OS pipe ┌────────────────┐
│ Reader thread ├──────────────────┤ Streamer thread│
│ │ (compressed │ │
│ stdin → tee │ bytes) │ pipe → POST │
│ → hash │ │ (chunked) │
│ → compress│ │ │
└──────────────┘ └────────────────┘
│ │
▼ ▼
stdout + Server stores blob
SHA-256 digest
```
- **Reader thread**: Reads stdin, tees output to stdout, computes SHA-256, compresses data, writes to OS pipe
- **Streamer thread**: Reads compressed bytes from pipe, streams to server via chunked HTTP POST
- **Main thread**: After streaming completes, sends computed metadata (digest, hostname, size) to server
Memory usage is O(PIPESIZE) — typically 64KB — regardless of how much data is being stored.
#### Example: Remote Pipeline
```sh
# On a build server, pipe logs to a central keep server
make build 2>&1 | keep --client-url http://logserver:21080 \
--save build-logs \
--meta project=myapp \
--meta branch=$(git branch --show-current)
# Retrieve from any machine
keep --client-url http://logserver:21080 --get build-logs
# List recent builds from a specific project
keep --client-url http://logserver:21080 --list --meta project=myapp
```
### API Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/status` | System status |
| `GET` | `/api/plugins/status` | Plugin status |
| `GET` | `/api/item/` | List items (`tags`, `order`, `start`, `count` params) |
| `POST` | `/api/item/` | Create item (body: raw content, params: `tags`, `metadata`, `compress`, `meta`) |
| `GET` | `/api/item/latest/content` | Latest item content |
| `GET` | `/api/item/latest/meta` | Latest item metadata |
| `GET` | `/api/item/{id}` | Item info by ID |
| `GET` | `/api/item/{id}/content` | Item content by ID |
| `GET` | `/api/item/{id}/meta` | Item metadata by ID |
| `GET` | `/api/item/{id}/info` | Item info by ID |
| `POST` | `/api/item/{id}/meta` | Add metadata to existing item (body: JSON object) |
| `DELETE` | `/api/item/{id}` | Delete item by ID |
| `GET` | `/api/diff` | Diff two items (`id_a`, `id_b` params) |
#### Authentication
```sh
# Bearer token
curl -H "Authorization: Bearer mypassword" http://localhost:21080/api/status
# Basic auth
curl -u keep:mypassword http://localhost:21080/api/status
```
When no password is configured, authentication is disabled.
#### Swagger UI
Build with the `swagger` feature to enable OpenAPI documentation:
```sh
cargo build --features server,swagger
```
Swagger UI available at `/swagger`, OpenAPI spec at `/openapi.json`.
## MCP (Model Context Protocol)
AI assistant integration via the Model Context Protocol. Enable with the `mcp` feature.
```sh
cargo build --features server,mcp
```
MCP endpoint available at `/mcp/sse` when the server is running.
### Available Tools
| Tool | Description | Parameters |
|------|-------------|------------|
| `save_item` | Save new content | `content`, `tags[]`, `metadata{}` |
| `get_item` | Get item by ID | `id` |
| `get_latest_item` | Get latest item | `tags[]` |
| `list_items` | List items | `tags[]`, `limit`, `offset` |
| `search_items` | Search items | `tags[]`, `metadata{}` |
## Shell Integration
Source `profile.bash` to enable shell integration:
```sh
source /path/to/keep/profile.bash
```
This provides:
- **`keep` function** — Captures the current command in metadata automatically
- **`@` alias** — Shorthand for `keep --save`
- **`@@` alias** — Shorthand for `keep --get`
```sh
# Save with automatic command capture
curl -s api.example.com | @ api-response
# Quick retrieve
@@ api-response
```
## Feature Flags
| Feature | Default | Description |
|---------|---------|-------------|
| `magic` | Yes | File type detection via libmagic |
| `lz4` | Yes | LZ4 compression (internal) |
| `gzip` | Yes | GZip compression (internal) |
| `server` | No | HTTP REST API server |
| `tls` | No | HTTPS/TLS server support (requires `server`) |
| `client` | No | HTTP client for remote server |
| `mcp` | No | Model Context Protocol support |
| `swagger` | No | Swagger UI for API docs |
| `bzip2` | No | BZip2 compression (external program) |
| `xz` | No | XZ compression (external program) |
| `zstd` | No | ZStd compression (external program) |
```sh
# Server with Swagger UI
cargo build --features server,swagger
# Server with HTTPS
cargo build --features server,tls
# Client only
cargo build --features client
# Everything
cargo build --features server,tls,client,mcp,swagger,magic
```
## License
MIT License - see [LICENSE](LICENSE) for details.
## Contact
Andrew Phillips - andrew@gt0.ca

141
README.org
View File

@@ -1,141 +0,0 @@
#+TITLE: Keep
#+AUTHOR: Andrew Phillips
* Introduction
Keep is a command-line utility designed to manage temporary files created on the command line. Instead of redirecting output to a temporary file (e.g., =command > ~/whatever.tmp=), you can use =keep= to handle the temporary files for you (e.g., =command | keep=).
* Installation
To install Keep, you need to have Rust and Cargo installed on your system. You can then build and install Keep using the following commands:
#+BEGIN_SRC sh
cargo build --release
cargo install --path .
#+END_SRC
* Usage
Keep provides several subcommands to manage temporary files. Below are some examples of how to use Keep.
** Saving an Item
To save an item with tags and metadata, you can use the =--save= option:
#+BEGIN_SRC sh
echo "Hello, world!" | keep --save example --meta key=value
#+END_SRC
** Getting an Item
To retrieve an item by its ID or by matching tags and metadata, you can use the =--get= option:
#+BEGIN_SRC sh
keep --get 1
keep --get example
keep --get --meta key=value
keep 1
keep example
#+END_SRC
** Listing Items
To list all items or filter them by tags and metadata, you can use the =--list= option:
#+BEGIN_SRC sh
keep --list
keep --list example
keep --list --meta key=value
#+END_SRC
** Updating an Item
To update an item's tags and metadata, you can use the =--update= option:
#+BEGIN_SRC sh
keep --update 1 newtag --meta key=newvalue
#+END_SRC
** Deleting an Item
To delete an item by its ID or by matching tags, you can use the =--delete= option:
#+BEGIN_SRC sh
keep --delete 1
keep --delete example
#+END_SRC
** Showing Status
To show the status of directories and supported compression algorithms, you can use the =--status= option:
#+BEGIN_SRC sh
keep --status
#+END_SRC
** Diffing Items
To show a diff between two items by ID, you can use the =--diff= option:
#+BEGIN_SRC sh
keep --diff 1 2
#+END_SRC
** Getting Information About an Item
To get detailed information about an item by its ID or by matching tags and metadata, you can use the =--info= option:
#+BEGIN_SRC sh
keep --info 1
keep --info example
keep --info --meta key=value
#+END_SRC
* Configuration
Keep can be configured using environment variables and command-line options. The following environment variables are supported:
- =KEEP_DIR=: Specify the directory to use for storage.
- =KEEP_LIST_FORMAT=: A comma-separated list of columns to display with =--list=.
- =KEEP_DIGEST=: Digest algorithm to use when saving items.
- =KEEP_COMPRESSION=: Compression algorithm to use when saving items.
* Examples
Here are some examples of how to use Keep with different options:
** Saving an Item with Compression and Digest
#+BEGIN_SRC sh
echo "Hello, world!" | keep --save example --meta key=value --compression gzip --digest sha256
#+END_SRC
** Getting an Item with Human-Readable Sizes
#+BEGIN_SRC sh
keep --get 1 --human-readable
#+END_SRC
** Listing Items with Custom Format
#+BEGIN_SRC sh
keep --list --list-format "id,time,size,tags,meta:hostname"
#+END_SRC
** Updating an Item with New Tags and Metadata
#+BEGIN_SRC sh
keep --update 1 newtag --meta key=newvalue
#+END_SRC
** Deleting an Item by Tag
#+BEGIN_SRC sh
keep --delete example
#+END_SRC
** Showing Status with Verbose Output
#+BEGIN_SRC sh
keep --status --verbose
#+END_SRC
** Diffing Items with IDs
#+BEGIN_SRC sh
keep --diff 1 2
#+END_SRC
** Getting Information About an Item with Metadata
#+BEGIN_SRC sh
keep --info 1
#+END_SRC
* License
Keep is licensed under the MIT License. See the LICENSE file for more details.
* Contributing
Contributions are welcome! Please open an issue or submit a pull request on the GitHub repository.
* Contact
For more information, please contact Andrew Phillips at andrew@gt0.ca.

20
src/args.rs
View File

@@ -75,6 +75,16 @@ pub struct ModeArgs {
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_PORT"))]
#[arg(help("Server port to bind to"))]
pub server_port: Option<u16>,
#[cfg(feature = "tls")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_CERT"))]
#[arg(help("Path to TLS certificate file (PEM) for HTTPS"))]
pub server_cert: Option<PathBuf>,
#[cfg(feature = "tls")]
#[arg(help_heading("Server Options"), long, env("KEEP_SERVER_KEY"))]
#[arg(help("Path to TLS private key file (PEM) for HTTPS"))]
pub server_key: Option<PathBuf>,
}
/// Struct for item-specific arguments, such as compression and plugins.
@@ -141,6 +151,16 @@ pub struct OptionsArgs {
#[arg(help("Password hash for server authentication (requires --server)"))]
pub server_password_hash: Option<String>,
#[cfg(feature = "client")]
#[arg(long, env("KEEP_CLIENT_URL"), help_heading("Client Options"))]
#[arg(help("Remote keep server URL for client mode"))]
pub client_url: Option<String>,
#[cfg(feature = "client")]
#[arg(long, env("KEEP_CLIENT_PASSWORD"), help_heading("Client Options"))]
#[arg(help("Password for remote keep server authentication"))]
pub client_password: Option<String>,
#[arg(
long,
help("Force output even when binary data would be sent to a TTY")

310
src/client.rs Normal file
View File

@@ -0,0 +1,310 @@
use crate::services::error::CoreError;
use serde::de::DeserializeOwned;
use std::collections::HashMap;
use std::io::Read;
/// Item information returned from the server API.
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
pub struct ItemInfo {
pub id: i64,
pub ts: String,
pub size: Option<i64>,
pub compression: String,
pub tags: Vec<String>,
pub metadata: HashMap<String, String>,
}
pub struct KeepClient {
base_url: String,
agent: ureq::Agent,
password: Option<String>,
}
impl KeepClient {
pub fn new(base_url: &str, password: Option<String>) -> Result<Self, CoreError> {
let base_url = base_url.trim_end_matches('/').to_string();
let agent = ureq::Agent::new_with_defaults();
Ok(Self {
base_url,
agent,
password,
})
}
pub fn base_url(&self) -> &str {
&self.base_url
}
pub fn password(&self) -> Option<&String> {
self.password.as_ref()
}
fn url(&self, path: &str) -> String {
format!("{}{}", self.base_url, path)
}
fn handle_error<T>(&self, result: Result<T, ureq::Error>) -> Result<T, CoreError> {
match result {
Ok(v) => Ok(v),
Err(ureq::Error::StatusCode(code)) => Err(CoreError::Other(anyhow::anyhow!(
"Server returned error: HTTP {}",
code
))),
Err(e) => Err(CoreError::Other(anyhow::anyhow!("Request failed: {}", e))),
}
}
pub fn get_json<T: DeserializeOwned>(&self, path: &str) -> Result<T, CoreError> {
let url = self.url(path);
let mut req = self.agent.get(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
let response = self.handle_error(req.call())?;
let body: T = self.handle_error(response.into_body().read_json())?;
Ok(body)
}
pub fn get_json_with_query<T: DeserializeOwned>(
&self,
path: &str,
params: &[(&str, &str)],
) -> Result<T, CoreError> {
let mut url = self.url(path);
if !params.is_empty() {
url.push('?');
for (i, (key, value)) in params.iter().enumerate() {
if i > 0 {
url.push('&');
}
url.push_str(&format!("{key}={value}"));
}
}
let mut req = self.agent.get(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
let response = self.handle_error(req.call())?;
let body: T = self.handle_error(response.into_body().read_json())?;
Ok(body)
}
pub fn get_bytes(&self, path: &str) -> Result<Vec<u8>, CoreError> {
let url = self.url(path);
let mut req = self.agent.get(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
let response = self.handle_error(req.call())?;
let mut body = response.into_body();
let bytes = body
.read_to_vec()
.map_err(|e| CoreError::Other(anyhow::anyhow!("{}", e)))?;
Ok(bytes)
}
pub fn post_bytes(
&self,
path: &str,
body_bytes: &[u8],
params: &[(&str, &str)],
) -> Result<ItemInfo, CoreError> {
let mut cursor = std::io::Cursor::new(body_bytes);
self.post_stream(path, &mut cursor, params)
}
/// Stream data from a reader to the server using chunked transfer encoding.
///
/// The reader is consumed in chunks and sent to the server without buffering
/// the entire body in memory. This enables true streaming for large payloads.
pub fn post_stream(
&self,
path: &str,
body_reader: &mut dyn Read,
params: &[(&str, &str)],
) -> Result<ItemInfo, CoreError> {
let mut url = self.url(path);
if !params.is_empty() {
url.push('?');
for (i, (key, value)) in params.iter().enumerate() {
if i > 0 {
url.push('&');
}
url.push_str(&format!("{key}={value}"));
}
}
let mut req = self.agent.post(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
req = req.header("Content-Type", "application/octet-stream");
let response = self.handle_error(req.send(ureq::SendBody::from_reader(body_reader)))?;
#[derive(serde::Deserialize)]
struct ApiResponse {
data: Option<ItemInfo>,
error: Option<String>,
}
let api_response: ApiResponse = self.handle_error(response.into_body().read_json())?;
if let Some(error) = api_response.error {
return Err(CoreError::Other(anyhow::anyhow!("Server error: {}", error)));
}
api_response
.data
.ok_or_else(|| CoreError::Other(anyhow::anyhow!("No data in response")))
}
pub fn delete(&self, path: &str) -> Result<(), CoreError> {
let url = self.url(path);
let mut req = self.agent.delete(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
self.handle_error(req.call())?;
Ok(())
}
pub fn get_status(&self) -> Result<serde_json::Value, CoreError> {
self.get_json("/api/status")
}
pub fn get_item_info(&self, id: i64) -> Result<ItemInfo, CoreError> {
#[derive(serde::Deserialize)]
struct ApiResponse {
data: Option<ItemInfo>,
}
let response: ApiResponse = self.get_json(&format!("/api/item/{id}/info"))?;
response
.data
.ok_or_else(|| CoreError::Other(anyhow::anyhow!("Item not found")))
}
pub fn list_items(
&self,
tags: &[String],
order: &str,
start: u64,
count: u64,
) -> Result<Vec<ItemInfo>, CoreError> {
#[derive(serde::Deserialize)]
struct ApiResponse {
data: Option<Vec<ItemInfo>>,
}
let mut params: Vec<(String, String)> = Vec::new();
params.push(("order".to_string(), order.to_string()));
params.push(("start".to_string(), start.to_string()));
params.push(("count".to_string(), count.to_string()));
if !tags.is_empty() {
params.push(("tags".to_string(), tags.join(",")));
}
let param_refs: Vec<(&str, &str)> = params
.iter()
.map(|(k, v)| (k.as_str(), v.as_str()))
.collect();
let response: ApiResponse = self.get_json_with_query("/api/item/", &param_refs)?;
Ok(response.data.unwrap_or_default())
}
pub fn save_item(
&self,
content: &[u8],
tags: &[String],
metadata: &HashMap<String, String>,
compress: bool,
meta: bool,
) -> Result<ItemInfo, CoreError> {
let mut params: Vec<(String, String)> = Vec::new();
if !tags.is_empty() {
params.push(("tags".to_string(), tags.join(",")));
}
if !metadata.is_empty() {
let meta_json = serde_json::to_string(metadata).map_err(|e| {
CoreError::Other(anyhow::anyhow!("Failed to serialize metadata: {}", e))
})?;
params.push(("metadata".to_string(), meta_json));
}
params.push(("compress".to_string(), compress.to_string()));
params.push(("meta".to_string(), meta.to_string()));
let param_refs: Vec<(&str, &str)> = params
.iter()
.map(|(k, v)| (k.as_str(), v.as_str()))
.collect();
self.post_bytes("/api/item/", content, &param_refs)
}
pub fn delete_item(&self, id: i64) -> Result<(), CoreError> {
self.delete(&format!("/api/item/{id}"))
}
/// Add metadata to an existing item.
pub fn post_metadata(
&self,
id: i64,
metadata: &HashMap<String, String>,
) -> Result<(), CoreError> {
let url = self.url(&format!("/api/item/{id}/meta"));
let mut req = self.agent.post(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
req = req.header("Content-Type", "application/json");
let body = serde_json::to_vec(metadata)
.map_err(|e| CoreError::Other(anyhow::anyhow!("Failed to serialize metadata: {e}")))?;
let mut cursor = std::io::Cursor::new(body);
self.handle_error(req.send(ureq::SendBody::from_reader(&mut cursor)))?;
Ok(())
}
pub fn get_item_content_raw(&self, id: i64) -> Result<(Vec<u8>, String), CoreError> {
let url = format!(
"{}?decompress=false",
self.url(&format!("/api/item/{id}/content"))
);
let mut req = self.agent.get(&url);
if let Some(ref password) = self.password {
req = req.header("Authorization", &format!("Bearer {password}"));
}
let response = self.handle_error(req.call())?;
let compression = response
.headers()
.get("X-Keep-Compression")
.and_then(|v| v.to_str().ok())
.unwrap_or("none")
.to_string();
let mut body = response.into_body();
let bytes = body
.read_to_vec()
.map_err(|e| CoreError::Other(anyhow::anyhow!("{}", e)))?;
Ok((bytes, compression))
}
pub fn diff_items(&self, id_a: i64, id_b: i64) -> Result<Vec<String>, CoreError> {
#[derive(serde::Deserialize)]
struct ApiResponse {
data: Option<Vec<String>>,
}
let params = [("id_a", id_a.to_string()), ("id_b", id_b.to_string())];
let param_refs: Vec<(&str, &str)> = params.iter().map(|(k, v)| (*k, v.as_str())).collect();
let response: ApiResponse = self.get_json_with_query("/api/diff", &param_refs)?;
Ok(response.data.unwrap_or_default())
}
}

130
src/common/binary_detection.rs
View File

@@ -1,130 +0,0 @@
use crate::services::async_item_service::AsyncItemService;
use crate::services::error::CoreError;
use axum::http::StatusCode;
use std::collections::HashMap;
/// Check if content is binary when allow_binary is false
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service
/// * `item_id` - The ID of the item to check
/// * `metadata` - Metadata associated with the item
/// * `allow_binary` - Whether binary content is allowed
///
/// # Returns
///
/// * `Result<(), StatusCode>` -
/// * `Ok(())` if binary content is allowed or content is not binary
/// * `Err(StatusCode::BAD_REQUEST)` if binary content is not allowed and content is binary
/// Check if content is binary when allow_binary is false
///
/// Validates whether binary content is permitted for the item. If not allowed and content
/// is detected as binary, returns a bad request status. Uses metadata or streams content
/// for detection if needed.
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service for content access.
/// * `item_id` - The ID of the item to check.
/// * `metadata` - Metadata associated with the item (checked for "text" key).
/// * `allow_binary` - Whether binary content is allowed (bypasses check if true).
///
/// # Returns
///
/// * `Result<(), StatusCode>` -
/// * `Ok(())` if binary content is allowed or content is not binary.
/// * `Err(StatusCode::BAD_REQUEST)` if binary content is not allowed and content is binary.
///
/// # Errors
///
/// Propagates `StatusCode` for validation failures.
///
/// # Examples
///
/// ```
/// // If allow_binary = false and content is text
/// check_binary_content_allowed(&service, 1, &metadata, false)?;
/// // Succeeds
///
/// // If allow_binary = false and content is binary
/// // Returns Err(StatusCode::BAD_REQUEST)
/// ```
pub async fn check_binary_content_allowed(
item_service: &AsyncItemService,
item_id: i64,
metadata: &HashMap<String, String>,
allow_binary: bool,
) -> Result<(), StatusCode> {
if !allow_binary {
let is_binary = is_content_binary(item_service, item_id, metadata).await?;
if is_binary {
return Err(StatusCode::BAD_REQUEST);
}
}
Ok(())
}
/// Helper function to determine if content is binary
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service
/// * `item_id` - The ID of the item to check
/// * `metadata` - Metadata associated with the item
///
/// # Returns
///
/// * `Result<bool, StatusCode>` -
/// * `Ok(true)` if content is binary
/// * `Ok(false)` if content is text
/// * `Err(StatusCode)` if an error occurs during checking
/// Helper function to determine if content is binary
///
/// Checks existing "text" metadata first; if absent or unset, streams and analyzes
/// the content to detect binary nature. Logs warnings on detection failures.
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service for content access.
/// * `item_id` - The ID of the item to check.
/// * `metadata` - Metadata associated with the item (checked for "text" key).
///
/// # Returns
///
/// * `Result<bool, StatusCode>` -
/// * `Ok(true)` if content is binary.
/// * `Ok(false)` if content is text.
/// * `Err(StatusCode)` if an error occurs during checking (e.g., INTERNAL_SERVER_ERROR).
///
/// # Errors
///
/// * `StatusCode::INTERNAL_SERVER_ERROR` if content access fails.
///
/// # Examples
///
/// ```
/// let is_bin = is_content_binary(&service, 1, &metadata).await?;
/// assert!(is_bin == false); // For text content
/// ```
pub async fn is_content_binary(
item_service: &AsyncItemService,
item_id: i64,
metadata: &HashMap<String, String>,
) -> Result<bool, StatusCode> {
if let Some(text_val) = metadata.get("text") {
Ok(text_val == "false")
} else {
// If text metadata isn't set, we need to check the content using streaming approach
match item_service.get_item_content_info_streaming(
item_id,
None
).await {
Ok((_, _, is_binary)) => Ok(is_binary),
Err(e) => {
log::warn!("Failed to get content info for binary check for item {}: {}", item_id, e);
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
}
}

4
src/compression_engine/gzip.rs
View File

@@ -11,12 +11,12 @@ use std::io::{Read, Write};
#[cfg(feature = "gzip")]
use std::path::PathBuf;
#[cfg(feature = "gzip")]
use flate2::Compression;
#[cfg(feature = "gzip")]
use flate2::read::GzDecoder;
#[cfg(feature = "gzip")]
use flate2::write::GzEncoder;
#[cfg(feature = "gzip")]
use flate2::Compression;
#[cfg(feature = "gzip")]
use crate::compression_engine::CompressionEngine;

5
src/compression_engine/mod.rs
View File

@@ -1,4 +1,4 @@
use anyhow::{anyhow, Result};
use anyhow::{Result, anyhow};
use std::io;
use std::io::{Read, Write};
use std::path::PathBuf;
@@ -28,8 +28,7 @@ use crate::compression_engine::program::CompressionEngineProgram;
///
/// # Examples
///
/// ```
/// use keep::compression_engine::CompressionType;
/// ```ignore
/// assert_eq!(CompressionType::GZip.to_string(), "gzip");
/// ```
#[derive(Debug, Eq, PartialEq, Clone, EnumIter, Display, EnumString, enum_map::Enum)]

2
src/compression_engine/program.rs
View File

@@ -1,4 +1,4 @@
use anyhow::{anyhow, Context, Result};
use anyhow::{Context, Result, anyhow};
use log::*;
use std::fs::File;
use std::io::{Read, Write};

49
src/config.rs
View File

@@ -146,6 +146,8 @@ pub struct ServerConfig {
pub password_file: Option<PathBuf>,
pub password: Option<String>,
pub password_hash: Option<String>,
pub cert_file: Option<PathBuf>,
pub key_file: Option<PathBuf>,
}
#[derive(Debug, Clone, Deserialize, Serialize)]
@@ -153,6 +155,12 @@ pub struct CompressionPluginConfig {
pub name: String,
}
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct ClientConfig {
pub url: Option<String>,
pub password: Option<String>,
}
#[derive(Debug, Clone, Deserialize, Serialize)]
#[cfg_attr(feature = "server", derive(utoipa::ToSchema))]
pub struct MetaPluginConfig {
@@ -184,6 +192,12 @@ pub struct Settings {
pub server: Option<ServerConfig>,
pub compression_plugin: Option<CompressionPluginConfig>,
pub meta_plugins: Option<Vec<MetaPluginConfig>>,
pub client: Option<ClientConfig>,
// Non-serializable fields populated from CLI args
#[serde(skip)]
pub client_url: Option<String>,
#[serde(skip)]
pub client_password: Option<String>,
}
impl Settings {
@@ -275,6 +289,18 @@ impl Settings {
config_builder = config_builder.set_override("server.port", server_port)?;
}
#[cfg(feature = "tls")]
if let Some(server_cert) = &args.mode.server_cert {
config_builder = config_builder
.set_override("server.cert_file", server_cert.to_string_lossy().as_ref())?;
}
#[cfg(feature = "tls")]
if let Some(server_key) = &args.mode.server_key {
config_builder = config_builder
.set_override("server.key_file", server_key.to_string_lossy().as_ref())?;
}
if let Some(compression) = &args.item.compression {
config_builder =
config_builder.set_override("compression_plugin.name", compression.as_str())?;
@@ -394,6 +420,21 @@ impl Settings {
settings.dir = default_dir;
}
// Populate client settings from CLI args and config
#[cfg(feature = "client")]
{
settings.client_url = args
.options
.client_url
.clone()
.or_else(|| settings.client.as_ref().and_then(|c| c.url.clone()));
settings.client_password = args
.options
.client_password
.clone()
.or_else(|| settings.client.as_ref().and_then(|c| c.password.clone()));
}
debug!("CONFIG: Final settings: {settings:?}");
Ok(settings)
}
@@ -453,6 +494,14 @@ impl Settings {
self.server.as_ref().and_then(|s| s.port)
}
pub fn server_cert_file(&self) -> Option<PathBuf> {
self.server.as_ref().and_then(|s| s.cert_file.clone())
}
pub fn server_key_file(&self) -> Option<PathBuf> {
self.server.as_ref().and_then(|s| s.key_file.clone())
}
pub fn compression(&self) -> Option<String> {
self.compression_plugin.as_ref().map(|c| c.name.clone())
}

400
src/db.rs
View File

@@ -1,4 +1,4 @@
use anyhow::{Context, Error, Result};
use anyhow::{Context, Error, Result, anyhow};
use chrono::prelude::*;
use lazy_static::lazy_static;
use log::*;
@@ -37,11 +37,11 @@ Automatic schema migrations are applied on database open using
# Usage
Open a connection:
```
```ignore
let conn = db::open(PathBuf::from("keep.db"))?;
```
Insert an item:
```
```ignore
let item = db::Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
let id = db::insert_item(&conn, item)?;
```
@@ -159,8 +159,14 @@ pub struct Meta {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// # Ok(())
/// # }
/// ```
pub fn open(path: PathBuf) -> Result<Connection, Error> {
debug!("DB: Opening file: {path:?}");
@@ -203,6 +209,13 @@ pub fn open(path: PathBuf) -> Result<Connection, Error> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item {
/// id: None,
/// ts: Utc::now(),
@@ -211,6 +224,8 @@ pub fn open(path: PathBuf) -> Result<Connection, Error> {
/// };
/// let id = db::insert_item(&conn, item)?;
/// assert!(id > 0);
/// # Ok(())
/// # }
/// ```
pub fn insert_item(conn: &Connection, item: Item) -> Result<i64> {
debug!("DB: Inserting item: {item:?}");
@@ -241,9 +256,18 @@ pub fn insert_item(conn: &Connection, item: Item) -> Result<i64> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use keep::compression_engine::CompressionType;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let compression = CompressionType::LZ4;
/// let item = db::create_item(&conn, compression)?;
/// assert!(item.id.is_some());
/// # Ok(())
/// # }
/// ```
pub fn create_item(
conn: &Connection,
@@ -284,7 +308,18 @@ pub fn create_item(
/// # Examples
///
/// ```
/// db::add_tag(&conn, 1, "important")?;
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// db::add_tag(&conn, item_id, "important")?;
/// # Ok(())
/// # }
/// ```
pub fn add_tag(conn: &Connection, item_id: i64, tag_name: &str) -> Result<()> {
let tag = Tag {
@@ -317,7 +352,18 @@ pub fn add_tag(conn: &Connection, item_id: i64, tag_name: &str) -> Result<()> {
/// # Examples
///
/// ```
/// db::add_meta(&conn, 1, "mime_type", "text/plain")?;
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// db::add_meta(&conn, item_id, "mime_type", "text/plain")?;
/// # Ok(())
/// # }
/// ```
pub fn add_meta(conn: &Connection, item_id: i64, name: &str, value: &str) -> Result<()> {
let meta = Meta {
@@ -349,8 +395,17 @@ pub fn add_meta(conn: &Connection, item_id: i64, name: &str, value: &str) -> Res
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), size: Some(1024), compression: "lz4".to_string(), ts: Utc::now() };
/// db::update_item(&conn, item)?;
/// # Ok(())
/// # }
/// ```
pub fn update_item(conn: &Connection, item: Item) -> Result<()> {
debug!("DB: Updating item: {item:?}");
@@ -382,8 +437,17 @@ pub fn update_item(conn: &Connection, item: Item) -> Result<()> {
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), ..default_item() };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// db::delete_item(&conn, item)?;
/// # Ok(())
/// # }
/// ```
pub fn delete_item(conn: &Connection, item: Item) -> Result<()> {
debug!("DB: Deleting item: {item:?}");
@@ -412,8 +476,16 @@ pub fn delete_item(conn: &Connection, item: Item) -> Result<()> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() };
/// db::query_delete_meta(&conn, meta)?;
/// # Ok(())
/// # }
/// ```
pub fn query_delete_meta(conn: &Connection, meta: Meta) -> Result<()> {
debug!("DB: Deleting meta: {meta:?}");
@@ -445,8 +517,19 @@ pub fn query_delete_meta(conn: &Connection, meta: Meta) -> Result<()> {
/// # Examples
///
/// ```
/// let meta = Meta { id: 1, name: "mime_type".to_string(), value: "text/plain".to_string() };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let meta = Meta { id: item_id, name: "mime_type".to_string(), value: "text/plain".to_string() };
/// db::query_upsert_meta(&conn, meta)?;
/// # Ok(())
/// # }
/// ```
pub fn query_upsert_meta(conn: &Connection, meta: Meta) -> Result<()> {
debug!("DB: Inserting meta: {meta:?}");
@@ -478,41 +561,24 @@ pub fn query_upsert_meta(conn: &Connection, meta: Meta) -> Result<()> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// // Insert new metadata
/// let meta = Meta { id: 1, name: "source".to_string(), value: "cli".to_string() };
/// let meta = Meta { id: item_id, name: "source".to_string(), value: "cli".to_string() };
/// db::store_meta(&conn, meta)?;
///
/// // Delete metadata with empty value
/// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() };
/// db::store_meta(&conn, meta)?;
/// ```
/// Stores a metadata entry, deleting it if the value is empty.
///
/// Handles both insertion/update and deletion based on value presence.
///
/// # Arguments
///
/// * `conn` - Database connection.
/// * `meta` - Metadata entry to store (empty value triggers deletion).
///
/// # Returns
///
/// * `Result<()>` - Success or error if the operation fails.
///
/// # Errors
///
/// * Database errors during insert/update/delete.
///
/// # Examples
///
/// ```
/// // Insert new metadata
/// let meta = Meta { id: 1, name: "source".to_string(), value: "cli".to_string() };
/// db::store_meta(&conn, meta)?;
///
/// // Delete metadata with empty value
/// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() };
/// let meta = Meta { id: item_id, name: "temp".to_string(), value: "".to_string() };
/// db::store_meta(&conn, meta)?;
/// # Ok(())
/// # }
/// ```
pub fn store_meta(conn: &Connection, meta: Meta) -> Result<()> {
if meta.value.is_empty() {
@@ -544,8 +610,19 @@ pub fn store_meta(conn: &Connection, meta: Meta) -> Result<()> {
/// # Examples
///
/// ```
/// let tag = Tag { id: 1, name: "work".to_string() };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let tag = Tag { id: item_id, name: "work".to_string() };
/// db::insert_tag(&conn, tag)?;
/// # Ok(())
/// # }
/// ```
pub fn insert_tag(conn: &Connection, tag: Tag) -> Result<()> {
debug!("DB: Inserting tag: {tag:?}");
@@ -576,8 +653,17 @@ pub fn insert_tag(conn: &Connection, tag: Tag) -> Result<()> {
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), .. };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// db::delete_item_tags(&conn, item)?;
/// # Ok(())
/// # }
/// ```
pub fn delete_item_tags(conn: &Connection, item: Item) -> Result<()> {
debug!("DB: Deleting all item tags: {item:?}");
@@ -607,24 +693,38 @@ pub fn delete_item_tags(conn: &Connection, item: Item) -> Result<()> {
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), .. };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let item = Item { id: Some(item_id), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let tags = vec!["project_a".to_string(), "urgent".to_string()];
/// db::set_item_tags(&conn, item, &tags)?;
/// # Ok(())
/// # }
/// ```
pub fn set_item_tags(conn: &Connection, item: Item, tags: &Vec<String>) -> Result<()> {
debug!("DB: Setting tags for item: {item:?} ?{tags:?}");
delete_item_tags(conn, item.clone())?;
let item_id = item.id.unwrap();
let item_id = item
.id
.ok_or_else(|| anyhow!("Item ID is required for set_item_tags"))?;
let tx = conn.unchecked_transaction()?;
delete_item_tags(&tx, item)?;
for tag_name in tags {
insert_tag(
conn,
&tx,
Tag {
id: item_id,
name: tag_name.to_string(),
},
)?;
}
tx.commit()?;
Ok(())
}
@@ -647,8 +747,16 @@ pub fn set_item_tags(conn: &Connection, item: Item, tags: &Vec<String>) -> Resul
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let all_items = db::query_all_items(&conn)?;
/// assert!(all_items.len() >= 0);
/// # Ok(())
/// # }
/// ```
pub fn query_all_items(conn: &Connection) -> Result<Vec<Item>> {
debug!("DB: Querying all items");
@@ -691,8 +799,16 @@ pub fn query_all_items(conn: &Connection) -> Result<Vec<Item>> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let tags = vec!["work".to_string(), "urgent".to_string()];
/// let tagged_items = db::query_tagged_items(&conn, &tags)?;
/// # Ok(())
/// # }
/// ```
pub fn query_tagged_items<'a>(conn: &'a Connection, tags: &'a Vec<String>) -> Result<Vec<Item>> {
debug!("DB: Querying tagged items: {tags:?}");
@@ -751,7 +867,15 @@ pub fn query_tagged_items<'a>(conn: &'a Connection, tags: &'a Vec<String>) -> Re
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let items = db::get_items(&conn)?;
/// # Ok(())
/// # }
/// ```
pub fn get_items(conn: &Connection) -> Result<Vec<Item>> {
debug!("DB: Getting all items");
@@ -780,9 +904,18 @@ pub fn get_items(conn: &Connection) -> Result<Vec<Item>> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::collections::HashMap;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let tags = vec!["project".to_string()];
/// let meta = HashMap::from([("status".to_string(), "active".to_string())]);
/// let matching = db::get_items_matching(&conn, &tags, &meta)?;
/// # Ok(())
/// # }
/// ```
pub fn get_items_matching(
conn: &Connection,
@@ -801,44 +934,35 @@ pub fn get_items_matching(
Ok(items)
} else {
debug!("DB: Filtering on meta");
let mut filtered_items: Vec<Item> = Vec::new();
for item in items.iter() {
let mut item_ok = true;
let mut item_meta: HashMap<String, String> = HashMap::new();
for meta in get_item_meta(conn, item)? {
item_meta.insert(meta.name, meta.value);
}
debug!("DB: Matching: {item:?}: {item_meta:?}");
for (k, v) in meta.iter() {
match item_meta.get(k) {
Some(value) => item_ok = v.eq(value),
None => item_ok = false,
}
if !item_ok {
break;
}
}
if item_ok {
filtered_items.push(item.clone());
}
}
let item_ids: Vec<i64> = items.iter().filter_map(|i| i.id).collect();
let meta_map = get_meta_for_items(conn, &item_ids)?;
let filtered_items: Vec<Item> = items
.into_iter()
.filter(|item| {
let item_id = match item.id {
Some(id) => id,
None => return false,
};
let item_meta = match meta_map.get(&item_id) {
Some(m) => m,
None => return false,
};
meta.iter().all(|(k, v)| item_meta.get(k) == Some(v))
})
.collect();
Ok(filtered_items)
}
}
/// Gets a single item matching specified tags.
/// Gets a single item matching specified tags and metadata.
///
/// Returns the most recent item matching all tags (ignores metadata).
/// Returns the most recent item matching all tags and metadata.
///
/// # Arguments
///
/// * `conn` - Database connection.
/// * `tags` - Vector of tag names to match (all must match).
/// * `_meta` - Unused metadata parameter (for API consistency).
/// * `meta` - HashMap of metadata key-value pairs to match (exact match).
///
/// # Returns
///
@@ -851,51 +975,26 @@ pub fn get_items_matching(
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::collections::HashMap;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let tags = vec!["latest".to_string()];
/// let item = db::get_item_matching(&conn, &tags, &HashMap::new())?;
/// # Ok(())
/// # }
/// ```
pub fn get_item_matching(
conn: &Connection,
tags: &Vec<String>,
_meta: &HashMap<String, String>,
meta: &HashMap<String, String>,
) -> Result<Option<Item>> {
debug!("DB: Get item matching tags: {tags:?}");
let mut statement = conn
.prepare_cached(
"
SELECT items.id,
items.ts,
items.size,
items.compression,
count(sel.id) as score
FROM items,
(SELECT tags.id FROM tags WHERE tags.name IN rarray(?1)) as sel
WHERE items.id = sel.id
GROUP BY items.id
HAVING score = ?2
ORDER BY items.id DESC
LIMIT 1",
)
.context("Problem preparing SQL statement")?;
let tags_values: Vec<rusqlite::types::Value> = tags
.iter()
.map(|s| rusqlite::types::Value::from(s.clone()))
.collect();
let tags_ptr = Rc::new(tags_values);
let mut rows = statement.query(params![&tags_ptr, &tags.len()])?;
match rows.next()? {
Some(row) => Ok(Some(Item {
id: row.get(0)?,
ts: row.get(1)?,
size: row.get(2)?,
compression: row.get(3)?,
})),
None => Ok(None),
}
debug!("DB: Get item matching tags: {tags:?}, meta: {meta:?}");
let items = get_items_matching(conn, tags, meta)?;
Ok(items.into_iter().last())
}
/// Gets an item by its ID.
@@ -918,8 +1017,19 @@ pub fn get_item_matching(
/// # Examples
///
/// ```
/// let item = db::get_item(&conn, 1)?;
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let item = db::get_item(&conn, item_id)?;
/// assert!(item.is_some());
/// # Ok(())
/// # }
/// ```
pub fn get_item(conn: &Connection, item_id: i64) -> Result<Option<Item>> {
debug!("DB: Getting item {item_id:?}");
@@ -964,7 +1074,15 @@ pub fn get_item(conn: &Connection, item_id: i64) -> Result<Option<Item>> {
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let latest = db::get_item_last(&conn)?;
/// # Ok(())
/// # }
/// ```
pub fn get_item_last(conn: &Connection) -> Result<Option<Item>> {
debug!("DB: Getting last item");
@@ -1011,8 +1129,17 @@ pub fn get_item_last(conn: &Connection) -> Result<Option<Item>> {
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), .. };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let tags = db::get_item_tags(&conn, &item)?;
/// # Ok(())
/// # }
/// ```
pub fn get_item_tags(conn: &Connection, item: &Item) -> Result<Vec<Tag>> {
debug!("DB: Getting tags for item: {item:?}");
@@ -1053,8 +1180,17 @@ pub fn get_item_tags(conn: &Connection, item: &Item) -> Result<Vec<Tag>> {
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), .. };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let meta = db::get_item_meta(&conn, &item)?;
/// # Ok(())
/// # }
/// ```
pub fn get_item_meta(conn: &Connection, item: &Item) -> Result<Vec<Meta>> {
debug!("DB: Getting item meta: {item:?}");
@@ -1097,8 +1233,17 @@ pub fn get_item_meta(conn: &Connection, item: &Item) -> Result<Vec<Meta>> {
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), .. };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let meta = db::get_item_meta_name(&conn, &item, "mime_type".to_string())?;
/// # Ok(())
/// # }
/// ```
pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Result<Option<Meta>> {
debug!("DB: Getting item meta name: {item:?} {name:?}");
@@ -1138,8 +1283,17 @@ pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Resul
/// # Examples
///
/// ```
/// let item = Item { id: Some(1), .. };
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let value = db::get_item_meta_value(&conn, &item, "source".to_string())?;
/// # Ok(())
/// # }
/// ```
pub fn get_item_meta_value(conn: &Connection, item: &Item, name: String) -> Result<Option<String>> {
debug!("DB: Getting item meta value: {item:?} {name:?}");
@@ -1174,8 +1328,16 @@ pub fn get_item_meta_value(conn: &Connection, item: &Item, name: String) -> Resu
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let ids = vec![1, 2, 3];
/// let tags_map = db::get_tags_for_items(&conn, &ids)?;
/// # Ok(())
/// # }
/// ```
pub fn get_tags_for_items(
conn: &Connection,
@@ -1233,8 +1395,16 @@ pub fn get_tags_for_items(
/// # Examples
///
/// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let ids = vec![1, 2, 3];
/// let meta_map = db::get_meta_for_items(&conn, &ids)?;
/// # Ok(())
/// # }
/// ```
pub fn get_meta_for_items(
conn: &Connection,

12
src/filter_plugin/grep.rs
View File

@@ -34,7 +34,9 @@ pub struct GrepFilter {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::GrepFilter;
/// let filter = GrepFilter::new("error|warn".to_string())?;
/// # Ok::<(), std::io::Error>(())
/// ```
impl GrepFilter {
pub fn new(pattern: String) -> Result<Self> {
@@ -65,7 +67,13 @@ impl GrepFilter {
/// # Examples
///
/// ```
/// # use std::io::{Read, Write, Cursor};
/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
/// # let mut filter = GrepFilter::new("error".to_string())?;
/// let mut input: &mut dyn Read = &mut Cursor::new(b"error: something failed\nok: all good\n");
/// let mut output = Vec::new();
/// filter.filter(&mut input, &mut output)?;
/// # Ok::<(), std::io::Error>(())
/// ```
impl FilterPlugin for GrepFilter {
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -90,6 +98,8 @@ impl FilterPlugin for GrepFilter {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
/// let filter = GrepFilter::new("test".to_string()).unwrap();
/// let cloned = filter.clone_box();
/// ```
fn clone_box(&self) -> Box<dyn FilterPlugin> {
@@ -109,6 +119,8 @@ impl FilterPlugin for GrepFilter {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
/// let filter = GrepFilter::new("test".to_string()).unwrap();
/// let opts = filter.options();
/// assert_eq!(opts.len(), 1);
/// assert!(opts[0].required);

62
src/filter_plugin/head.rs
View File

@@ -37,8 +37,8 @@ impl HeadBytesFilter {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::HeadBytesFilter;
/// let filter = HeadBytesFilter::new(1024);
/// assert_eq!(filter.remaining, 1024);
/// ```
pub fn new(count: usize) -> Self {
Self { remaining: count }
@@ -66,8 +66,14 @@ impl HeadBytesFilter {
/// # Examples
///
/// ```
/// // Assuming a filter chain with head_bytes(5)
/// // Input "Hello World" becomes "Hello"
/// # use std::io::{Read, Write, Cursor};
/// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
/// # let mut filter = HeadBytesFilter::new(5);
/// let mut input: &mut dyn Read = &mut Cursor::new(b"Hello World");
/// let mut output = Vec::new();
/// filter.filter(&mut input, &mut output)?;
/// assert_eq!(output, b"Hello");
/// # Ok::<(), std::io::Error>(())
/// ```
impl FilterPlugin for HeadBytesFilter {
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -95,6 +101,14 @@ impl FilterPlugin for HeadBytesFilter {
/// # Returns
///
/// A new `Box<dyn FilterPlugin>` clone.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
/// let filter = HeadBytesFilter::new(100);
/// let cloned = filter.clone_box();
/// ```
fn clone_box(&self) -> Box<dyn FilterPlugin> {
Box::new(Self {
remaining: self.remaining,
@@ -108,6 +122,17 @@ impl FilterPlugin for HeadBytesFilter {
/// # Returns
///
/// Vector of `FilterOption` describing parameters.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
/// let filter = HeadBytesFilter::new(100);
/// let opts = filter.options();
/// assert_eq!(opts.len(), 1);
/// assert_eq!(opts[0].name, "count");
/// assert!(opts[0].required);
/// ```
fn options(&self) -> Vec<FilterOption> {
vec![FilterOption {
name: "count".to_string(),
@@ -144,8 +169,8 @@ impl HeadLinesFilter {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::HeadLinesFilter;
/// let filter = HeadLinesFilter::new(3);
/// assert_eq!(filter.remaining, 3);
/// ```
pub fn new(count: usize) -> Self {
Self { remaining: count }
@@ -172,8 +197,14 @@ impl HeadLinesFilter {
/// # Examples
///
/// ```
/// // Assuming a filter chain with head_lines(2)
/// // Input: "Line1\nLine2\nLine3" becomes "Line1\nLine2\n"
/// # use std::io::{Read, Write, Cursor};
/// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
/// # let mut filter = HeadLinesFilter::new(2);
/// let mut input: &mut dyn Read = &mut Cursor::new(b"Line1\nLine2\nLine3\n");
/// let mut output = Vec::new();
/// filter.filter(&mut input, &mut output)?;
/// assert_eq!(output, b"Line1\nLine2\n");
/// # Ok::<(), std::io::Error>(())
/// ```
impl FilterPlugin for HeadLinesFilter {
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -200,6 +231,14 @@ impl FilterPlugin for HeadLinesFilter {
/// # Returns
///
/// A new `Box<dyn FilterPlugin>` clone.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
/// let filter = HeadLinesFilter::new(5);
/// let cloned = filter.clone_box();
/// ```
fn clone_box(&self) -> Box<dyn FilterPlugin> {
Box::new(Self {
remaining: self.remaining,
@@ -213,6 +252,17 @@ impl FilterPlugin for HeadLinesFilter {
/// # Returns
///
/// Vector of `FilterOption` describing parameters.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
/// let filter = HeadLinesFilter::new(5);
/// let opts = filter.options();
/// assert_eq!(opts.len(), 1);
/// assert_eq!(opts[0].name, "count");
/// assert!(opts[0].required);
/// ```
fn options(&self) -> Vec<FilterOption> {
vec![FilterOption {
name: "count".to_string(),

71
src/filter_plugin/mod.rs
View File

@@ -14,8 +14,13 @@ pub mod grep;
/// Parse a filter string and apply to a reader:
///
/// ```
/// let chain = parse_filter_string("head_lines(10)|grep(pattern=error)")?;
/// chain.filter(&mut reader, &mut writer)?;
/// # use std::io::{Read, Write};
/// # use keep::filter_plugin::parse_filter_string;
/// let mut chain = parse_filter_string("head_lines(10)|grep(pattern=error)")?;
/// # let mut reader: &mut dyn Read = &mut std::io::empty();
/// # let mut writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut reader, &mut writer)?;
/// # Ok::<(), std::io::Error>(())
/// ```
pub mod head;
pub mod skip;
@@ -62,11 +67,20 @@ pub struct FilterOption {
/// # Examples
///
/// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterPlugin, FilterOption};
/// struct MyFilter;
/// impl FilterPlugin for MyFilter {
/// fn filter(&mut self, reader: Box<&mut dyn Read>, writer: Box<&mut dyn Write>) -> Result<()> {
/// fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
/// // Implementation
/// Ok(())
/// }
/// fn clone_box(&self) -> Box<dyn FilterPlugin> {
/// Box::new(MyFilter)
/// }
/// fn options(&self) -> Vec<FilterOption> {
/// vec![]
/// }
/// // ...
/// }
/// ```
pub trait FilterPlugin: Send {
@@ -77,8 +91,8 @@ pub trait FilterPlugin: Send {
///
/// # Arguments
///
/// * `reader` - A boxed mutable reference to the input reader providing the data to filter.
/// * `writer` - A boxed mutable reference to the output writer where the processed data is written.
/// * `reader` - A mutable reference to the input reader providing the data to filter.
/// * `writer` - A mutable reference to the output writer where the processed data is written.
///
/// # Returns
///
@@ -87,18 +101,27 @@ pub trait FilterPlugin: Send {
/// # Examples
///
/// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterPlugin, FilterOption};
/// struct MyFilter;
/// impl FilterPlugin for MyFilter {
/// fn filter(&mut self, reader: Box<&mut dyn Read>, writer: Box<&mut dyn Write>) -> Result<()> {
/// fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
/// // Read and filter data
/// let mut buf = [0; 1024];
/// while let Ok(n) = reader.as_mut().read(&mut buf) {
/// loop {
/// let n = reader.read(&mut buf)?;
/// if n == 0 { break; }
/// // Apply filter logic to buf[0..n]
/// writer.as_mut().write_all(&buf[0..n])?;
/// writer.write_all(&buf[0..n])?;
/// }
/// Ok(())
/// }
/// // ... other methods
/// fn clone_box(&self) -> Box<dyn FilterPlugin> {
/// Box::new(MyFilter)
/// }
/// fn options(&self) -> Vec<FilterOption> {
/// vec![]
/// }
/// }
/// ```
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -117,8 +140,9 @@ pub trait FilterPlugin: Send {
/// # Examples
///
/// ```
/// fn clone_box(&self) -> Box<dyn FilterPlugin> {
/// Box::new(self.clone())
/// # use keep::filter_plugin::FilterPlugin;
/// fn example_clone_box(filter: &dyn FilterPlugin) -> Box<dyn FilterPlugin> {
/// filter.clone_box()
/// }
/// ```
fn clone_box(&self) -> Box<dyn FilterPlugin>;
@@ -134,7 +158,8 @@ pub trait FilterPlugin: Send {
/// # Examples
///
/// ```
/// fn options(&self) -> Vec<FilterOption> {
/// # use keep::filter_plugin::FilterOption;
/// fn example_options() -> Vec<FilterOption> {
/// vec![
/// FilterOption {
/// name: "pattern".to_string(),
@@ -191,9 +216,14 @@ pub struct FilterChain {
/// # Examples
///
/// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterChain, HeadLinesFilter};
/// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(HeadLinesFilter::new(10)));
/// chain.filter(&mut reader, &mut writer)?;
/// # let mut reader: &mut dyn Read = &mut std::io::empty();
/// # let mut writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut reader, &mut writer)?;
/// # Ok::<(), std::io::Error>(())
/// ```
impl Clone for FilterChain {
/// Clones this filter chain.
@@ -237,8 +267,9 @@ impl FilterChain {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::FilterChain;
/// let chain = FilterChain::new();
/// assert!(chain.plugins.is_empty());
/// // Chain starts empty
/// ```
pub fn new() -> Self {
Self {
@@ -257,8 +288,9 @@ impl FilterChain {
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterChain, GrepFilter};
/// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(GrepFilter::new("error".to_string())));
/// chain.add_plugin(Box::new(GrepFilter::new("error".to_string()).unwrap()));
/// ```
pub fn add_plugin(&mut self, plugin: Box<dyn FilterPlugin>) {
self.plugins.push(plugin);
@@ -281,9 +313,14 @@ impl FilterChain {
/// # Examples
///
/// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterChain, HeadBytesFilter};
/// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(HeadBytesFilter::new(100)));
/// chain.filter(&mut input_reader, &mut output_writer)?;
/// # let mut input_reader: &mut dyn Read = &mut std::io::empty();
/// # let mut output_writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut input_reader, &mut output_writer)?;
/// # Ok::<(), std::io::Error>(())
/// ```
pub fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
if self.plugins.is_empty() {

6
src/lib.rs
View File

@@ -18,7 +18,8 @@
//! ```
//!
//! ```rust
//! use keep::Args;
//! # use keep::Args;
//! # use clap::Parser;
//! let args = Args::parse();
//! ```
//!
@@ -39,6 +40,9 @@ pub mod meta_plugin;
pub mod modes;
pub mod services;
#[cfg(feature = "client")]
pub mod client;
// Re-export Args struct for library usage
pub use args::Args;
// Re-export PIPESIZE constant

78
src/main.rs
View File

@@ -168,6 +168,68 @@ fn main() -> Result<(), Error> {
debug!("MAIN: mode: {mode:?}");
debug!("MAIN: settings: {settings:?}");
// Parse filter chain early for better error reporting
let filter_chain = if let Some(filter_str) = &args.item.filters {
match keep::filter_plugin::parse_filter_string(filter_str) {
Ok(chain) => Some(chain),
Err(e) => {
cmd.error(
ErrorKind::InvalidValue,
format!("Invalid filter string: {e}"),
)
.exit();
}
}
} else {
None
};
// Check for client mode
#[cfg(feature = "client")]
{
if let Some(ref client_url) = settings.client_url {
let client =
keep::client::KeepClient::new(client_url, settings.client_password.clone())?;
return match mode {
KeepModes::Save => {
let metadata = std::collections::HashMap::new();
keep::modes::client::save::mode(&client, &mut cmd, &settings, tags, metadata)
}
KeepModes::Get => keep::modes::client::get::mode(
&client,
&mut cmd,
&settings,
ids,
tags,
filter_chain,
),
KeepModes::List => {
keep::modes::client::list::mode(&client, &mut cmd, &settings, tags)
}
KeepModes::Delete => {
keep::modes::client::delete::mode(&client, &mut cmd, &settings, ids)
}
KeepModes::Info => {
keep::modes::client::info::mode(&client, &mut cmd, &settings, ids, tags)
}
KeepModes::Diff => {
keep::modes::client::diff::mode(&client, &mut cmd, &settings, ids)
}
KeepModes::Status => {
keep::modes::client::status::mode(&client, &mut cmd, &settings)
}
_ => {
cmd.error(
ErrorKind::InvalidValue,
format!("Mode {mode:?} is not supported in client mode"),
)
.exit();
}
};
}
}
unsafe {
libc::umask(0o077);
}
@@ -186,22 +248,6 @@ fn main() -> Result<(), Error> {
// Initialize database
let mut conn = db::open(db_path.clone())?;
// Parse filter chain early for better error reporting
let filter_chain = if let Some(filter_str) = &args.item.filters {
match keep::filter_plugin::parse_filter_string(filter_str) {
Ok(chain) => Some(chain),
Err(e) => {
cmd.error(
ErrorKind::InvalidValue,
format!("Invalid filter string: {e}"),
)
.exit();
}
}
} else {
None
};
match mode {
KeepModes::Save => {
modes::save::mode_save(&mut cmd, &settings, ids, tags, &mut conn, data_path)

3
src/meta_plugin/exec.rs
View File

@@ -66,7 +66,8 @@ impl MetaPluginExec {
/// # Examples
///
/// ```
/// let plugin = MetaPluginExec::new("date", &[], "date_output", false, None, None);
/// # use keep::meta_plugin::MetaPluginExec;
/// let plugin = MetaPluginExec::new("date", &[], "date_output".to_string(), false, None, None);
/// ```
pub fn new(
program: &str,

1
src/meta_plugin/read_rate.rs
View File

@@ -40,6 +40,7 @@ impl ReadRateMetaPlugin {
/// # Examples
///
/// ```
/// # use keep::meta_plugin::{ReadRateMetaPlugin, MetaPlugin};
/// let plugin = ReadRateMetaPlugin::new(None, None);
/// assert!(!plugin.is_finalized());
/// ```

2
src/meta_plugin/shell.rs
View File

@@ -31,6 +31,7 @@ impl ShellMetaPlugin {
/// # Examples
///
/// ```
/// # use keep::meta_plugin::ShellMetaPlugin;
/// let plugin = ShellMetaPlugin::new(None, None);
/// ```
pub fn new(
@@ -141,6 +142,7 @@ impl MetaPlugin for ShellMetaPlugin {
/// # Examples
///
/// ```
/// # use keep::meta_plugin::{ShellMetaPlugin, MetaPlugin};
/// let mut plugin = ShellMetaPlugin::new(None, None);
/// let response = plugin.initialize();
/// assert!(response.is_finalized);

2
src/meta_plugin/text.rs
View File

@@ -1,5 +1,5 @@
use crate::common::is_binary::is_binary;
use crate::common::PIPESIZE;
use crate::common::is_binary::is_binary;
use crate::meta_plugin::{MetaPlugin, MetaPluginResponse, MetaPluginType};
#[derive(Debug, Clone)]

21
src/modes/client/delete.rs Normal file
View File

@@ -0,0 +1,21 @@
use crate::client::KeepClient;
use clap::Command;
use log::debug;
pub fn mode(
client: &KeepClient,
_cmd: &mut Command,
settings: &crate::config::Settings,
ids: &[i64],
) -> Result<(), anyhow::Error> {
debug!("CLIENT_DELETE: Deleting items via remote server");
for &id in ids {
client.delete_item(id)?;
if !settings.quiet {
eprintln!("Deleted item {id}");
}
}
Ok(())
}

24
src/modes/client/diff.rs Normal file
View File

@@ -0,0 +1,24 @@
use crate::client::KeepClient;
use clap::Command;
use log::debug;
pub fn mode(
client: &KeepClient,
_cmd: &mut Command,
_settings: &crate::config::Settings,
ids: &[i64],
) -> Result<(), anyhow::Error> {
debug!("CLIENT_DIFF: Getting diff via remote server");
if ids.len() != 2 {
return Err(anyhow::anyhow!("Diff requires exactly 2 item IDs"));
}
let diff_lines = client.diff_items(ids[0], ids[1])?;
for line in &diff_lines {
println!("{line}");
}
Ok(())
}

95
src/modes/client/get.rs Normal file
View File

@@ -0,0 +1,95 @@
use crate::client::KeepClient;
use crate::compression_engine::CompressionType;
use crate::filter_plugin::FilterChain;
use anyhow::Result;
use clap::Command;
use is_terminal::IsTerminal;
use log::debug;
use std::io::{Read, Write};
use std::str::FromStr;
pub fn mode(
client: &KeepClient,
_cmd: &mut Command,
settings: &crate::config::Settings,
ids: &[i64],
tags: &[String],
filter_chain: Option<FilterChain>,
) -> Result<(), anyhow::Error> {
debug!("CLIENT_GET: Getting item via remote server");
// Find the item ID
let item_id = if !ids.is_empty() {
ids[0]
} else if !tags.is_empty() {
// Find item by tags
let items = client.list_items(tags, "newest", 0, 1)?;
if items.is_empty() {
return Err(anyhow::anyhow!("No items found matching tags: {:?}", tags));
}
items[0].id
} else {
// Get latest item
let items = client.list_items(&[], "newest", 0, 1)?;
if items.is_empty() {
return Err(anyhow::anyhow!("No items found"));
}
items[0].id
};
// Get item info to determine compression type
let item_info = client.get_item_info(item_id)?;
// Get raw content from server
let (raw_bytes, compression) = client.get_item_content_raw(item_id)?;
// Check if binary content would be sent to TTY
let is_text = item_info
.metadata
.get("text")
.map(|v| v == "true")
.unwrap_or(false);
if std::io::stdout().is_terminal() && !is_text && !settings.force {
// Check if content is binary
let sample_len = std::cmp::min(raw_bytes.len(), 8192);
if crate::common::is_binary::is_binary(&raw_bytes[..sample_len]) {
return Err(anyhow::anyhow!(
"Refusing to output binary data to a terminal. Use --force to override."
));
}
}
// Decompress locally
let compression_type = CompressionType::from_str(&compression).unwrap_or(CompressionType::None);
let decompressed = match compression_type {
CompressionType::GZip => {
use flate2::read::GzDecoder;
let mut decoder = GzDecoder::new(&raw_bytes[..]);
let mut content = Vec::new();
decoder.read_to_end(&mut content)?;
content
}
CompressionType::LZ4 => lz4_flex::decompress_size_prepended(&raw_bytes)
.map_err(|e| anyhow::anyhow!("LZ4 decompression failed: {}", e))?,
_ => raw_bytes,
};
// Apply filters if present
let output = if let Some(mut chain) = filter_chain {
let mut filtered = Vec::new();
chain.filter(&mut &decompressed[..], &mut filtered)?;
filtered
} else {
decompressed
};
// Stream to stdout
let stdout = std::io::stdout();
let mut stdout = stdout.lock();
stdout.write_all(&output)?;
stdout.flush()?;
Ok(())
}

65
src/modes/client/info.rs Normal file
View File

@@ -0,0 +1,65 @@
use crate::client::KeepClient;
use crate::modes::common::{OutputFormat, format_size, settings_output_format};
use clap::Command;
use log::debug;
pub fn mode(
client: &KeepClient,
_cmd: &mut Command,
settings: &crate::config::Settings,
ids: &[i64],
tags: &[String],
) -> Result<(), anyhow::Error> {
debug!("CLIENT_INFO: Getting item info via remote server");
let output_format = settings_output_format(settings);
// If tags provided, find matching item first
let item_ids: Vec<i64> = if !tags.is_empty() {
let items = client.list_items(tags, "newest", 0, 1)?;
if items.is_empty() {
return Err(anyhow::anyhow!("No items found matching tags: {:?}", tags));
}
items.into_iter().map(|i| i.id).collect()
} else {
ids.to_vec()
};
for &id in &item_ids {
let item = client.get_item_info(id)?;
match output_format {
OutputFormat::Json => {
println!("{}", serde_json::to_string_pretty(&item)?);
}
OutputFormat::Yaml => {
println!("{}", serde_yaml::to_string(&item)?);
}
OutputFormat::Table => {
use comfy_table::{Table, presets::UTF8_FULL};
let mut table = Table::new();
table.load_preset(UTF8_FULL);
let size_str = item
.size
.map(|s| format_size(s as u64, settings.human_readable))
.unwrap_or_else(|| "N/A".to_string());
table.add_row(vec!["ID".to_string(), item.id.to_string()]);
table.add_row(vec!["Time".to_string(), item.ts.clone()]);
table.add_row(vec!["Size".to_string(), size_str]);
table.add_row(vec!["Compression".to_string(), item.compression.clone()]);
table.add_row(vec!["Tags".to_string(), item.tags.join(", ")]);
for (key, value) in &item.metadata {
table.add_row(vec![format!("Meta: {}", key), value.clone()]);
}
println!("{table}");
}
}
}
Ok(())
}

55
src/modes/client/list.rs Normal file
View File

@@ -0,0 +1,55 @@
use crate::client::KeepClient;
use crate::modes::common::{OutputFormat, format_size, settings_output_format};
use clap::Command;
use log::debug;
pub fn mode(
client: &KeepClient,
_cmd: &mut Command,
settings: &crate::config::Settings,
tags: &[String],
) -> Result<(), anyhow::Error> {
debug!("CLIENT_LIST: Listing items via remote server");
let items = client.list_items(tags, "newest", 0, 100)?;
let output_format = settings_output_format(settings);
match output_format {
OutputFormat::Json => {
println!("{}", serde_json::to_string_pretty(&items)?);
}
OutputFormat::Yaml => {
println!("{}", serde_yaml::to_string(&items)?);
}
OutputFormat::Table => {
use comfy_table::{Table, presets::UTF8_FULL};
let mut table = Table::new();
table.load_preset(UTF8_FULL);
// Header
let headers = ["ID", "Time", "Size", "Compression", "Tags"];
table.set_header(headers.iter().map(|h| h.to_string()).collect::<Vec<_>>());
for item in &items {
let size_str = item
.size
.map(|s| format_size(s as u64, settings.human_readable))
.unwrap_or_default();
table.add_row(vec![
item.id.to_string(),
item.ts.clone(),
size_str,
item.compression.clone(),
item.tags.join(", "),
]);
}
println!("{table}");
}
}
Ok(())
}

7
src/modes/client/mod.rs Normal file
View File

@@ -0,0 +1,7 @@
pub mod delete;
pub mod diff;
pub mod get;
pub mod info;
pub mod list;
pub mod save;
pub mod status;

171
src/modes/client/save.rs Normal file
View File

@@ -0,0 +1,171 @@
use crate::client::{ItemInfo, KeepClient};
use crate::compression_engine::CompressionType;
use crate::config::Settings;
use crate::modes::common::settings_compression_type;
use anyhow::Result;
use clap::Command;
use is_terminal::IsTerminal;
use log::debug;
use sha2::{Digest, Sha256};
use std::collections::HashMap;
use std::io::{Read, Write};
use std::sync::{Arc, Mutex};
/// Streaming save mode for client.
///
/// Uses three threads for true streaming with constant memory:
/// - Reader thread: reads stdin, tees to stdout, computes SHA-256,
/// compresses data, writes to OS pipe
/// - Pipe: zero-copy transfer of compressed bytes between threads
/// - Streamer thread: reads from pipe, streams to server via chunked HTTP
///
/// Memory usage is O(PIPESIZE) regardless of data size.
pub fn mode(
client: &KeepClient,
cmd: &mut Command,
settings: &Settings,
tags: &mut Vec<String>,
metadata: HashMap<String, String>,
) -> Result<(), anyhow::Error> {
debug!("CLIENT_SAVE: Saving item via remote server (streaming)");
if tags.is_empty() {
tags.push("none".to_string());
}
// Determine compression type from settings
let compression_type = settings_compression_type(cmd, settings);
let server_compress = matches!(compression_type, CompressionType::None);
// Create OS pipe for streaming compressed bytes between threads
let (pipe_reader, pipe_writer) = os_pipe::pipe()?;
// Shared state for reader thread results
let shared = Arc::new(Mutex::new((0u64, String::new())));
let shared_reader = Arc::clone(&shared);
// Reader thread: stdin → tee(stdout) → hash → compress → pipe
let compression_type_clone = compression_type.clone();
let reader_handle = std::thread::spawn(move || -> Result<(u64, String)> {
let stdin = std::io::stdin();
let stdout = std::io::stdout();
let mut stdin_lock = stdin.lock();
let mut stdout_lock = stdout.lock();
let mut hasher = Sha256::new();
let mut total_bytes = 0u64;
let mut buffer = [0u8; 8192];
// Wrap pipe writer with appropriate compression
let mut compressor: Box<dyn Write> = match compression_type_clone {
CompressionType::GZip => {
use flate2::Compression;
use flate2::write::GzEncoder;
Box::new(GzEncoder::new(pipe_writer, Compression::default()))
}
CompressionType::LZ4 => Box::new(lz4_flex::frame::FrameEncoder::new(pipe_writer)),
_ => Box::new(pipe_writer),
};
loop {
let n = stdin_lock.read(&mut buffer)?;
if n == 0 {
break;
}
// Tee to stdout
stdout_lock.write_all(&buffer[..n])?;
// Update hash
hasher.update(&buffer[..n]);
total_bytes += n as u64;
// Compress and write to pipe
compressor.write_all(&buffer[..n])?;
}
// Finalize compression (flushes any buffered compressed data)
drop(compressor);
// Pipe writer is now dropped (inside compressor), signaling EOF to streamer
let digest = format!("{:x}", hasher.finalize());
// Set shared state for main thread
let mut shared = shared_reader.lock().unwrap();
*shared = (total_bytes, digest.clone());
Ok((total_bytes, digest))
});
// Streamer thread: reads compressed bytes from pipe → POST to server
let client_url = client.base_url().to_string();
let client_password = client.password().cloned();
let tags_clone = tags.clone();
let streamer_handle = std::thread::spawn(move || -> Result<ItemInfo> {
let streaming_client = KeepClient::new(&client_url, client_password)?;
let params = [
("compress".to_string(), server_compress.to_string()),
("meta".to_string(), "false".to_string()),
("tags".to_string(), tags_clone.join(",")),
];
let param_refs: Vec<(&str, &str)> = params
.iter()
.map(|(k, v)| (k.as_str(), v.as_str()))
.collect();
let mut reader: Box<dyn Read> = Box::new(pipe_reader);
let item_info = streaming_client.post_stream("/api/item/", &mut reader, &param_refs)?;
Ok(item_info)
});
// Wait for streaming to complete, capture item info
let item_info = streamer_handle
.join()
.map_err(|e| anyhow::anyhow!("Streamer thread panicked: {:?}", e))??;
// Wait for reader thread (should complete quickly after pipe is drained)
reader_handle
.join()
.map_err(|e| anyhow::anyhow!("Reader thread panicked: {:?}", e))??;
// Read results from shared state
let (uncompressed_size, digest) = {
let shared = shared.lock().unwrap();
shared.clone()
};
// Build local metadata and send to server
let mut local_metadata = metadata;
local_metadata.insert("digest_sha256".to_string(), digest);
local_metadata.insert(
"uncompressed_size".to_string(),
uncompressed_size.to_string(),
);
// Add hostname
if let Ok(hostname) = gethostname::gethostname().into_string() {
local_metadata.insert("hostname".to_string(), hostname.clone());
let short = hostname.split('.').next().unwrap_or(&hostname).to_string();
local_metadata.insert("hostname_short".to_string(), short);
}
// Send metadata to server
if !local_metadata.is_empty() {
client.post_metadata(item_info.id, &local_metadata)?;
}
// Print status to stderr
if !settings.quiet {
if std::io::stderr().is_terminal() {
eprintln!("KEEP: New item (streaming) tags: {}", tags.join(" "));
} else {
eprintln!("KEEP: New item (streaming) tags: {tags:?}");
}
}
debug!("CLIENT_SAVE: Streaming complete, {uncompressed_size} bytes uncompressed");
Ok(())
}

33
src/modes/client/status.rs Normal file
View File

@@ -0,0 +1,33 @@
use crate::client::KeepClient;
use crate::modes::common::OutputFormat;
use crate::modes::common::settings_output_format;
use clap::Command;
use log::debug;
pub fn mode(
client: &KeepClient,
_cmd: &mut Command,
settings: &crate::config::Settings,
) -> Result<(), anyhow::Error> {
debug!("CLIENT_STATUS: Getting status from remote server");
let status = client.get_status()?;
let output_format = settings_output_format(settings);
match output_format {
OutputFormat::Json => {
println!("{}", serde_json::to_string_pretty(&status)?);
}
OutputFormat::Yaml => {
println!("{}", serde_yaml::to_string(&status)?);
}
OutputFormat::Table => {
println!("Remote Server Status");
println!("====================");
println!("{}", serde_json::to_string_pretty(&status)?);
}
}
Ok(())
}

29
src/modes/common.rs
View File

@@ -9,9 +9,9 @@ use crate::compression_engine::CompressionType;
/// These utilities are typically used internally by mode implementations:
///
/// ```
/// use crate::modes::common::{format_size, OutputFormat};
/// # use keep::modes::common::{format_size, OutputFormat};
/// let formatted = format_size(1024, true); // "1.0K"
/// let format = OutputFormat::from_str("json")?;
/// // let format = OutputFormat::from_str("json")?;
/// ```
use crate::config;
use crate::meta_plugin::MetaPluginType;
@@ -42,7 +42,8 @@ use strum::IntoEnumIterator;
/// # Examples
///
/// ```
/// use keep::modes::common::OutputFormat;
/// # use keep::modes::common::OutputFormat;
/// # use std::str::FromStr;
/// assert_eq!(OutputFormat::from_str("json").unwrap(), OutputFormat::Json);
/// ```
pub enum OutputFormat {
@@ -66,11 +67,10 @@ pub enum OutputFormat {
///
/// # Examples
///
/// ```
/// # use std::env;
/// # use std::collections::HashMap;
/// ```ignore
/// use std::env;
/// env::set_var("KEEP_META_COMMAND", "ls -la");
/// let meta = get_meta_from_env();
/// let meta = keep::modes::common::get_meta_from_env();
/// assert_eq!(meta.get("COMMAND"), Some(&"ls -la".to_string()));
/// ```
pub fn get_meta_from_env() -> HashMap<String, String> {
@@ -106,6 +106,7 @@ pub fn get_meta_from_env() -> HashMap<String, String> {
/// # Examples
///
/// ```
/// # use keep::modes::common::format_size;
/// let raw = format_size(1024, false); // "1024"
/// let human = format_size(1024, true); // "1.0K"
/// ```
@@ -136,7 +137,8 @@ pub fn format_size(size: u64, human_readable: bool) -> String {
/// # Examples
///
/// ```
/// use keep::modes::common::ColumnType;
/// # use keep::modes::common::ColumnType;
/// # use std::str::FromStr;
/// assert_eq!(ColumnType::from_str("id").unwrap(), ColumnType::Id);
/// assert_eq!(ColumnType::from_str("meta:hostname").unwrap(), ColumnType::Meta);
/// ```
@@ -277,8 +279,9 @@ pub fn settings_compression_type(
/// # Examples
///
/// ```
/// let format = settings_output_format(&settings);
/// assert_eq!(format, OutputFormat::Json); // If settings.output_format = Some("json")
/// # use keep::modes::common::{settings_output_format, OutputFormat};
/// // Example usage requires a Settings instance
/// // let format = settings_output_format(&settings);
/// ```
pub fn settings_output_format(settings: &config::Settings) -> OutputFormat {
settings
@@ -303,6 +306,7 @@ pub fn settings_output_format(settings: &config::Settings) -> OutputFormat {
/// # Examples
///
/// ```
/// # use keep::modes::common::trim_lines_end;
/// let cleaned = trim_lines_end("line1 \nline2 ");
/// assert_eq!(cleaned, "line1\nline2");
/// ```
@@ -328,7 +332,8 @@ pub fn trim_lines_end(s: &str) -> String {
/// # Examples
///
/// ```
/// let table = create_table(true);
/// # use keep::modes::common::create_table;
/// let mut table = create_table(true);
/// table.add_row(vec!["Header1", "Header2"]);
/// ```
pub fn create_table(use_styling: bool) -> Table {
@@ -368,6 +373,8 @@ pub fn create_table(use_styling: bool) -> Table {
/// # Examples
///
/// ```
/// # use keep::modes::common::create_table_with_config;
/// # use keep::config::TableConfig;
/// let config = TableConfig::default();
/// let table = create_table_with_config(&config);
/// ```

2
src/modes/delete.rs
View File

@@ -36,7 +36,7 @@ use rusqlite::Connection;
///
/// # Examples
///
/// ```
/// ```ignore
/// // This would be called from main after parsing args
/// mode_delete(&mut cmd, &settings, &config, &mut vec![1, 2], &mut vec![], &mut conn, data_path)?;
/// ```

3
src/modes/generate_config.rs
View File

@@ -87,7 +87,8 @@ struct MetaPluginConfig {
///
/// # Examples
///
/// ```
/// ```ignore
/// // Example usage requires Command and Settings instances
/// mode_generate_config(&mut cmd, &settings)?;
/// ```
pub fn mode_generate_config(_cmd: &mut Command, _settings: &crate::config::Settings) -> Result<()> {

4
src/modes/get.rs
View File

@@ -1,8 +1,8 @@
use anyhow::{anyhow, Result};
use anyhow::{Result, anyhow};
use std::io::Write;
use crate::common::is_binary::is_binary;
use crate::common::PIPESIZE;
use crate::common::is_binary::is_binary;
use crate::config;
use crate::filter_plugin::FilterChain;
use crate::services::item_service::ItemService;

9
src/modes/info.rs
View File

@@ -36,7 +36,8 @@ use comfy_table::{Attribute, Cell};
///
/// # Examples
///
/// ```
/// ```ignore
/// // Example usage requires Command, Settings, Connection, and PathBuf instances
/// mode_info(&mut cmd, &settings, &mut vec![123], &mut vec![], &mut conn, data_path)?;
/// ```
pub fn mode_info(
@@ -124,7 +125,8 @@ pub struct ItemInfo {
///
/// # Examples
///
/// ```
/// ```ignore
/// // Example usage requires ItemWithMeta, Settings, and PathBuf instances
/// show_item(item_with_meta, &settings, data_path)?;
/// ```
fn show_item(
@@ -234,7 +236,8 @@ fn show_item(
///
/// # Examples
///
/// ```
/// ```ignore
/// // Example usage requires ItemWithMeta, Settings, PathBuf, and OutputFormat instances
/// show_item_structured(item_with_meta, &settings, data_path, OutputFormat::Json)?;
/// ```
fn show_item_structured(

3
src/modes/mod.rs
View File

@@ -1,6 +1,9 @@
#[cfg(feature = "server")]
pub mod server;
#[cfg(feature = "client")]
pub mod client;
/// Common utilities for all modes, including column types and output formatting.
pub mod common;

4
src/modes/save.rs
View File

@@ -63,7 +63,7 @@ impl<R: Read, W: Write> Read for TeeReader<R, W> {
///
/// # Examples
///
/// ```
/// ```ignore
/// let mut tee = TeeReader {
/// reader: std::io::Cursor::new(b"Hello, world!"),
/// writer: std::io::sink(),
@@ -104,7 +104,7 @@ impl<R: Read, W: Write> Read for TeeReader<R, W> {
///
/// # Examples
///
/// ```
/// ```ignore
/// // In CLI context, this would be called internally
/// mode_save(&mut cmd, &settings, &mut vec![], &mut vec!["important".to_string()], &mut conn, data_path)?;
/// ```

6
src/modes/server/api/common.rs
View File

@@ -10,7 +10,7 @@ pub struct ResponseBuilder;
impl ResponseBuilder {
pub fn json<T: Serialize>(data: T) -> Result<Response, StatusCode> {
let json = serde_json::to_vec(&data).map_err(|e| {
log::warn!("Failed to serialize response: {}", e);
log::warn!("Failed to serialize response: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
@@ -19,7 +19,7 @@ impl ResponseBuilder {
.header(header::CONTENT_LENGTH, json.len().to_string())
.body(axum::body::Body::from(json))
.map_err(|e| {
log::warn!("Failed to build response: {}", e);
log::warn!("Failed to build response: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})
}
@@ -30,7 +30,7 @@ impl ResponseBuilder {
.header(header::CONTENT_LENGTH, content.len().to_string())
.body(axum::body::Body::from(content.to_vec()))
.map_err(|e| {
log::warn!("Failed to build response: {}", e);
log::warn!("Failed to build response: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})
}

319
src/modes/server/api/item.rs
View File

@@ -17,8 +17,58 @@ use http_body_util::BodyExt;
use log::{debug, warn};
use std::collections::HashMap;
use std::io::{Cursor, Read};
use tokio::sync::mpsc;
use tokio::task;
/// Bridges an async mpsc receiver to a synchronous `Read` trait.
///
/// Used in `spawn_blocking` contexts to consume data from an async body
/// stream as a regular reader. Blocks on each `read()` call until data
/// is available or the channel is closed.
struct ChannelReader {
rx: mpsc::Receiver<Result<Vec<u8>, std::io::Error>>,
current: Vec<u8>,
pos: usize,
}
impl ChannelReader {
fn new(rx: mpsc::Receiver<Result<Vec<u8>, std::io::Error>>) -> Self {
Self {
rx,
current: Vec::new(),
pos: 0,
}
}
}
impl Read for ChannelReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
// If we have buffered data, return it first
if self.pos < self.current.len() {
let remaining = &self.current[self.pos..];
let n = std::cmp::min(buf.len(), remaining.len());
buf[..n].copy_from_slice(&remaining[..n]);
self.pos += n;
return Ok(n);
}
// Need more data from the channel - block until available
match self.rx.blocking_recv() {
Some(Ok(data)) => {
let n = std::cmp::min(buf.len(), data.len());
buf[..n].copy_from_slice(&data[..n]);
if n < data.len() {
self.current = data;
self.pos = n;
}
Ok(n)
}
Some(Err(e)) => Err(e),
None => Ok(0), // Channel closed, EOF
}
}
}
// Helper functions to replace the missing binary_detection module
async fn check_binary_content_allowed(
data_service: &AsyncDataService,
@@ -51,11 +101,7 @@ async fn is_content_binary(
{
Ok((_, _, is_binary)) => Ok(is_binary),
Err(e) => {
log::warn!(
"Failed to get content info for binary check for item {}: {}",
item_id,
e
);
log::warn!("Failed to get content info for binary check for item {item_id}: {e}");
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
@@ -92,7 +138,7 @@ fn handle_item_error(error: CoreError) -> StatusCode {
match error {
CoreError::ItemNotFound(_) | CoreError::ItemNotFoundGeneric => StatusCode::NOT_FOUND,
_ => {
warn!("Failed to get item: {}", error);
warn!("Failed to get item: {error}");
StatusCode::INTERNAL_SERVER_ERROR
}
}
@@ -136,13 +182,7 @@ pub async fn handle_list_items(
let tags: Vec<String> = params
.tags
.as_ref()
.map(|s| {
parse_comma_tags(s).map_err(|e| {
warn!("Failed to parse tags: {}", e);
StatusCode::BAD_REQUEST
})
})
.transpose()?
.map(|s| parse_comma_tags(s))
.unwrap_or_default();
let data_service = create_data_service(&state);
@@ -150,7 +190,7 @@ pub async fn handle_list_items(
.list_items(tags, HashMap::new())
.await
.map_err(|e| {
warn!("Failed to get items: {}", e);
warn!("Failed to get items: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
@@ -205,7 +245,7 @@ async fn handle_as_meta_response(
) -> Result<Response, StatusCode> {
// Get the item with metadata
let item_with_meta = data_service.get_item(item_id).await.map_err(|e| {
warn!("Failed to get item {} for as_meta content: {}", item_id, e);
warn!("Failed to get item {item_id} for as_meta content: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
@@ -290,7 +330,7 @@ async fn handle_as_meta_response_with_metadata(
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)
}
Err(e) => {
warn!("Failed to get content for item {}: {}", item_id, e);
warn!("Failed to get content for item {item_id}: {e}");
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
@@ -337,51 +377,104 @@ pub async fn handle_post_item(
let tags: Vec<String> = params
.tags
.as_deref()
.map(|s| {
parse_comma_tags(s).map_err(|e| {
warn!("Failed to parse tags query parameter: {}", e);
StatusCode::BAD_REQUEST
})
})
.transpose()?
.map(parse_comma_tags)
.unwrap_or_default();
// Parse metadata from query parameter
let metadata: HashMap<String, String> = if let Some(ref meta_str) = params.metadata {
serde_json::from_str(meta_str).map_err(|e| {
warn!("Failed to parse metadata JSON string: {}", e);
warn!("Failed to parse metadata JSON string: {e}");
StatusCode::BAD_REQUEST
})?
} else {
HashMap::new()
};
// Convert body to bytes first (simpler than streaming for this use case)
let body_bytes = body
.collect()
let compress = params.compress;
let run_meta = params.meta;
// When server handles both compression and meta, save_item_with_reader
// buffers internally anyway, so collect body in memory.
// When client handles compression/meta, stream the body to avoid buffering.
let item_with_meta = if compress && run_meta {
let body_bytes = body
.collect()
.await
.map_err(|e| {
warn!("Failed to read request body: {e}");
StatusCode::BAD_REQUEST
})?
.to_bytes();
task::spawn_blocking(move || {
let mut conn = db.blocking_lock();
let sync_service =
crate::services::SyncDataService::new(data_dir, settings.as_ref().clone());
let mut cursor = Cursor::new(body_bytes.to_vec());
sync_service.save_item_with_reader(&mut conn, &mut cursor, tags, metadata)
})
.await
.map_err(|e| {
warn!("Failed to read request body: {}", e);
StatusCode::BAD_REQUEST
warn!("Failed to save item (task error): {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?
.to_bytes();
.map_err(|e| {
warn!("Failed to save item: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?
} else {
// Stream body through a channel to avoid buffering in memory
let (tx, rx) = tokio::sync::mpsc::channel::<Result<Vec<u8>, std::io::Error>>(16);
let item_with_meta = task::spawn_blocking(move || {
let mut conn = db.blocking_lock();
let mut cursor = Cursor::new(body_bytes.to_vec());
let sync_service =
crate::services::SyncDataService::new(data_dir, settings.as_ref().clone());
sync_service.save_item_with_reader(&mut conn, &mut cursor, tags, metadata)
})
.await
.map_err(|e| {
warn!("Failed to save item: {}", e);
StatusCode::INTERNAL_SERVER_ERROR
})?
.map_err(|e| {
warn!("Failed to save item: {}", e);
StatusCode::INTERNAL_SERVER_ERROR
})?;
// Task to read body frames and send through channel
tokio::spawn(async move {
let mut body = body;
loop {
match body.frame().await {
None => break, // Body complete
Some(Err(e)) => {
let _ = tx
.send(Err(std::io::Error::other(format!("Body error: {e}"))))
.await;
break;
}
Some(Ok(frame)) => {
if let Ok(data) = frame.into_data() {
if tx.send(Ok(data.to_vec())).await.is_err() {
break; // Receiver dropped
}
}
}
}
}
});
task::spawn_blocking(move || {
let mut conn = db.blocking_lock();
let sync_service =
crate::services::SyncDataService::new(data_dir, settings.as_ref().clone());
// Convert async mpsc receiver into a sync Read
let mut stream_reader = ChannelReader::new(rx);
sync_service.save_item_raw_streaming(
&mut conn,
&mut stream_reader,
tags,
metadata,
compress,
run_meta,
)
})
.await
.map_err(|e| {
warn!("Failed to save item (task error): {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?
.map_err(|e| {
warn!("Failed to save item: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?
};
let compression = item_with_meta.item.compression.clone();
let tags = item_with_meta.tags.iter().map(|t| t.name.clone()).collect();
@@ -439,13 +532,7 @@ pub async fn handle_get_item_latest_content(
let tags: Vec<String> = params
.tags
.as_ref()
.map(|s| {
parse_comma_tags(s).map_err(|e| {
warn!("Failed to parse tags: {}", e);
StatusCode::BAD_REQUEST
})
})
.transpose()?
.map(|s| parse_comma_tags(s))
.unwrap_or_default();
let data_service = create_data_service(&state);
@@ -478,13 +565,14 @@ pub async fn handle_get_item_latest_content(
params.length,
params.stream,
None,
params.decompress,
)
.await
}
}
Err(CoreError::ItemNotFoundGeneric) => Err(StatusCode::NOT_FOUND),
Err(e) => {
warn!("Failed to find latest item for content: {}", e);
warn!("Failed to find latest item for content: {e}");
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
@@ -528,8 +616,8 @@ pub async fn handle_get_item_content(
}
debug!(
"ITEM_API: Getting content for item {} with stream={}, allow_binary={}, offset={}, length={}",
item_id, params.stream, params.allow_binary, params.offset, params.length
"ITEM_API: Getting content for item {item_id} with stream={}, allow_binary={}, offset={}, length={}",
params.stream, params.allow_binary, params.offset, params.length
);
let data_service = create_data_service(&state);
@@ -554,6 +642,7 @@ pub async fn handle_get_item_content(
params.length,
params.stream,
None,
params.decompress,
)
.await;
if let Ok(response) = &result {
@@ -566,6 +655,7 @@ pub async fn handle_get_item_content(
}
}
#[allow(clippy::too_many_arguments)]
async fn stream_item_content_response(
data_service: &AsyncDataService,
item_id: i64,
@@ -574,11 +664,12 @@ async fn stream_item_content_response(
length: u64,
stream: bool,
_filter: Option<String>,
decompress: bool,
) -> Result<Response, StatusCode> {
debug!("STREAM_ITEM_CONTENT_RESPONSE: stream={}", stream);
debug!("STREAM_ITEM_CONTENT_RESPONSE: stream={stream}, decompress={decompress}");
// Get the item with metadata once
let item_with_meta = data_service.get_item(item_id).await.map_err(|e| {
warn!("Failed to get item {} for content: {}", item_id, e);
warn!("Failed to get item {item_id} for content: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
@@ -592,10 +683,50 @@ async fn stream_item_content_response(
length,
stream,
None,
decompress,
)
.await
}
/// Stream raw (unprocessed) content directly from the item file.
///
/// Returns the stored file bytes without decompression or filtering.
async fn stream_raw_content_response(
data_service: &AsyncDataService,
item_id: i64,
offset: u64,
length: u64,
) -> Result<Response, StatusCode> {
// Get item info to find the file path and compression type
let item_with_meta = data_service.get_item(item_id).await.map_err(|e| {
warn!("Failed to get item {item_id} for raw content: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
let compression = item_with_meta.item.compression.clone();
// Read raw file bytes
let content = data_service
.get_raw_item_content(item_id)
.await
.map_err(|e| {
warn!("Failed to get raw content for item {item_id}: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
let response_content = apply_offset_length(&content, offset, length);
let response = Response::builder()
.header(header::CONTENT_TYPE, "application/octet-stream")
.header("X-Keep-Compression", &compression)
.header(header::CONTENT_LENGTH, response_content.len())
.body(axum::body::Body::from(response_content.to_vec()))
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
Ok(response)
}
#[allow(clippy::too_many_arguments)]
async fn stream_item_content_response_with_metadata(
data_service: &AsyncDataService,
item_id: i64,
@@ -605,11 +736,15 @@ async fn stream_item_content_response_with_metadata(
length: u64,
stream: bool,
_filter: Option<String>,
decompress: bool,
) -> Result<Response, StatusCode> {
debug!(
"STREAM_ITEM_CONTENT_RESPONSE_WITH_METADATA: stream={}",
stream
);
debug!("STREAM_ITEM_CONTENT_RESPONSE_WITH_METADATA: stream={stream}, decompress={decompress}");
// When decompress=false, return raw stored bytes
if !decompress {
return stream_raw_content_response(data_service, item_id, offset, length).await;
}
let mime_type = get_mime_type(metadata);
// Check if content is binary when allow_binary is false
@@ -630,7 +765,7 @@ async fn stream_item_content_response_with_metadata(
Ok(response)
}
Err(e) => {
warn!("Failed to stream content for item {}: {}", item_id, e);
warn!("Failed to stream content for item {item_id}: {e}");
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
@@ -649,7 +784,7 @@ async fn stream_item_content_response_with_metadata(
ResponseBuilder::binary(response_content, &mime_type)
}
Err(e) => {
warn!("Failed to get content for item {}: {}", item_id, e);
warn!("Failed to get content for item {item_id}: {e}");
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
@@ -683,13 +818,7 @@ pub async fn handle_get_item_latest_meta(
let tags: Vec<String> = params
.tags
.as_ref()
.map(|s| {
parse_comma_tags(s).map_err(|e| {
warn!("Failed to parse tags: {}", e);
StatusCode::BAD_REQUEST
})
})
.transpose()?
.map(|s| parse_comma_tags(s))
.unwrap_or_default();
let data_service = create_data_service(&state);
@@ -753,6 +882,39 @@ pub async fn handle_get_item_meta(
}
}
pub async fn handle_post_item_meta(
State(state): State<AppState>,
Path(item_id): Path<i64>,
Json(metadata): Json<HashMap<String, String>>,
) -> Result<Json<ApiResponse<()>>, StatusCode> {
let data_service = create_data_service(&state);
// Verify item exists
data_service
.get_item(item_id)
.await
.map_err(handle_item_error)?;
// Add each metadata entry
for (key, value) in &metadata {
data_service
.add_item_meta(item_id, key, value)
.await
.map_err(|e| {
warn!("Failed to add metadata {key} for item {item_id}: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
}
let response = ApiResponse {
success: true,
data: Some(()),
error: None,
};
Ok(Json(response))
}
#[utoipa::path(
delete,
path = "/api/item/{item_id}",
@@ -920,8 +1082,8 @@ pub async fn handle_diff_items(
return Err(StatusCode::BAD_REQUEST);
};
let id_a = item_a.item.id.ok_or_else(|| StatusCode::BAD_REQUEST)?;
let id_b = item_b.item.id.ok_or_else(|| StatusCode::BAD_REQUEST)?;
let id_a = item_a.item.id.ok_or(StatusCode::BAD_REQUEST)?;
let id_b = item_b.item.id.ok_or(StatusCode::BAD_REQUEST)?;
let (mut reader_a, _) = sync_service
.get_content(&mut conn, id_a)
@@ -932,13 +1094,13 @@ pub async fn handle_diff_items(
let mut content_a = Vec::new();
reader_a.read_to_end(&mut content_a).map_err(|e| {
log::error!("Failed to read content A: {}", e);
log::error!("Failed to read content A: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
let mut content_b = Vec::new();
reader_b.read_to_end(&mut content_b).map_err(|e| {
log::error!("Failed to read content B: {}", e);
log::error!("Failed to read content B: {e}");
StatusCode::INTERNAL_SERVER_ERROR
})?;
@@ -960,11 +1122,8 @@ fn compute_diff(a: &[u8], b: &[u8]) -> Vec<String> {
let old_lines: Vec<&str> = text_a.lines().collect();
let new_lines: Vec<&str> = text_b.lines().collect();
let ops = similar::TextDiff::from_lines(
text_a.as_ref(),
text_b.as_ref(),
)
.ops();
let text_diff = similar::TextDiff::from_lines(text_a.as_ref(), text_b.as_ref());
let ops = text_diff.ops();
let mut diff_lines = Vec::new();

8
src/modes/server/api/mod.rs
View File

@@ -1,5 +1,4 @@
pub mod common;
#[cfg(feature = "swagger")]
pub mod item;
#[cfg(feature = "mcp")]
pub mod mcp;
@@ -57,9 +56,11 @@ use utoipa_swagger_ui::SwaggerUi;
(url = "/", description = "Local server")
)
)]
#[allow(dead_code)]
struct ApiDoc;
pub fn add_routes(router: Router<AppState>) -> Router<AppState> {
#[cfg_attr(not(feature = "mcp"), allow(unused_mut))]
let mut router = router
// Status endpoints
.route("/api/status", get(status::handle_status))
@@ -77,7 +78,10 @@ pub fn add_routes(router: Router<AppState>) -> Router<AppState> {
"/api/item/latest/content",
get(item::handle_get_item_latest_content),
)
.route("/api/item/{item_id}/meta", get(item::handle_get_item_meta))
.route(
"/api/item/{item_id}/meta",
get(item::handle_get_item_meta).post(item::handle_post_item_meta),
)
.route(
"/api/item/{item_id}/content",
get(item::handle_get_item_content),

38
src/modes/server/common.rs
View File

@@ -67,6 +67,14 @@ pub struct ServerConfig {
/// Pre-hashed password (Unix crypt format) for secure authentication. Preferred
/// over plain text password for production use.
pub password_hash: Option<String>,
/// Optional path to TLS certificate file (PEM).
///
/// When both cert_file and key_file are set, the server uses HTTPS.
pub cert_file: Option<PathBuf>,
/// Optional path to TLS private key file (PEM).
///
/// When both cert_file and key_file are set, the server uses HTTPS.
pub key_file: Option<PathBuf>,
}
/// Application state shared across all routes.
@@ -488,6 +496,10 @@ pub struct ItemQuery {
/// Boolean flag to return content and metadata in a structured JSON format.
#[serde(default = "default_as_meta")]
pub as_meta: bool,
/// Whether the server should decompress the content (default: true).
/// Set to false when the client wants raw stored bytes for local decompression.
#[serde(default = "default_true")]
pub decompress: bool,
}
/// Query parameters for item content retrieval.
@@ -538,6 +550,10 @@ pub struct ItemContentQuery {
/// Boolean flag to return content and metadata in a structured JSON format.
#[serde(default = "default_as_meta")]
pub as_meta: bool,
/// Whether the server should decompress the content (default: true).
/// Set to false when the client wants raw stored bytes for local decompression.
#[serde(default = "default_true")]
pub decompress: bool,
}
/// Default function for allow_binary parameter.
@@ -567,6 +583,15 @@ fn default_as_meta() -> bool {
false
}
/// Default function for true boolean parameters.
///
/// # Returns
///
/// `true` as the default value.
fn default_true() -> bool {
true
}
/// Query parameters for creating an item via POST.
///
/// Query parameters for POST /api/item/ with streaming binary body.
@@ -576,6 +601,14 @@ pub struct CreateItemQuery {
pub tags: Option<String>,
/// Optional metadata as JSON string.
pub metadata: Option<String>,
/// Whether the server should compress the content (default: true).
/// Set to false when the client has already compressed the content.
#[serde(default = "default_true")]
pub compress: bool,
/// Whether the server should run meta plugins (default: true).
/// Set to false when the client has already collected metadata.
#[serde(default = "default_true")]
pub meta: bool,
}
/// Request body for creating a new item.
@@ -672,7 +705,7 @@ fn check_basic_auth(
}
// Otherwise, do direct comparison
let expected_credentials = format!("keep:{}", expected_password);
let expected_credentials = format!("keep:{expected_password}");
return decoded_str == expected_credentials;
}
}
@@ -803,6 +836,7 @@ pub async fn logging_middleware(
/// let auth_middleware = create_auth_middleware(Some("pass".to_string()), None);
/// router.layer(auth_middleware);
/// ```
#[allow(clippy::type_complexity)]
pub fn create_auth_middleware(
password: Option<String>,
password_hash: Option<String>,
@@ -822,7 +856,7 @@ pub fn create_auth_middleware(
let uri = request.uri().clone();
if !check_auth(&headers, &password, &password_hash) {
warn!("Unauthorized request to {} from {}", uri, addr);
warn!("Unauthorized request to {uri} from {addr}");
// Add WWW-Authenticate header to trigger basic auth in browsers
let mut response = Response::new(axum::body::Body::from("Unauthorized"));
*response.status_mut() = StatusCode::UNAUTHORIZED;

34
src/modes/server/mod.rs
View File

@@ -52,6 +52,8 @@ pub fn mode_server(
port: Some(server_port),
password: settings.server_password(),
password_hash: settings.server_password_hash(),
cert_file: settings.server_cert_file(),
key_file: settings.server_key_file(),
};
// Create ItemService once
@@ -88,7 +90,7 @@ async fn run_server(
format!("{}:21080", config.address)
};
debug!("SERVER: Starting REST HTTP server on {}", bind_address);
debug!("SERVER: Starting REST HTTP server on {bind_address}");
// Use the existing database connection
let db_conn = Arc::new(Mutex::new(conn));
@@ -106,6 +108,7 @@ async fn run_server(
.route("/mcp", post(mcp::handle_mcp_request))
.with_state(state.clone());
#[cfg_attr(not(feature = "mcp"), allow(unused_mut))]
let mut protected_router = Router::new()
.merge(api::add_routes(Router::new()))
.merge(pages::add_routes(Router::new()));
@@ -137,14 +140,31 @@ async fn run_server(
let addr: SocketAddr = bind_address.parse()?;
info!("SERVER: HTTP server listening on {}", addr);
// Build the app into a service
let service = app.into_make_service_with_connect_info::<SocketAddr>();
// Use TLS if both cert and key files are provided
#[cfg(feature = "tls")]
if let (Some(cert_file), Some(key_file)) = (&config.cert_file, &config.key_file) {
info!("SERVER: HTTPS server listening on {addr}");
use axum_server::tls_rustls::RustlsConfig;
let tls_config = RustlsConfig::from_pem_file(cert_file, key_file)
.await
.map_err(|e| anyhow::anyhow!("Failed to load TLS config: {e}"))?;
axum_server::bind_rustls(addr, tls_config)
.serve(service)
.await?;
return Ok(());
}
info!("SERVER: HTTP server listening on {addr}");
let listener = tokio::net::TcpListener::bind(addr).await?;
axum::serve(
listener,
app.into_make_service_with_connect_info::<SocketAddr>(),
)
.await?;
axum::serve(listener, service).await?;
Ok(())
}

32
src/modes/server/pages.rs
View File

@@ -47,12 +47,6 @@ fn default_count() -> usize {
1000
}
/// Provides the default number of items to display per page.
///
/// # Returns
///
/// The default count: 1000.
/// Adds the web page routes to the Axum router.
///
/// This function configures the routes for the web interface, including the
@@ -96,7 +90,7 @@ async fn list_items(
.map_err(|_| Html("<html><body>Internal Server Error</body></html>".to_string()))?;
Ok(response)
}
Err(e) => Err(Html(format!("<html><body>Error: {}</body></html>", e))),
Err(e) => Err(Html(format!("<html><body>Error: {e}</body></html>"))),
}
}
@@ -190,8 +184,7 @@ fn build_item_list(
html.push_str("<p>");
for tag in recent_tags {
html.push_str(&format!(
"<a href=\"/?tags={}\" style=\"margin-right: 8px;\">{}</a>",
tag, tag
"<a href=\"/?tags={tag}\" style=\"margin-right: 8px;\">{tag}</a>"
));
}
html.push_str("</p>");
@@ -228,7 +221,7 @@ fn build_item_list(
"id" => {
let id_value = item.id.map(|id| id.to_string()).unwrap_or_default();
// Make the ID a link to the item details page
format!("<a href=\"/item/{}\">{}</a>", item_id, id_value)
format!("<a href=\"/item/{item_id}\">{id_value}</a>")
}
"time" => item.ts.format("%Y-%m-%d %H:%M:%S").to_string(),
"size" => item.size.map(|s| s.to_string()).unwrap_or_default(),
@@ -257,7 +250,7 @@ fn build_item_list(
if let Ok(max_len) = max_len_str.parse::<usize>() {
if value.chars().count() > max_len {
let truncated: String = value.chars().take(max_len).collect();
format!("{}...", truncated)
format!("{truncated}...")
} else {
value
}
@@ -275,16 +268,12 @@ fn build_item_list(
crate::config::ColumnAlignment::Center => "text-align: center;",
};
html.push_str(&format!(
"<td style=\"{}\">{}</td>",
align_style, display_value
));
html.push_str(&format!("<td style=\"{align_style}\">{display_value}</td>"));
}
// Actions column
html.push_str(&format!(
"<td><a href=\"/item/{}\">View</a> | <a href=\"/api/item/{}/content\">Download</a></td>",
item_id, item_id
"<td><a href=\"/item/{item_id}\">View</a> | <a href=\"/api/item/{item_id}/content\">Download</a></td>"
));
html.push_str("</tr>");
@@ -372,7 +361,7 @@ async fn show_item(
.map_err(|_| Html("<html><body>Internal Server Error</body></html>".to_string()))?;
Ok(response)
}
Err(e) => Err(Html(format!("<html><body>Error: {}</body></html>", e))),
Err(e) => Err(Html(format!("<html><body>Error: {e}</body></html>"))),
}
}
@@ -386,10 +375,10 @@ fn build_item_details(conn: &Connection, id: i64) -> Result<String> {
let metas = db::get_item_meta(conn, &item)?;
let mut html = String::new();
html.push_str(&format!("<html><head><title>Keep - Item #{}</title>", id));
html.push_str(&format!("<html><head><title>Keep - Item #{id}</title>"));
html.push_str("<link rel=\"stylesheet\" href=\"/style.css\">");
html.push_str("</head><body>");
html.push_str(&format!("<h1>Item #{}</h1>", id));
html.push_str(&format!("<h1>Item #{id}</h1>"));
// Single table for all details
html.push_str("<table>");
@@ -439,8 +428,7 @@ fn build_item_details(conn: &Connection, id: i64) -> Result<String> {
// Links
html.push_str("<h2>Actions</h2>");
html.push_str(&format!(
"<p><a href=\"/api/item/{}/content\">Download Content</a></p>",
id
"<p><a href=\"/api/item/{id}/content\">Download Content</a></p>"
));
html.push_str("<p><a href=\"/\">Back to list</a></p>");

25
src/plugin.rs
View File

@@ -1,25 +0,0 @@
use std::io::Write;
use derive_more::{Deref, DerefMut};
/// A wrapper around a child process's stdin that implements the Write trait.
///
/// This struct allows writing data to an external process's standard input
/// in a way that's compatible with Rust's I/O traits.
#[derive(Deref, DerefMut)]
pub struct ProgramWriter {
/// The stdin handle of a spawned child process
#[deref]
#[deref_mut]
pub stdin: std::process::ChildStdin,
}
impl Write for ProgramWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.stdin.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
self.stdin.flush()
}
}

30
src/plugins.rs
View File

@@ -1,30 +0,0 @@
//! Shared plugin utilities for the keep application.
//!
//! This module provides common functionality that can be used by different
//! plugin implementations throughout the application.
use std::io::Write;
use derive_more::{Deref, DerefMut};
/// A wrapper around a child process's stdin that implements the Write trait.
///
/// This struct allows writing data to an external process's standard input
/// in a way that's compatible with Rust's I/O traits.
#[derive(Deref, DerefMut)]
pub struct ProgramWriter {
/// The stdin handle of a spawned child process
#[deref]
#[deref_mut]
pub stdin: std::process::ChildStdin,
}
impl Write for ProgramWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.stdin.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
self.stdin.flush()
}
}

37
src/services/async_data_service.rs
View File

@@ -51,6 +51,17 @@ impl AsyncDataService {
self.get(&mut conn, id)
}
pub async fn add_item_meta(
&self,
item_id: i64,
name: &str,
value: &str,
) -> Result<(), CoreError> {
let conn = self.db.lock().await;
crate::db::add_meta(&conn, item_id, name, value)?;
Ok(())
}
pub async fn list_items(
&self,
tags: Vec<String>,
@@ -184,6 +195,32 @@ impl AsyncDataService {
Ok((Box::pin(stream), content_length))
}
/// Get raw item content without decompression.
///
/// Reads the stored file bytes directly from disk, bypassing decompression.
/// Used when the client requests raw bytes with `decompress=false`.
pub async fn get_raw_item_content(&self, id: i64) -> Result<Vec<u8>, CoreError> {
let data_path = self.data_path.clone();
tokio::task::spawn_blocking(move || {
let mut item_path = data_path;
item_path.push(id.to_string());
let mut file = std::fs::File::open(&item_path).map_err(|e| {
CoreError::Io(std::io::Error::new(
std::io::ErrorKind::NotFound,
format!("Item file not found: {item_path:?}: {e}"),
))
})?;
let mut content = Vec::new();
file.read_to_end(&mut content)?;
Ok(content)
})
.await
.map_err(|e| CoreError::Other(anyhow::anyhow!("Task join error: {}", e)))?
}
}
impl DataService for AsyncDataService {

11
src/services/compression_service.rs
View File

@@ -1,4 +1,4 @@
use crate::compression_engine::{get_compression_engine, CompressionType};
use crate::compression_engine::{CompressionType, get_compression_engine};
use crate::services::error::CoreError;
use anyhow::anyhow;
use std::io::Read;
@@ -15,7 +15,7 @@ pub struct CompressionService;
///
/// # Examples
///
/// ```
/// ```ignore
/// let service = CompressionService::new();
/// let content = service.get_item_content(path, "gzip")?;
/// ```
@@ -24,7 +24,7 @@ pub struct CompressionService;
///
/// # Examples
///
/// ```
/// ```ignore
/// let service = CompressionService::new();
/// let content = service.get_item_content(path, "gzip")?;
/// ```
@@ -40,6 +40,7 @@ impl CompressionService {
/// # Examples
///
/// ```
/// # use keep::services::CompressionService;
/// let service = CompressionService::new();
/// ```
pub fn new() -> Self {
@@ -67,7 +68,7 @@ impl CompressionService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let content = service.get_item_content(item_path, "lz4")?;
/// assert_eq!(content.len(), expected_size);
/// ```
@@ -111,7 +112,7 @@ impl CompressionService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let mut reader = service.stream_item_content(item_path, "gzip")?;
/// let mut buf = [0; 1024];
/// let n = reader.read(&mut buf)?;

17
src/services/filter_service.rs
View File

@@ -16,9 +16,8 @@ type FilterConstructor = fn() -> Box<dyn crate::filter_plugin::FilterPlugin>;
/// # Usage
///
/// ```rust
/// use keep::services::FilterService;
/// let service = FilterService::new();
/// let chain = service.create_filter_chain(Some("head_lines(10)")).unwrap();
/// service.filter_data(&mut chain, &mut reader, &mut writer)?;
/// ```
pub struct FilterService;
@@ -38,6 +37,7 @@ impl FilterService {
/// # Examples
///
/// ```
/// # use keep::services::FilterService;
/// let service = FilterService::new();
/// ```
pub fn new() -> Self {
@@ -63,7 +63,7 @@ impl FilterService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let chain = service.create_filter_chain(Some("head_lines(10)"))?;
/// assert!(chain.is_some());
/// let empty = service.create_filter_chain(None)?;
@@ -99,7 +99,7 @@ impl FilterService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let mut chain = parse_filter_string("head_lines(5)")?;
/// service.filter_data(&mut chain, &mut reader, &mut writer)?;
/// ```
@@ -139,7 +139,7 @@ impl FilterService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let filtered = service.process_with_filter(b"Hello\nWorld\n", Some("head_lines(1)"))?;
/// assert_eq!(filtered, b"Hello\n");
/// ```
@@ -185,7 +185,7 @@ static FILTER_PLUGIN_REGISTRY: Lazy<Mutex<HashMap<String, FilterConstructor>>> =
///
/// # Examples
///
/// ```rust
/// ```ignore
/// register_filter_plugin("custom_filter", || Box::new(CustomFilter::default()));
/// ```
pub fn register_filter_plugin(name: &str, constructor: FilterConstructor) {
@@ -209,9 +209,10 @@ pub fn register_filter_plugin(name: &str, constructor: FilterConstructor) {
///
/// # Examples
///
/// ```rust
/// ```
/// # use keep::services::filter_service::get_available_filter_plugins;
/// let plugins = get_available_filter_plugins();
/// assert!(plugins.contains_key("head_bytes"));
/// // Plugins are registered at startup via ctors; specific names may vary by configuration.
/// ```
pub fn get_available_filter_plugins() -> HashMap<String, FilterConstructor> {
FILTER_PLUGIN_REGISTRY.lock().unwrap().clone()

30
src/services/item_service.rs
View File

@@ -1,5 +1,5 @@
use crate::common::PIPESIZE;
use crate::compression_engine::{get_compression_engine, CompressionType};
use crate::compression_engine::{CompressionType, get_compression_engine};
use crate::config::Settings;
use crate::db::{self, Item, Meta};
use crate::filter_plugin;
@@ -50,6 +50,8 @@ impl ItemService {
/// # Examples
///
/// ```
/// # use keep::services::ItemService;
/// # use std::path::PathBuf;
/// let service = ItemService::new(PathBuf::from("/data"));
/// ```
pub fn new(data_path: PathBuf) -> Self {
@@ -82,7 +84,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let item_with_meta = item_service.get_item(&conn, 1)?;
/// assert_eq!(item_with_meta.item.id, Some(1));
/// ```
@@ -121,7 +123,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let item_with_content = item_service.get_item_content(&conn, 1)?;
/// assert!(!item_with_content.content.is_empty());
/// ```
@@ -183,7 +185,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let (content, mime, is_binary) = item_service.get_item_content_info(&conn, 1, Some("head_lines(10)"))?;
/// ```
pub fn get_item_content_info(
@@ -223,7 +225,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let is_bin = item_service.is_content_binary(path, "gzip", &meta)?;
/// ```
fn is_content_binary(
@@ -269,7 +271,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let (reader, mime, is_bin) = item_service.get_item_content_info_streaming(&conn, 1, Some("grep(error)"))?;
/// ```
pub fn get_item_content_info_streaming(
@@ -311,7 +313,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let chain = parse_filter_string("head(100)")?;
/// let (reader, mime, is_bin) = item_service.get_item_content_info_streaming_with_chain(&conn, 1, Some(&chain))?;
/// ```
@@ -417,7 +419,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let item = item_service.find_item(&conn, vec![1], &vec![], &HashMap::new())?;
/// ```
pub fn find_item(
@@ -486,7 +488,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let items = item_service.list_items(&conn, &vec!["work"], &HashMap::new())?;
/// ```
pub fn list_items(
@@ -556,7 +558,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// item_service.delete_item(&mut conn, 1)?;
/// ```
pub fn delete_item(&self, conn: &mut Connection, id: i64) -> Result<(), CoreError> {
@@ -608,7 +610,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let reader = std::io::stdin();
/// let item = item_service.save_item(reader, &mut cmd, &settings, &mut vec![], &mut conn)?;
/// ```
@@ -739,7 +741,7 @@ impl ItemService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let content = b"Hello, world!";
/// let tags = vec!["mcp".to_string()];
/// let meta = HashMap::from([("source".to_string(), "api".to_string())]);
@@ -869,7 +871,7 @@ impl<R: Read> FilteringReader<R> {
///
/// # Examples
///
/// ```
/// ```ignore
/// let reader = std::io::Cursor::new(b"data");
/// let filter_chain = parse_filter_string("head(10)")?;
/// let filtered = FilteringReader::new(reader, Some(filter_chain));
@@ -905,7 +907,7 @@ impl<R: Read> Read for FilteringReader<R> {
///
/// # Examples
///
/// ```
/// ```ignore
/// let mut filtered = FilteringReader::new(std::io::Cursor::new(b"Hello"), None);
/// let mut buf = [0; 5];
/// let n = filtered.read(&mut buf).unwrap();

1
src/services/meta_service.rs
View File

@@ -200,6 +200,7 @@ impl MetaService {
/// # Examples
///
/// ```
/// # use keep::services::MetaService;
/// let service = MetaService::new();
/// let initial_meta = service.collect_initial_meta();
/// ```

5
src/services/status_service.rs
View File

@@ -16,7 +16,7 @@ use std::str::FromStr;
///
/// # Examples
///
/// ```
/// ```ignore
/// let service = StatusService::new();
/// let status = service.generate_status(&mut cmd, &settings, data_path, db_path);
/// ```
@@ -34,6 +34,7 @@ impl StatusService {
/// # Examples
///
/// ```
/// # use keep::services::StatusService;
/// let service = StatusService::new();
/// ```
pub fn new() -> Self {
@@ -63,7 +64,7 @@ impl StatusService {
///
/// # Examples
///
/// ```
/// ```ignore
/// let status = service.generate_status(&mut cmd, &settings, data_path, db_path);
/// assert!(!status.filter_plugins.is_empty());
/// ```

203
src/services/sync_data_service.rs
View File

@@ -1,16 +1,19 @@
use crate::common::status::StatusInfo;
use crate::compression_engine::{CompressionType, get_compression_engine};
use crate::config::Settings;
use crate::db::Item;
use crate::db::Meta;
use crate::modes::common::settings_compression_type;
use crate::services::data_service::DataService;
use crate::services::error::CoreError;
use crate::services::item_service::ItemService;
use crate::services::meta_service::MetaService;
use crate::services::status_service::StatusService;
use crate::services::types::{ItemWithContent, ItemWithMeta};
use clap::Command;
use rusqlite::Connection;
use std::collections::HashMap;
use std::io::{Cursor, Read};
use std::io::{Cursor, Read, Write};
use std::path::{Path, PathBuf};
pub struct SyncDataService {
@@ -80,6 +83,204 @@ impl SyncDataService {
self.get_item(conn, item_id)
}
/// Save an item with granular control over compression and meta plugins.
///
/// This method allows clients to control whether compression and meta plugins
/// run server-side or were already handled by the client.
///
/// # Arguments
///
/// * `conn` - Database connection.
/// * `content` - Raw content bytes.
/// * `tags` - Tags to associate with the item.
/// * `metadata` - Client-provided metadata.
/// * `compress` - Whether the server should compress the content.
/// * `run_meta` - Whether the server should run meta plugins.
///
/// # Returns
///
/// * `Result<ItemWithMeta, CoreError>` - The saved item with full details.
pub fn save_item_raw(
&self,
conn: &mut Connection,
content: &[u8],
tags: Vec<String>,
metadata: HashMap<String, String>,
compress: bool,
run_meta: bool,
) -> Result<ItemWithMeta, CoreError> {
let mut cmd = Command::new("keep");
let settings = &self.settings;
let mut tags = tags;
if tags.is_empty() {
tags.push("none".to_string());
}
let compression_type = if compress {
settings_compression_type(&mut cmd, settings)
} else {
CompressionType::None
};
let compression_engine = get_compression_engine(compression_type.clone())?;
let item_id;
let mut item;
{
item = crate::db::create_item(conn, compression_type.clone())?;
item_id = item.id.unwrap();
crate::db::set_item_tags(conn, item.clone(), &tags)?;
}
// Initialize meta plugins if requested
let meta_service = MetaService::new();
let mut plugins = if run_meta {
meta_service.get_plugins(&mut cmd, settings)
} else {
Vec::new()
};
if run_meta {
meta_service.initialize_plugins(&mut plugins, conn, item_id);
}
// Write content to file
let mut item_path = self.item_service.get_data_path().clone();
item_path.push(item_id.to_string());
let mut item_out = compression_engine.create(item_path)?;
let mut total_bytes = 0i64;
const PIPESIZE: usize = 65536;
if run_meta && !plugins.is_empty() {
// Process in chunks for meta plugins
let mut offset = 0;
while offset < content.len() {
let end = std::cmp::min(offset + PIPESIZE, content.len());
let chunk = &content[offset..end];
item_out.write_all(chunk)?;
total_bytes += chunk.len() as i64;
meta_service.process_chunk(&mut plugins, chunk, conn, item_id);
offset = end;
}
} else {
// Write all at once, no meta processing
item_out.write_all(content)?;
total_bytes = content.len() as i64;
}
item_out.flush()?;
drop(item_out);
// Finalize meta plugins
if run_meta {
meta_service.finalize_plugins(&mut plugins, conn, item_id);
}
// Add client-provided metadata
for (key, value) in &metadata {
crate::db::add_meta(conn, item_id, key, value)?;
}
item.size = Some(total_bytes);
crate::db::update_item(conn, item)?;
self.get_item(conn, item_id)
}
/// Save an item from a streaming reader with granular control over compression.
///
/// Unlike `save_item_raw` which takes a pre-buffered `&[u8]`, this method
/// reads from the reader in chunks and writes directly to the compression
/// engine, avoiding buffering the entire content in memory.
pub fn save_item_raw_streaming(
&self,
conn: &mut Connection,
reader: &mut dyn Read,
tags: Vec<String>,
metadata: HashMap<String, String>,
compress: bool,
run_meta: bool,
) -> Result<ItemWithMeta, CoreError> {
let mut cmd = Command::new("keep");
let settings = &self.settings;
let mut tags = tags;
if tags.is_empty() {
tags.push("none".to_string());
}
let compression_type = if compress {
settings_compression_type(&mut cmd, settings)
} else {
CompressionType::None
};
let compression_engine = get_compression_engine(compression_type.clone())?;
let item_id;
let mut item;
{
item = crate::db::create_item(conn, compression_type.clone())?;
item_id = item.id.unwrap();
crate::db::set_item_tags(conn, item.clone(), &tags)?;
}
// Initialize meta plugins if requested
let meta_service = MetaService::new();
let mut plugins = if run_meta {
meta_service.get_plugins(&mut cmd, settings)
} else {
Vec::new()
};
if run_meta {
meta_service.initialize_plugins(&mut plugins, conn, item_id);
}
// Write content to file via streaming
let mut item_path = self.item_service.get_data_path().clone();
item_path.push(item_id.to_string());
let mut item_out = compression_engine.create(item_path)?;
let mut buffer = [0u8; 65536];
let mut total_bytes = 0i64;
loop {
let n = reader.read(&mut buffer)?;
if n == 0 {
break;
}
item_out.write_all(&buffer[..n])?;
total_bytes += n as i64;
if run_meta {
meta_service.process_chunk(&mut plugins, &buffer[..n], conn, item_id);
}
}
item_out.flush()?;
drop(item_out);
// Finalize meta plugins
if run_meta {
meta_service.finalize_plugins(&mut plugins, conn, item_id);
}
// Add client-provided metadata
for (key, value) in &metadata {
crate::db::add_meta(conn, item_id, key, value)?;
}
item.size = Some(total_bytes);
crate::db::update_item(conn, item)?;
self.get_item(conn, item_id)
}
pub fn get_item(&self, conn: &mut Connection, id: i64) -> Result<ItemWithMeta, CoreError> {
self.item_service.get_item(conn, id)
}

2
src/services/types.rs
View File

@@ -28,7 +28,7 @@ impl ItemWithMeta {
///
/// # Examples
///
/// ```
/// ```ignore
/// let item_with_meta = ItemWithMeta { /* ... */ };
/// let meta_map = item_with_meta.meta_as_map();
/// assert_eq!(meta_map.get("hostname"), Some(&"example.com".to_string()));

7
src/tests/common/test_helpers.rs
View File

@@ -16,7 +16,7 @@ pub fn create_temp_dir() -> TempDir {
pub fn create_temp_file_with_content(dir: &TempDir, filename: &str, content: &str) -> PathBuf {
let file_path = dir.path().join(filename);
let mut file = File::create(&file_path).expect("Failed to create test file");
write!(file, "{}", content).expect("Failed to write to test file");
write!(file, "{content}").expect("Failed to write to test file");
file_path
}
@@ -95,14 +95,13 @@ pub fn get_file_size(file_path: &PathBuf) -> u64 {
/// Assert that a file exists
pub fn assert_file_exists(file_path: &PathBuf) {
assert!(file_path.exists(), "File {:?} does not exist", file_path);
assert!(file_path.exists(), "File {file_path:?} does not exist");
}
/// Assert that a file does not exist
pub fn assert_file_not_exists(file_path: &PathBuf) {
assert!(
!file_path.exists(),
"File {:?} should not exist but it does",
file_path
"File {file_path:?} should not exist but it does"
);
}