- Move ItemInfo to services/types.rs for sharing between client and server
- Replace .expect() in compression_service with proper error handling
- Add CoreError::PayloadTooLarge variant for semantic error handling
- Export CoreError from lib.rs for library users
- Unify get_item_meta_name/value to take &str instead of String
- Extract item_path() helper in ItemService to reduce duplication
- Add warning logs for silent errors in list.rs
- Fix pre-existing borrow errors: tx moved in export handler,
item_with_meta partial move in TryFrom implementation
- Fix unused data_dir variables in server code
- diff.rs: graceful error instead of expect() on item ID in spawned thread
- common.rs: lazy_static regex, avoid unwrap on regex captures
- db.rs: ok_or_else guard on item.id in delete_item
- list/get/info/export/client/list: use settings.meta_filter() helper
- item_service.rs: expect() on meta lock instead of silent swallow
- filter_plugin/mod.rs: extract parse_encoding_option() helper
- main.rs: document unsafe libc::umask block with safety rationale
Schema changes:
- Rename items.size to items.uncompressed_size for clarity
- Add compressed_size (INTEGER NULL) - tracks compressed file size on disk
- Add closed (BOOLEAN NOT NULL DEFAULT 1) - tracks whether item is fully written
- Existing items default to closed=true via migration
Lifecycle:
- Items created with closed=false, set to true on successful save/import
- Compressed size captured via fs::metadata() after compression writer closes
- Truncated uploads (413) get compressed_size set, closed=true, uncompressed_size=None
- Update command now backfills both uncompressed_size and compressed_size
Also includes bug fixes and dedup from prior review:
- Fix stream_raw_content_response using uncompressed_size for raw byte Content-Length
- ApiResponse::ok()/empty() constructors, TryFrom<ItemWithMeta> for ItemInfo
- tag_names() method on ItemWithMeta, meta_filter() on Settings
- Fix .unwrap() panics in compression engine Read/Write impls
- Fix TOCTOU race in stream_raw_content_response (now uses compressed_size)
- Fix swallowed write errors in meta plugins (digest, magic_file, exec)
- Fix term::stderr().unwrap() panic in item_service
- Deduplicate ItemService::new() calls across 20 API handlers
- ImportMeta supports #[serde(alias = "size")] for backward compat
All 75 tests, 67 doc tests pass. Clippy clean.
- Add streaming tar-based export (--export produces .keep.tar)
- Add streaming tar import (--import reads .keep.tar archives)
- Add server endpoints GET /api/export and POST /api/import
- Rename CompressionType::None to CompressionType::Raw with "none" as alias
- Add DB migration to update existing "none" compression values to "raw"
- Fix export endpoint to propagate errors to client instead of swallowing
- Fix import endpoint to return 413 on max_body_size instead of truncating
Export streams items as tar archives without loading entire files into memory.
Import extracts items with new IDs, preserving original order. Both work
locally and via client/server mode.
Co-Authored-By: opencode <noreply@opencode.ai>
- Local and client/server modes now support ID-based filtering
- keep -l 1 2 3 lists specific items by ID
- keep -l --ids-only 1 2 3 outputs just those IDs
- Server API adds optional 'ids' query parameter to GET /api/item/
- KeepClient.list_items gains ids parameter
- Add infer crate as meta plugin for MIME type detection
- Add tree_magic_mini crate as alternative meta plugin for MIME type detection
- Add zstd, infer, tree_magic_mini to default features
- Fix static build script to use musl target instead of glibc+crt-static
- Remove hardcoded shell list from --generate-completion help text
- Fix update() in both new plugins to emit MIME metadata when buffer fills
Export/import:
- Add --export and --import modes for both local and client paths
- Use strfmt crate for --export-filename-format templates ({id}, {tags}, {ts}, {compression})
- Import preserves original timestamps via server ?ts= param
- --import-data-file for file-based import; stdin fallback streams with PIPESIZE buffers
Service unification:
- Merge SyncDataService unique methods into ItemService (delete_item now returns Result<Item>)
- Delete AsyncDataService, AsyncItemService, DataService trait (dead code / async-blocking anti-pattern)
- All server handlers use spawn_blocking + ItemService directly
- Extract shared types (ExportMeta, ImportMeta) and helpers (resolve_item_id(s), check_binary_tty)
Binary detection fix:
- Replace broken metadata.get("map") + is_binary(&[]) with actual content sampling
- Both as_meta and allow_binary paths read PIPESIZE sample before deciding
- Never load entire item into memory for binary check
Other fixes:
- Fix lock consistency: all handlers use blocking_lock() in spawn_blocking (no mixed lock().await)
- Use ISO 8601 format for {ts} in export filenames
- Fix resolve_item_ids returning only 1 item for tag lookups
- Fix client get.rs triple-buffering and export.rs whole-file buffering
- Add KeepClient::get_item_content_stream() for streaming reads
- Pass all clippy --features server lints (Path vs PathBuf, &mut conn, etc.)
- server_compress was true when compression_type=None, telling server to
recompress with its default (lz4) instead of storing raw
- compression_type query param was only sent when !server_compress,
so 'none' was never sent to server
- Fix: server_compress always false in client mode (client handles all
compression), compression_type always sent to server
Tested: save/get/list/info/filters/delete for lz4, none, gzip on both
local and client/server modes. All operations produce matching results.
- Client save now logs 'New item: {id}' immediately after server response
- Compression type sent as query param, stored in DB compression field (not _client_compression metadata)
- Client set_item_size() sends uncompressed size via POST /api/item/{id}/update?size=N
- Server raw content GET uses actual file size for Content-Length (not uncompressed item.size)
- Removed _client_compression metadata hack from client save and get
- Fixed server handle_update_item to support size-only updates
- Fixed clippy: collapsible_if, too_many_arguments, unnecessary mut refs
- Fixed ListItemsQuery doctest missing meta field
- Add SaveMetaFn callback pattern: meta plugins receive a closure instead of
&Connection, enabling the same plugin code to work in local, client, and
server contexts (collect-to-Vec, collect-to-HashMap, or direct DB write)
- Client save now runs meta plugins locally during streaming (smart client
sets meta=false, server skips its own plugins)
- Add POST /api/item/{id}/update endpoint for re-running plugins on stored
content without downloading compressed data
- Add client update mode (--update with --meta-plugin flags)
- Extract shared utilities: stream_copy, print_serialized, build_path_table,
ensure_default_tag to reduce duplication across modes
- Add upsert_tag for idempotent tag addition (INSERT OR IGNORE)
- Add warn logging on save_meta lock failure in BaseMetaPlugin and MetaService
Move apply_color/apply_table_attribute to common.rs for sharing.
Add render_list_table_with_format() that takes ColumnConfig slice
and pre-computed row values. Client list now renders columns based
on settings.list_format, showing empty for columns where server
data is unavailable (e.g. text_line_count, token_count).
Extract shared render_item_info_table() and render_list_table() in
modes/common.rs. Update client/info, client/list, client/status,
info, status, and status_plugins to use create_table_with_config
with settings.table_config instead of hardcoded presets.
Previously only local --list used table_config; all other tables
(client modes, status, status-plugins) ignored it.
- Include error field in get_status() ApiResponse so server error
messages are surfaced instead of generic 'No status data returned'
- Use trim_lines_end() on table output to match local mode formatting
Change client get_status() to return StatusInfo struct instead of
serde_json::Value, then render paths, meta plugins, and compression
tables matching the local mode's output style.
- URL-encode all query parameter keys and values in get_json_with_query
and post_stream. Previously raw JSON like {"project":"alpha"} was
sent unencoded, causing 'invalid uri character' errors.
- Pass settings.meta (key=value pairs) from client save to server as
metadata. Previously always passed empty HashMap, so --meta was
silently ignored in client save mode.
The --server-password, --server-password-hash, --server-username,
--server-jwt-secret, --server-jwt-secret-file, and --server-max-body-size
options were appearing in the generic Options section instead of the
Server Options section.
Per project convention, enum string representations should use
snake_case. Use explicit strum serialize attributes instead of
serialize_all to avoid incorrect splitting of acronyms like
GZip → g_zip and ZStd → z_std.
Plumb metadata filter from client CLI through the HTTP API to the
server's data_service.list_items(). The server accepts a JSON-encoded
meta query parameter where null values mean 'key exists' and string
values mean 'exact match'.
Also fix LZ4 compression round-trip for client mode:
- Explicit flush FrameEncoder before drop to avoid sending only the
frame header when compress=false
- Send _client_compression metadata so client knows actual compression
on retrieval (server records compression=None when compress=false)
- Use FrameDecoder (frame format) instead of decompress_size_prepended
(size-prepended format) to match server storage format
FilteringReader::read() returned Ok(0) (EOF) when a filter consumed a
chunk without producing output. Filters like skip_lines need to see
multiple chunks before outputting anything — returning 0 prematurely
truncated the stream. Loop until the filter produces output or the
underlying reader is truly exhausted.
Replace 4 unsafe sites with safe wrappers:
- libc::pipe2 → nix::unistd::pipe2 (safe OwnedFd return)
- File::from_raw_fd → File::from(OwnedFd) (safe ownership transfer)
- unsafe impl Send for SendCookie → thread_local! lazy Cookie
(each thread gets its own independent Cookie, no Send needed)
- pre_exec + libc::fcntl → command-fds crate fd_mappings()
(handles CLOEXEC clearing safely, also fixes potential fd leak
on spawn failure via OwnedFd RAII)
Only libc::umask remains as a single unavoidable unsafe site
(no safe Rust wrapper exists for the umask syscall).
Also updates AGENTS.md to remove stale SendCookie exception.
All 27 doctests in db.rs wrote keep.db to the project root via
PathBuf::from("keep.db"). Now use tempfile::tempdir() so the
database is created in a temp directory and cleaned up automatically.
- Add --update mode to modify tags and metadata for existing items by ID
- Add --meta key=value flag to set metadata during save/update
- Add --meta key (bare) to delete metadata keys or filter by existence
- Add --meta-plugin/-M name:{json} flag for plugin options via CLI
- Env meta plugin now uses options from --meta-plugin instead of only env vars
- Stream decompressed content to diff via /dev/fd pipes (no temp files)
- Wire --list-format CLI arg to settings (was parsed but ignored)
- Allow --info to accept tags (was restricted to numeric IDs only)
- Change DB meta filtering to HashMap<String, Option<String>> for exact match + key existence
- Fix fcntl error checking in diff pre_exec
- Fix README inaccuracies (delete by tag, nonexistent --digest flag, meta plugin key names)
- Quick Start: show bare keep <tag> (save) and keep <#> (get) shortcuts
- Save Mode: note that --save is optional when piping content
- Get Mode: clarify that only numeric IDs default to Get mode;
fix incorrect keep <tag> example that would actually save
- Add clap_complete dependency for bash/zsh/fish/elvish/powershell
- Add --generate-completion <shell> flag that prints completion script to stdout
- profile.bash sources completions via command keep --generate-completion bash
- @ and @@ aliases get completions via wrapper functions that delegate to _keep
- README updated with Shell Completion section
Major overhaul of server architecture and security posture:
- Streaming: Unified all I/O through PIPESIZE (8192-byte) buffers.
POST bodies stream via MpscReader through the save pipeline. GET
content streams from disk via decompression to client. Removed
save_item_with_reader, get_item_content_info, ChannelReader.
413 responses keep partial items (nonfatal by design).
- Security: XSS protection in all HTML pages via html_escape crate.
Security headers middleware (nosniff, frame deny, referrer policy).
CORS tightened to explicit headers. Input validation for tags
(256 chars), metadata (128/4096), pagination (10k cap). Config
file reads use from_utf8_lossy. Generic error messages in HTML.
Diff endpoint has 10 MB per-item cap. max_body_size config option.
- Panics eliminated: Path unwraps → proper error propagation.
Mutex unwraps → map_err (registries) / expect with message (local).
- MCP removed: Deleted all MCP code, rmcp dependency, mcp feature.
- Docs: Updated README, DESIGN, AGENTS to reflect all changes.
- Add plugin schema types and runtime discovery for meta/filter plugins
- Rewrite --generate-config to use schema system instead of hardcoded types
- Add Settings::validate_config() for startup validation
- Cache tokenizer instances via static Lazy to avoid repeated BPE loading
- Add split_by_token_iter() and count_bounded() to Tokenizer
- Fix double-counting bug in TokensMetaPlugin when buffer < max_buffer_size
- Eliminate unnecessary allocations in token count methods
- Refactor token filters: remove Option<Tokenizer>, use iterator API
- Fix TailTokensFilter correctness: unbounded buffer instead of ring buffer
- Add encoding option to all token filters
- Add description() to MetaPlugin and FilterPlugin traits
- Fix unused_mut warning in compression engine (feature-gated code)
Co-Authored-By: code-review-bot <noreply@anthropic.com>
Add tiktoken-based token counting via new 'tokens' feature flag.
New components:
- Shared tokenizer module wrapping tiktoken CoreBPE (cl100k_base, o200k_base)
- TokensMetaPlugin: streaming token counter, tokenizes each chunk independently
- head_tokens(N): stream first N tokens, split at exact boundary when mid-chunk
- skip_tokens(N): skip first N tokens, stream the rest
- tail_tokens(N): bounded ring buffer (~16KB), outputs last N tokens at finalize
All filters are fully streaming — no full-stream buffering.
Meta plugin accuracy: exact for normal text, ±1-2 tokens if long whitespace
sequence spans a chunk boundary.
Also: add 'client' and 'tokens' to default features, add curl to Dockerfile builder stage.
Add server-side JWT authentication with permission-based access control
(read/write/delete claims). Password authentication now uses HTTP Basic
auth only (replacing Bearer). Add configurable username for both server
and client (--server-username/--client-username, defaults to "keep").
JWT secret supports file-based loading via --server-jwt-secret-file for
Docker secrets. OPTIONS preflight requests bypass auth. HEAD mapped to
read permission.
Co-Authored-By: opencode <noreply@opencode.ai>
- Build with server, mcp, swagger, client, tls features (all except magic)
- Add KEEP_* environment variable documentation and defaults
- Copy CA certificates for HTTPS client support in scratch image
- Add docker-compose.yml with keep-data and keep-config volumes
- Add program-based lz4 command fallback when lz4 feature is disabled
- Feature-gate lz4.rs and lz4 tests to compile without lz4_flex
- Delete legacy magic.rs (unused, no feature gating, superseded by magic_file.rs)
Security:
- Use constant-time password comparison (subtle crate) to prevent timing attacks
- Replace permissive CORS with configurable origin-restricted CORS
- Add TLS warning when password auth is used without HTTPS
Bug fixes:
- Convert MetaPlugin panics to anyhow::Result (get_meta_plugin, outputs_mut, options_mut)
- Replace item.id.unwrap() with proper error handling across 15 call sites
- Fix panic on unknown column type in list mode
- Fix conflicting PIPESIZE constant (was 8192 vs 65536, now unified to 8192)
- Add 256MB filter chain buffer limit to prevent OOM
- Gracefully skip unregistered plugins instead of panicking
Dead code removal:
- Delete unused filter parser files (filter_parser.rs, filter.pest, parser/ module)
- ~260 lines of dead PEG parser code removed
Code consolidation:
- Add is_content_binary_from_metadata() helper (was duplicated in 4 places)
- Simplify save_item_raw() to delegate to save_item_raw_streaming() (~90 lines removed)
Incomplete features:
- Populate filter_plugins in status output from global registry
- Add FallbackMagicFileMetaPlugin (was referenced but never implemented)
- Document init_plugins() as intentional no-op
Infrastructure:
- Add Dockerfile (static musl binary on scratch, 4.8MB)
- Add .dockerignore
- Add cors_origin to ServerConfig and config.rs
Add optional TLS support for the server using axum-server with the
tls-rustls feature. When --server-cert and --server-key are provided
(and tls feature is enabled), the server binds with TLS instead of
plain HTTP.
Changes:
- Add axum-server dependency with optional tls-rustls feature
- New 'tls' feature flag (independent of 'server')
- --server-cert/--server-key CLI args gated behind tls feature
- ServerConfig extended with cert_file/key_file fields
- Conditional TLS/HTTP binding in server mod.rs
- Fix PathBuf::to_str().unwrap() panic risk -> to_string_lossy()
- Update README.md and DESIGN.md with TLS documentation
Server changes for client mode streaming:
- POST /api/item/ now streams body via async channel → ChannelReader
→ save_item_raw_streaming when compress=false or meta=false
- Add POST /api/item/{id}/meta endpoint for client-side metadata
- Add save_item_raw_streaming<R: Read> to SyncDataService
- Add add_item_meta to AsyncDataService
Fix pre-existing issues that were hidden behind swagger cfg gate:
- Remove #[cfg(feature = "swagger")] from item module so it compiles
with just the server feature
- Fix parse_comma_tags usage (returns Vec, not Result)
- Fix TextDiff temporary value lifetime issue
- Fix io::Error::new → io::Error::other
- Fix ok_or_else → ok_or for Copy types
- Inline format args throughout server code
- Fix empty line after doc comment in pages.rs
- Add cfg_attr for unused_mut where mcp feature gates mutation
- Add type_complexity allow on create_auth_middleware
- Distinguish task error vs save error in spawn_blocking handlers
Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
Add client mode enabling the keep CLI to connect to a remote keep
server over HTTP. Local plugins (compression, meta, filters) run on
the client; the server stores/retrieves binary blobs.
Architecture:
- Client save uses 3-thread streaming pipeline: reader thread (stdin
→ tee/stdout → hash → compress), OS pipe, streamer thread (pipe →
chunked HTTP POST). Memory usage is O(PIPESIZE) regardless of data
size.
- Server accepts compress=false, meta=false, decompress=false query
params for granular control of server-side processing.
- Streaming body handling on server via async channel → sync reader
bridge (ChannelReader).
Key additions:
- src/client.rs: KeepClient with post_stream() for chunked upload
- src/modes/client/: save, get, list, info, delete, diff, status
- --client-url / KEEP_CLIENT_URL configuration
- --client-password / KEEP_CLIENT_PASSWORD for auth
- os_pipe dependency for zero-copy pipe streaming
Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
- Rewrite README.md with comprehensive documentation covering all
features: compression engines, meta plugins, filter plugins, server
mode, MCP integration, and configuration
- Add MIT LICENSE file
- Delete README.org (consolidated into README.md)
- Delete empty PLAN.md
- Update AGENTS.md with current build/test commands and conventions
Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
