Commit Graph

707 Commits

Author SHA1 Message Date
0af74000d2 fix: eliminate unsafe code via nix, command-fds, and thread-local cookie
Replace 4 unsafe sites with safe wrappers:

- libc::pipe2 → nix::unistd::pipe2 (safe OwnedFd return)
- File::from_raw_fd → File::from(OwnedFd) (safe ownership transfer)
- unsafe impl Send for SendCookie → thread_local! lazy Cookie
  (each thread gets its own independent Cookie, no Send needed)
- pre_exec + libc::fcntl → command-fds crate fd_mappings()
  (handles CLOEXEC clearing safely, also fixes potential fd leak
  on spawn failure via OwnedFd RAII)

Only libc::umask remains as a single unavoidable unsafe site
(no safe Rust wrapper exists for the umask syscall).

Also updates AGENTS.md to remove stale SendCookie exception.
2026-03-14 16:01:54 -03:00
b3ca673b52 feat: add --update mode, --meta/--meta-plugin flags, streaming diff
- Add --update mode to modify tags and metadata for existing items by ID
- Add --meta key=value flag to set metadata during save/update
- Add --meta key (bare) to delete metadata keys or filter by existence
- Add --meta-plugin/-M name:{json} flag for plugin options via CLI
- Env meta plugin now uses options from --meta-plugin instead of only env vars
- Stream decompressed content to diff via /dev/fd pipes (no temp files)
- Wire --list-format CLI arg to settings (was parsed but ignored)
- Allow --info to accept tags (was restricted to numeric IDs only)
- Change DB meta filtering to HashMap<String, Option<String>> for exact match + key existence
- Fix fcntl error checking in diff pre_exec
- Fix README inaccuracies (delete by tag, nonexistent --digest flag, meta plugin key names)
2026-03-14 15:02:16 -03:00
17be6abaab refactor: streaming, security hardening, and MCP removal
Major overhaul of server architecture and security posture:

- Streaming: Unified all I/O through PIPESIZE (8192-byte) buffers.
  POST bodies stream via MpscReader through the save pipeline. GET
  content streams from disk via decompression to client. Removed
  save_item_with_reader, get_item_content_info, ChannelReader.
  413 responses keep partial items (nonfatal by design).

- Security: XSS protection in all HTML pages via html_escape crate.
  Security headers middleware (nosniff, frame deny, referrer policy).
  CORS tightened to explicit headers. Input validation for tags
  (256 chars), metadata (128/4096), pagination (10k cap). Config
  file reads use from_utf8_lossy. Generic error messages in HTML.
  Diff endpoint has 10 MB per-item cap. max_body_size config option.

- Panics eliminated: Path unwraps → proper error propagation.
  Mutex unwraps → map_err (registries) / expect with message (local).

- MCP removed: Deleted all MCP code, rmcp dependency, mcp feature.

- Docs: Updated README, DESIGN, AGENTS to reflect all changes.
2026-03-14 00:03:42 -03:00
560ba6e20c fix: count_bounded error counting, clippy if-let, auth test dedup, doc tests
- count_bounded: break on iterator error instead of counting errors as tokens
- collapse nested if-let chains with let-chains in auth middleware
- document JWT/Basic Auth as mutually exclusive
- TailTokensFilter::clone uses empty buffer (always pre-filter)
- fix 9 broken doc examples in server/common.rs
- remove 7 duplicate auth tests from auth.rs (covered by auth_tests.rs)
2026-03-13 22:04:38 -03:00
a07bb6b350 feat: plugin-declared parallel execution, switch to env_logger, update deps
Parallel execution (opt-in via MetaPlugin::parallel_safe):
- Add Send bound to MetaPlugin, parallel_safe() method (default false)
- Override to true in digest, tokens, exec, magic_file plugins
- MetaService: std::thread::scope for initialize_plugins and process_chunk
- Extract plugins via NullMetaPlugin sentinel + std::mem::replace (no unsafe)
- Panic tracking: join errors logged, NullMetaPlugin restored and finalized
- MetaPluginExec: Box<dyn Write> -> Box<dyn Write + Send>
- SendCookie wrapper for libmagic Cookie with unsafe impl Send

Logging (stderrlog -> env_logger):
- Custom format: [SSSSSS.mmm] LEVEL [module:] message (time-since-start ms)
- Default level: Warn (matches previous behavior)
- -v: Debug, -vv+: Trace, -q: off
- -vv+ shows module path

Maintenance:
- Bump deps: thiserror 2.0, config 0.15, dns-lookup 3.0, lz4_flex 0.12,
  ringbuf 0.4, rand 0.9, lazy_static 1.5, env_logger 0.11
- Update Cargo.lock (186 transitive packages)
- Clippy fixes: is_multiple_of, to_string_in_format_args, collapsible_if
- Fix double-counting bug in TokensMetaPlugin::update
- Fix schema description using plugin.description()

Co-Authored-By: opencode <noreply@opencode.ai>
2026-03-13 21:49:51 -03:00
e7d8a83369 feat: add plugin schema system, tokenizer cache, and config validation
- Add plugin schema types and runtime discovery for meta/filter plugins
- Rewrite --generate-config to use schema system instead of hardcoded types
- Add Settings::validate_config() for startup validation
- Cache tokenizer instances via static Lazy to avoid repeated BPE loading
- Add split_by_token_iter() and count_bounded() to Tokenizer
- Fix double-counting bug in TokensMetaPlugin when buffer < max_buffer_size
- Eliminate unnecessary allocations in token count methods
- Refactor token filters: remove Option<Tokenizer>, use iterator API
- Fix TailTokensFilter correctness: unbounded buffer instead of ring buffer
- Add encoding option to all token filters
- Add description() to MetaPlugin and FilterPlugin traits
- Fix unused_mut warning in compression engine (feature-gated code)

Co-Authored-By: code-review-bot <noreply@anthropic.com>
2026-03-13 20:23:17 -03:00
e672ec751e feat: add JWT auth, configurable username, switch password auth to Basic
Add server-side JWT authentication with permission-based access control
(read/write/delete claims). Password authentication now uses HTTP Basic
auth only (replacing Bearer). Add configurable username for both server
and client (--server-username/--client-username, defaults to "keep").

JWT secret supports file-based loading via --server-jwt-secret-file for
Docker secrets. OPTIONS preflight requests bypass auth. HEAD mapped to
read permission.

Co-Authored-By: opencode <noreply@opencode.ai>
2026-03-13 13:56:35 -03:00
b166477202 fix: harden security, eliminate panics, remove dead code, add Dockerfile
Security:
- Use constant-time password comparison (subtle crate) to prevent timing attacks
- Replace permissive CORS with configurable origin-restricted CORS
- Add TLS warning when password auth is used without HTTPS

Bug fixes:
- Convert MetaPlugin panics to anyhow::Result (get_meta_plugin, outputs_mut, options_mut)
- Replace item.id.unwrap() with proper error handling across 15 call sites
- Fix panic on unknown column type in list mode
- Fix conflicting PIPESIZE constant (was 8192 vs 65536, now unified to 8192)
- Add 256MB filter chain buffer limit to prevent OOM
- Gracefully skip unregistered plugins instead of panicking

Dead code removal:
- Delete unused filter parser files (filter_parser.rs, filter.pest, parser/ module)
- ~260 lines of dead PEG parser code removed

Code consolidation:
- Add is_content_binary_from_metadata() helper (was duplicated in 4 places)
- Simplify save_item_raw() to delegate to save_item_raw_streaming() (~90 lines removed)

Incomplete features:
- Populate filter_plugins in status output from global registry
- Add FallbackMagicFileMetaPlugin (was referenced but never implemented)
- Document init_plugins() as intentional no-op

Infrastructure:
- Add Dockerfile (static musl binary on scratch, 4.8MB)
- Add .dockerignore
- Add cors_origin to ServerConfig and config.rs
2026-03-13 07:57:36 -03:00
bee980605f feat: add HTTPS/TLS server support via rustls
Add optional TLS support for the server using axum-server with the
tls-rustls feature. When --server-cert and --server-key are provided
(and tls feature is enabled), the server binds with TLS instead of
plain HTTP.

Changes:
- Add axum-server dependency with optional tls-rustls feature
- New 'tls' feature flag (independent of 'server')
- --server-cert/--server-key CLI args gated behind tls feature
- ServerConfig extended with cert_file/key_file fields
- Conditional TLS/HTTP binding in server mod.rs
- Fix PathBuf::to_str().unwrap() panic risk -> to_string_lossy()
- Update README.md and DESIGN.md with TLS documentation
2026-03-12 22:18:42 -03:00
237a581429 fix: add server streaming support, fix pre-existing compilation errors
Server changes for client mode streaming:
- POST /api/item/ now streams body via async channel → ChannelReader
  → save_item_raw_streaming when compress=false or meta=false
- Add POST /api/item/{id}/meta endpoint for client-side metadata
- Add save_item_raw_streaming<R: Read> to SyncDataService
- Add add_item_meta to AsyncDataService

Fix pre-existing issues that were hidden behind swagger cfg gate:
- Remove #[cfg(feature = "swagger")] from item module so it compiles
  with just the server feature
- Fix parse_comma_tags usage (returns Vec, not Result)
- Fix TextDiff temporary value lifetime issue
- Fix io::Error::new → io::Error::other
- Fix ok_or_else → ok_or for Copy types
- Inline format args throughout server code
- Fix empty line after doc comment in pages.rs
- Add cfg_attr for unused_mut where mcp feature gates mutation
- Add type_complexity allow on create_auth_middleware
- Distinguish task error vs save error in spawn_blocking handlers

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
2026-03-12 18:02:56 -03:00
c5529bedbf feat: add client mode with streaming support
Add client mode enabling the keep CLI to connect to a remote keep
server over HTTP. Local plugins (compression, meta, filters) run on
the client; the server stores/retrieves binary blobs.

Architecture:
- Client save uses 3-thread streaming pipeline: reader thread (stdin
  → tee/stdout → hash → compress), OS pipe, streamer thread (pipe →
  chunked HTTP POST). Memory usage is O(PIPESIZE) regardless of data
  size.
- Server accepts compress=false, meta=false, decompress=false query
  params for granular control of server-side processing.
- Streaming body handling on server via async channel → sync reader
  bridge (ChannelReader).

Key additions:
- src/client.rs: KeepClient with post_stream() for chunked upload
- src/modes/client/: save, get, list, info, delete, diff, status
- --client-url / KEEP_CLIENT_URL configuration
- --client-password / KEEP_CLIENT_PASSWORD for auth
- os_pipe dependency for zero-copy pipe streaming

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
2026-03-12 18:01:36 -03:00
9b7cbd5244 fix: resolve doctest failures, database bugs, and remove dead code
- Fix all 96 doctest failures across 20 files by adding hidden imports and
  proper test setup (68 pass, 33 intentionally ignored)
- Fix set_item_tags: wrap in transaction and replace item.id.unwrap() with
  proper error handling
- Fix get_items_matching: replace N+1 per-item meta queries with batch
  get_meta_for_items() call
- Fix get_item_matching: apply meta filtering instead of ignoring the parameter
- Remove duplicate doc comment in store_meta
- Remove dead code files: plugin.rs, plugins.rs, binary_detection.rs
  (never declared as modules)
- Apply cargo fmt formatting fixes
- Add keep.db to .gitignore
2026-03-12 11:58:44 -03:00
8a8a6e1c4b fix: correct critical bugs and improve pipe streaming performance
Critical bug fixes:
- save_item now returns real Item from database, not a hardcoded fake
- AsyncDataService::save() reuses self.sync_service instead of creating redundant instance
- GenerateStatus trait signature mismatch fixed (CLI/API decoupling)

Performance improvements (pipe path untouched):
- CompressionEngine::open() returns Box<dyn Read + Send> enabling true streaming
- mode_get eliminates triple full-file read (was sampling then re-reading entire file)
- FilteringReader adds fast-path bypass when no filters, pre-allocates temp buffer
- text.rs meta plugin processes &[u8] slice directly, eliminates data.to_vec() clone

API correctness:
- Tag parse errors now return 400 instead of being silently discarded
- compute_diff uses similar crate (LCS-based) instead of naive positional comparison

Cleanup:
- Modernize string formatting (format!({x})) across codebase
- Remove redundant DB query in get mode
- Derive Debug/ToSchema on public types
- Delete placeholder test files with no real assertions
- Extract parse_comma_tags utility function
2026-03-11 20:45:05 -03:00
e8ea42506e feat: unify CLI and API with DataService trait
- Add DataService trait with streaming support for save/get operations
- Implement SyncDataService for CLI and AsyncDataService for API
- Add missing API endpoints: DELETE /api/item/{id}, GET /api/item/{id}/info, GET /api/diff
- Add GET /api/plugins/status endpoint
- Preserve stdin/stdout streaming performance via Read trait
2026-03-10 22:31:31 -03:00
Andrew Phillips
fdeb5f7951 Ugh 2026-02-19 13:57:39 -04:00
Andrew Phillips
0be54abe60 fix: resolve compilation errors and warnings
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 11:55:04 -03:00
Andrew Phillips
5d7c0658b9 fix: fix typos and improve error handling in compression engines
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 12:00:45 -03:00
Andrew Phillips
94e82e132a docs: Update function documentation to use block comments
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 16:53:50 -03:00
Andrew Phillips
ec95bc8028 refactor: Convert unused doc comments to regular comments in status_plugins.rs
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 16:53:05 -03:00
Andrew Phillips
b0e359989a fix: Resolve build errors and warnings, and update grammar
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 16:20:08 -03:00
Andrew Phillips
0d68f39c08 fix: Correct Pest grammar and update ItemService initialization
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 16:14:00 -03:00
Andrew Phillips
22cd07284b fix: Resolve compilation errors for multiple definitions and parser issues
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 16:12:11 -03:00
Andrew Phillips
11ec6cf01e docs: Add Rustdoc for pages.rs structs and functions
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 15:56:45 -03:00
Andrew Phillips
9f7534f0ae docs: Improve Rustdoc for compression engine, delete mode, and parser modules
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 15:40:04 -03:00
Andrew Phillips
6a79f0455c docs: Add Rustdoc to handle_status function in server status API
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 15:33:42 -03:00
Andrew Phillips
9f48d7980b docs: document src/services/types.rs, src/modes/common.rs, and src/services/error.rs
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 15:27:45 -03:00
Andrew Phillips
f3a4894699 docs: Add Rustdoc comments for info and mcp modules and grep plugin
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 14:30:54 -03:00
Andrew Phillips
7ee8ef6ce6 docs: Add Rustdoc comments to mode_status_plugins function
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 14:29:49 -03:00
Andrew Phillips
88c7290a7b docs: Add comprehensive documentation for modes, services, and plugins
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 14:29:16 -03:00
Andrew Phillips
d44f3fbb5b docs: Add Rustdoc comments for various structs and functions
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 14:23:12 -03:00
Andrew Phillips
d219f557db docs: Add Rustdoc to list mode's show_list_structured function
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 14:15:59 -03:00
Andrew Phillips
6cff3bd165 docs: Add Rustdoc examples to server common types and compression engine
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 14:05:24 -03:00
Andrew Phillips
538a67ee14 docs: Add comprehensive rustdoc to save mode functions and types
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:54:57 -03:00
Andrew Phillips
a7bcad40bb docs: Add comprehensive rustdoc to common module functions
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:53:21 -03:00
Andrew Phillips
c145974ce3 docs: Add Rustdoc comments to CompressionService methods
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:45:32 -03:00
Andrew Phillips
e4fc653397 docs: Add comprehensive Rustdoc comments to public APIs
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:36:41 -03:00
Andrew Phillips
58b5c8187b docs: Add Rustdoc for modules, functions, and structs
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:28:47 -03:00
Andrew Phillips
6ccea1872c fix: Correctly handle string slices and no newline
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:17:01 -03:00
Andrew Phillips
978dae32d8 docs: Add Rustdoc to code and comments to grammar file
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:13:22 -03:00
Andrew Phillips
c965e9f51c docs: Add rustdoc to all files, document arguments and returns
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:10:36 -03:00
Andrew Phillips
c5f43b56f2 fix: Restore missing lines and refactor setup_diff_paths_and_compression
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:05:38 -03:00
Andrew Phillips
ddafeb3a28 docs: Add rustdoc for server, diff, and gzip components
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 12:04:46 -03:00
Andrew Phillips
d9a36012bc feat: Make 'server' feature optional and add compile-time check
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 10:33:47 -03:00
Andrew Phillips
146bd2e569 feat: Make swagger an optional dependency, enabled by default
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 10:24:21 -03:00
Andrew Phillips
ea817ad629 refactor: Make router mutable in add_routes 2025-09-10 10:24:20 -03:00
Andrew Phillips
a010b232bf refactor: Remove deprecated server module file 2025-09-10 10:21:43 -03:00
Andrew Phillips
d5ff2e639d refactor: Deprecate src/modes/server.rs and move content to mod.rs
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 10:18:28 -03:00
Andrew Phillips
a0e0126ff3 fix: Correct module paths and conditional compilation for magic feature
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 10:11:25 -03:00
Andrew Phillips
39c3375cf5 refactor: Simplify server router merging with conditional compilation
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 10:00:30 -03:00
Andrew Phillips
8f3f6c05db feat: Make mcp support an optional feature, disabled by default
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-10 09:49:51 -03:00