Commit Graph

1535 Commits

Author SHA1 Message Date
af1e0ca570 feat: expand Docker build to all features, add docker-compose.yml
- Build with server, mcp, swagger, client, tls features (all except magic)
- Add KEEP_* environment variable documentation and defaults
- Copy CA certificates for HTTPS client support in scratch image
- Add docker-compose.yml with keep-data and keep-config volumes
2026-03-13 10:08:28 -03:00
d5d58bc52c feat: add lz4 command fallback, remove unused magic.rs
- Add program-based lz4 command fallback when lz4 feature is disabled
- Feature-gate lz4.rs and lz4 tests to compile without lz4_flex
- Delete legacy magic.rs (unused, no feature gating, superseded by magic_file.rs)
2026-03-13 08:51:10 -03:00
b166477202 fix: harden security, eliminate panics, remove dead code, add Dockerfile
Security:
- Use constant-time password comparison (subtle crate) to prevent timing attacks
- Replace permissive CORS with configurable origin-restricted CORS
- Add TLS warning when password auth is used without HTTPS

Bug fixes:
- Convert MetaPlugin panics to anyhow::Result (get_meta_plugin, outputs_mut, options_mut)
- Replace item.id.unwrap() with proper error handling across 15 call sites
- Fix panic on unknown column type in list mode
- Fix conflicting PIPESIZE constant (was 8192 vs 65536, now unified to 8192)
- Add 256MB filter chain buffer limit to prevent OOM
- Gracefully skip unregistered plugins instead of panicking

Dead code removal:
- Delete unused filter parser files (filter_parser.rs, filter.pest, parser/ module)
- ~260 lines of dead PEG parser code removed

Code consolidation:
- Add is_content_binary_from_metadata() helper (was duplicated in 4 places)
- Simplify save_item_raw() to delegate to save_item_raw_streaming() (~90 lines removed)

Incomplete features:
- Populate filter_plugins in status output from global registry
- Add FallbackMagicFileMetaPlugin (was referenced but never implemented)
- Document init_plugins() as intentional no-op

Infrastructure:
- Add Dockerfile (static musl binary on scratch, 4.8MB)
- Add .dockerignore
- Add cors_origin to ServerConfig and config.rs
2026-03-13 07:57:36 -03:00
bee980605f feat: add HTTPS/TLS server support via rustls
Add optional TLS support for the server using axum-server with the
tls-rustls feature. When --server-cert and --server-key are provided
(and tls feature is enabled), the server binds with TLS instead of
plain HTTP.

Changes:
- Add axum-server dependency with optional tls-rustls feature
- New 'tls' feature flag (independent of 'server')
- --server-cert/--server-key CLI args gated behind tls feature
- ServerConfig extended with cert_file/key_file fields
- Conditional TLS/HTTP binding in server mod.rs
- Fix PathBuf::to_str().unwrap() panic risk -> to_string_lossy()
- Update README.md and DESIGN.md with TLS documentation
2026-03-12 22:18:42 -03:00
237a581429 fix: add server streaming support, fix pre-existing compilation errors
Server changes for client mode streaming:
- POST /api/item/ now streams body via async channel → ChannelReader
  → save_item_raw_streaming when compress=false or meta=false
- Add POST /api/item/{id}/meta endpoint for client-side metadata
- Add save_item_raw_streaming<R: Read> to SyncDataService
- Add add_item_meta to AsyncDataService

Fix pre-existing issues that were hidden behind swagger cfg gate:
- Remove #[cfg(feature = "swagger")] from item module so it compiles
  with just the server feature
- Fix parse_comma_tags usage (returns Vec, not Result)
- Fix TextDiff temporary value lifetime issue
- Fix io::Error::new → io::Error::other
- Fix ok_or_else → ok_or for Copy types
- Inline format args throughout server code
- Fix empty line after doc comment in pages.rs
- Add cfg_attr for unused_mut where mcp feature gates mutation
- Add type_complexity allow on create_auth_middleware
- Distinguish task error vs save error in spawn_blocking handlers

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
2026-03-12 18:02:56 -03:00
c5529bedbf feat: add client mode with streaming support
Add client mode enabling the keep CLI to connect to a remote keep
server over HTTP. Local plugins (compression, meta, filters) run on
the client; the server stores/retrieves binary blobs.

Architecture:
- Client save uses 3-thread streaming pipeline: reader thread (stdin
  → tee/stdout → hash → compress), OS pipe, streamer thread (pipe →
  chunked HTTP POST). Memory usage is O(PIPESIZE) regardless of data
  size.
- Server accepts compress=false, meta=false, decompress=false query
  params for granular control of server-side processing.
- Streaming body handling on server via async channel → sync reader
  bridge (ChannelReader).

Key additions:
- src/client.rs: KeepClient with post_stream() for chunked upload
- src/modes/client/: save, get, list, info, delete, diff, status
- --client-url / KEEP_CLIENT_URL configuration
- --client-password / KEEP_CLIENT_PASSWORD for auth
- os_pipe dependency for zero-copy pipe streaming

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
2026-03-12 18:01:36 -03:00
d2581358e9 docs: rewrite README, add LICENSE, remove outdated files
- Rewrite README.md with comprehensive documentation covering all
  features: compression engines, meta plugins, filter plugins, server
  mode, MCP integration, and configuration
- Add MIT LICENSE file
- Delete README.org (consolidated into README.md)
- Delete empty PLAN.md
- Update AGENTS.md with current build/test commands and conventions

Co-Authored-By: andrew/openrouter/hunter-alpha <noreply@opencode.ai>
2026-03-12 18:01:23 -03:00
79930f4b01 chore: remove outdated tool usage notes from AGENTS.md 2026-03-12 12:00:32 -03:00
9b7cbd5244 fix: resolve doctest failures, database bugs, and remove dead code
- Fix all 96 doctest failures across 20 files by adding hidden imports and
  proper test setup (68 pass, 33 intentionally ignored)
- Fix set_item_tags: wrap in transaction and replace item.id.unwrap() with
  proper error handling
- Fix get_items_matching: replace N+1 per-item meta queries with batch
  get_meta_for_items() call
- Fix get_item_matching: apply meta filtering instead of ignoring the parameter
- Remove duplicate doc comment in store_meta
- Remove dead code files: plugin.rs, plugins.rs, binary_detection.rs
  (never declared as modules)
- Apply cargo fmt formatting fixes
- Add keep.db to .gitignore
2026-03-12 11:58:44 -03:00
8a8a6e1c4b fix: correct critical bugs and improve pipe streaming performance
Critical bug fixes:
- save_item now returns real Item from database, not a hardcoded fake
- AsyncDataService::save() reuses self.sync_service instead of creating redundant instance
- GenerateStatus trait signature mismatch fixed (CLI/API decoupling)

Performance improvements (pipe path untouched):
- CompressionEngine::open() returns Box<dyn Read + Send> enabling true streaming
- mode_get eliminates triple full-file read (was sampling then re-reading entire file)
- FilteringReader adds fast-path bypass when no filters, pre-allocates temp buffer
- text.rs meta plugin processes &[u8] slice directly, eliminates data.to_vec() clone

API correctness:
- Tag parse errors now return 400 instead of being silently discarded
- compute_diff uses similar crate (LCS-based) instead of naive positional comparison

Cleanup:
- Modernize string formatting (format!({x})) across codebase
- Remove redundant DB query in get mode
- Derive Debug/ToSchema on public types
- Delete placeholder test files with no real assertions
- Extract parse_comma_tags utility function
2026-03-11 20:45:05 -03:00
e8ea42506e feat: unify CLI and API with DataService trait
- Add DataService trait with streaming support for save/get operations
- Implement SyncDataService for CLI and AsyncDataService for API
- Add missing API endpoints: DELETE /api/item/{id}, GET /api/item/{id}/info, GET /api/diff
- Add GET /api/plugins/status endpoint
- Preserve stdin/stdout streaming performance via Read trait
2026-03-10 22:31:31 -03:00
fb4c1a2b11 fix: add missing serde default to list_format field
Fixes deserialization failure in generate-config mode by adding
#[serde(default)] attribute to list_format field in Settings struct.
This allows the config library to provide sensible defaults when
no config file exists, resolving the error "missing field list_format".

Also unstages AGENT.md naming change since that's a different fix.
2026-03-09 20:13:55 -03:00
Andrew Phillips
fdeb5f7951 Ugh 2026-02-19 13:57:39 -04:00
Andrew Phillips
a72395fe83 refactor: simplify filter plugin interface to use &mut dyn Read/Write 2025-09-15 17:42:35 -03:00
Andrew Phillips
a8871a9575 docs: update build and formatting instructions in AGENT.md 2025-09-15 17:25:58 -03:00
Andrew Phillips
b538e2f8c1 feat: add magic file plugin with fallback to file command
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 12:29:33 -03:00
Andrew Phillips
82ec29f6a1 fix: resolve compilation errors by standardizing filter signatures and fixing ownership issues
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 12:28:51 -03:00
Andrew Phillips
02d9872b95 feat: implement MetaPluginExec for external command execution
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 12:21:30 -03:00
Andrew Phillips
cb1f330231 refactor: compose BaseMetaPlugin in remaining meta plugins
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 12:21:03 -03:00
Andrew Phillips
8693061338 refactor: use BaseMetaPlugin in ShellPidMetaPlugin
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 12:15:44 -03:00
Andrew Phillips
80e9457305 refactor: compose HostnameMetaPlugin with BaseMetaPlugin
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 12:15:26 -03:00
Andrew Phillips
9b85af439d refactor: update filter plugins to use boxed reader/writer parameters
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 11:56:11 -03:00
Andrew Phillips
0be54abe60 fix: resolve compilation errors and warnings
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 11:55:04 -03:00
Andrew Phillips
059bde09e4 refactor: simplify filter plugin signatures by removing boxed parameters 2025-09-12 10:36:09 -03:00
Andrew Phillips
9c354d5ef4 fix: complete magic file plugin implementation and error handling
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 10:27:16 -03:00
Andrew Phillips
84666155c4 feat: add fallback to file command when magic crate is disabled
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-12 10:26:06 -03:00
Andrew Phillips
27d3ecad04 refactor: simplify error handling and conditionals in meta plugins 2025-09-12 10:26:02 -03:00
Andrew Phillips
022bc70f53 docs: add AGENT.md and update compression engine module 2025-09-11 17:24:38 -03:00
Andrew Phillips
d27776ac23 chore: mark clippy fixes as completed in PLAN.md
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 12:34:49 -03:00
Andrew Phillips
fb70b7cc0b refactor: move doc comments above CompressionService struct
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 12:30:43 -03:00
Andrew Phillips
de1d546b67 chore: add clippy lint fixes plan to PLAN.md
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 12:14:10 -03:00
Andrew Phillips
91fc41f56e docs: remove outdated rustdoc plan 2025-09-11 12:14:05 -03:00
Andrew Phillips
5d7c0658b9 fix: fix typos and improve error handling in compression engines
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 12:00:45 -03:00
Andrew Phillips
f2951bf78e fix: resolve type mismatches and missing args module
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:58:47 -03:00
Andrew Phillips
4b67ff5763 fix: add missing clone_box impls for gzip and lz4 engines
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:58:19 -03:00
Andrew Phillips
dca1d6c6a4 fix: resolve compilation errors by adding Sync+Send bounds and fixing syntax
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:57:14 -03:00
Andrew Phillips
4dcbb7c942 feat: Add ProgramWriter utility struct for plugin communication
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:39:53 -03:00
Andrew Phillips
4c7b174dd5 feat: Add plugin system to allow external extensions 2025-09-11 11:39:52 -03:00
Andrew Phillips
b151998144 fix: Implement clone_box for CompressionEngineProgram
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:32:05 -03:00
Andrew Phillips
a23cf1bf3b fix: Close the unclosed delimiter in the compression engine trait
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:19:31 -03:00
Andrew Phillips
f7a73dd5e1 style: Remove trailing newline from trait definition 2025-09-11 11:19:30 -03:00
Andrew Phillips
0a267cf9ec docs: Update documentation for CompressionEngine trait methods
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:14:30 -03:00
Andrew Phillips
24e66ca75a refactor: Remove unused methods from CompressionEngine trait
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:10:51 -03:00
Andrew Phillips
0aaf22d4b6 docs: Fix missing docstring and implement cat and size methods
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:10:08 -03:00
Andrew Phillips
7be0b6735e fix: Remove stray doc comment lines that do not document anything
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:01:41 -03:00
Andrew Phillips
faeba25c53 feat: Add size method to CompressionEngine trait
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:00:43 -03:00
Andrew Phillips
b67018d981 refactor: Remove get_status_info from CompressionEngine trait
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 11:00:10 -03:00
Andrew Phillips
f2d340e778 refactor: Use is_internal instead of comparing against empty string for compression engine
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 10:58:19 -03:00
Andrew Phillips
08b5f284d3 feat: Add is_internal method to CompressionEngine trait
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 10:57:48 -03:00
Andrew Phillips
358df8acea fix: Restore methods for AutoFinishGzEncoder
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
2025-09-11 10:55:13 -03:00