asp/keep
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,541 Commits 1 Branch 0 Tags
17be6abaab0e4e7006fcdffe12b5e23ae6992252
Commit Graph

74 Commits

Author SHA1 Message Date
Andrew Phillips
17be6abaab refactor: streaming, security hardening, and MCP removal 
Major overhaul of server architecture and security posture:

- Streaming: Unified all I/O through PIPESIZE (8192-byte) buffers.
  POST bodies stream via MpscReader through the save pipeline. GET
  content streams from disk via decompression to client. Removed
  save_item_with_reader, get_item_content_info, ChannelReader.
  413 responses keep partial items (nonfatal by design).

- Security: XSS protection in all HTML pages via html_escape crate.
  Security headers middleware (nosniff, frame deny, referrer policy).
  CORS tightened to explicit headers. Input validation for tags
  (256 chars), metadata (128/4096), pagination (10k cap). Config
  file reads use from_utf8_lossy. Generic error messages in HTML.
  Diff endpoint has 10 MB per-item cap. max_body_size config option.

- Panics eliminated: Path unwraps → proper error propagation.
  Mutex unwraps → map_err (registries) / expect with message (local).

- MCP removed: Deleted all MCP code, rmcp dependency, mcp feature.

- Docs: Updated README, DESIGN, AGENTS to reflect all changes.
 2026-03-14 00:03:42 -03:00
Andrew Phillips
a07bb6b350 feat: plugin-declared parallel execution, switch to env_logger, update deps 
Parallel execution (opt-in via MetaPlugin::parallel_safe):
- Add Send bound to MetaPlugin, parallel_safe() method (default false)
- Override to true in digest, tokens, exec, magic_file plugins
- MetaService: std::thread::scope for initialize_plugins and process_chunk
- Extract plugins via NullMetaPlugin sentinel + std::mem::replace (no unsafe)
- Panic tracking: join errors logged, NullMetaPlugin restored and finalized
- MetaPluginExec: Box<dyn Write> -> Box<dyn Write + Send>
- SendCookie wrapper for libmagic Cookie with unsafe impl Send

Logging (stderrlog -> env_logger):
- Custom format: [SSSSSS.mmm] LEVEL [module:] message (time-since-start ms)
- Default level: Warn (matches previous behavior)
- -v: Debug, -vv+: Trace, -q: off
- -vv+ shows module path

Maintenance:
- Bump deps: thiserror 2.0, config 0.15, dns-lookup 3.0, lz4_flex 0.12,
  ringbuf 0.4, rand 0.9, lazy_static 1.5, env_logger 0.11
- Update Cargo.lock (186 transitive packages)
- Clippy fixes: is_multiple_of, to_string_in_format_args, collapsible_if
- Fix double-counting bug in TokensMetaPlugin::update
- Fix schema description using plugin.description()

Co-Authored-By: opencode <noreply@opencode.ai>
 2026-03-13 21:49:51 -03:00
Andrew Phillips
b166477202 fix: harden security, eliminate panics, remove dead code, add Dockerfile 
Security:
- Use constant-time password comparison (subtle crate) to prevent timing attacks
- Replace permissive CORS with configurable origin-restricted CORS
- Add TLS warning when password auth is used without HTTPS

Bug fixes:
- Convert MetaPlugin panics to anyhow::Result (get_meta_plugin, outputs_mut, options_mut)
- Replace item.id.unwrap() with proper error handling across 15 call sites
- Fix panic on unknown column type in list mode
- Fix conflicting PIPESIZE constant (was 8192 vs 65536, now unified to 8192)
- Add 256MB filter chain buffer limit to prevent OOM
- Gracefully skip unregistered plugins instead of panicking

Dead code removal:
- Delete unused filter parser files (filter_parser.rs, filter.pest, parser/ module)
- ~260 lines of dead PEG parser code removed

Code consolidation:
- Add is_content_binary_from_metadata() helper (was duplicated in 4 places)
- Simplify save_item_raw() to delegate to save_item_raw_streaming() (~90 lines removed)

Incomplete features:
- Populate filter_plugins in status output from global registry
- Add FallbackMagicFileMetaPlugin (was referenced but never implemented)
- Document init_plugins() as intentional no-op

Infrastructure:
- Add Dockerfile (static musl binary on scratch, 4.8MB)
- Add .dockerignore
- Add cors_origin to ServerConfig and config.rs
 2026-03-13 07:57:36 -03:00
Andrew Phillips
8a8a6e1c4b fix: correct critical bugs and improve pipe streaming performance 
Critical bug fixes:
- save_item now returns real Item from database, not a hardcoded fake
- AsyncDataService::save() reuses self.sync_service instead of creating redundant instance
- GenerateStatus trait signature mismatch fixed (CLI/API decoupling)

Performance improvements (pipe path untouched):
- CompressionEngine::open() returns Box<dyn Read + Send> enabling true streaming
- mode_get eliminates triple full-file read (was sampling then re-reading entire file)
- FilteringReader adds fast-path bypass when no filters, pre-allocates temp buffer
- text.rs meta plugin processes &[u8] slice directly, eliminates data.to_vec() clone

API correctness:
- Tag parse errors now return 400 instead of being silently discarded
- compute_diff uses similar crate (LCS-based) instead of naive positional comparison

Cleanup:
- Modernize string formatting (format!({x})) across codebase
- Remove redundant DB query in get mode
- Derive Debug/ToSchema on public types
- Delete placeholder test files with no real assertions
- Extract parse_comma_tags utility function
 2026-03-11 20:45:05 -03:00
Andrew Phillips
fdeb5f7951 Ugh 2026-02-19 13:57:39 -04:00
Andrew Phillips
cb1f330231 refactor: compose BaseMetaPlugin in remaining meta plugins 
Co-authored-by: aider (openai/andrew/openrouter/sonoma-sky-alpha) <aider@aider.chat>
 2025-09-12 12:21:03 -03:00
Andrew Phillips
059bde09e4 refactor: simplify filter plugin signatures by removing boxed parameters 2025-09-12 10:36:09 -03:00
Andrew Phillips
fed3722ef9 fix: Resolve compilation errors by refactoring imports and type annotations 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-09-03 09:39:22 -03:00
Andrew Phillips
15496345d9 feat: Implement registry for meta plugins 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-09-03 09:33:39 -03:00
Andrew Phillips
db5b3153c0 fix: implement debug and default for meta plugin components 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:55:29 -03:00
Andrew Phillips
1025f1bc01 feat: add thiserror and derive_more for error handling and boilerplate reduction 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:45:49 -03:00
Andrew Phillips
2fe9d593b1 refactor: remove manual Default implementation from DigestMetaPlugin 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:38:40 -03:00
Andrew Phillips
bd32c68056 fix: remove meta_name from DigestMetaPlugin default implementation 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:27:30 -03:00
Andrew Phillips
ea475386d6 fix: remove unused meta_name fields from meta plugin structs 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:26:00 -03:00
Andrew Phillips
a820078214 fix: add missing MetaPluginType imports 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:19:59 -03:00
Andrew Phillips
79fdf05d84 refactor: replace meta_name with MetaPluginType from strum 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 21:04:52 -03:00
Andrew Phillips
10cd22ee1a refactor: rename meta to plugin in table header 2025-08-27 20:07:29 -03:00
Andrew Phillips
42a10cc2d5 fix: ensure outputs are properly disabled in constructors 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 18:11:32 -03:00
Andrew Phillips
324b96d1e1 refactor: remove initialize() call and use plugin outputs directly 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 18:09:17 -03:00
Andrew Phillips
aa2534c901 fix: handle disabled digest outputs properly 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 18:00:24 -03:00
Andrew Phillips
97fb35b5f0 fix: update process_metadata_outputs to handle serde_yaml::Value 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 17:47:48 -03:00
Andrew Phillips
d013b60fc1 feat: set unused digest outputs to null based on method 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 17:42:55 -03:00
Andrew Phillips
14c5f926a4 feat: display default options for digest and magic_file plugins 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 16:00:06 -03:00
Andrew Phillips
fdcccc844e refactor: remove new_simple methods and replace with new(None, None) 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 13:59:37 -03:00
Andrew Phillips
b7bf9b20de refactor: remove redundant configure_options and default_options methods 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-27 10:43:30 -03:00
Andrew Phillips
6e8ff406c8 fix: remove unused digest field from Settings struct 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:52:07 -03:00
Andrew Phillips
560acc0235 fix: add missing std::io::Write import for md5::Context write method 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:48:58 -03:00
Andrew Phillips
af4f88a0fc fix: implement debug for hasher and fix md5 update method 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:48:16 -03:00
Andrew Phillips
8f5ec6381f fix: update md5 usage and remove unused imports 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:47:18 -03:00
Andrew Phillips
34c942e73b refactor: unify digest plugin type handling 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:44:56 -03:00
Andrew Phillips
53df5d9260 feat: implement single hasher selection for digest plugin 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:40:49 -03:00
Andrew Phillips
538d565341 feat: unify digest plugin types and always compute all hashes 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:38:35 -03:00
Andrew Phillips
cb83cc4b77 feat: add support for multiple hash methods in digest plugin 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 23:35:40 -03:00
Andrew Phillips
8c1fad68a5 feat: add cwd and user meta plugins and remove unused imports 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 20:44:01 -03:00
Andrew Phillips
0b0a601483 refactor: extract read_time and read_rate plugins from digest.rs 
Co-authored-by: aider (openai/andrew/openrouter/qwen/qwen3-coder) <aider@aider.chat>
 2025-08-26 20:20:40 -03:00
Andrew Phillips
04fb10d006 feat: add finalization tracking to meta plugins 
Co-authored-by: aider (openai/andrew/openrouter/qwen/qwen3-coder) <aider@aider.chat>
 2025-08-26 18:37:30 -03:00
Andrew Phillips
302fe010bd refactor: remove unused item_id fields from meta plugins 
Co-authored-by: aider (openai/andrew/openrouter/qwen/qwen3-coder) <aider@aider.chat>
 2025-08-26 18:16:16 -03:00
Andrew Phillips
3fb436dc44 fix: add missing meta_name implementation and fix compilation errors 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 18:05:02 -03:00
Andrew Phillips
e2b434e52c refactor: update ReadRateMetaPlugin to match new MetaPlugin trait signatures 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 17:59:52 -03:00
Andrew Phillips
c6c81088b8 fix: update meta plugin implementations to match trait signatures 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 17:59:16 -03:00
Andrew Phillips
f44866a2cc feat: update default is_internal() to return true and clean up overrides 
Co-authored-by: aider (openai/andrew/openrouter/deepseek/deepseek-chat-v3.1) <aider@aider.chat>
 2025-08-26 17:27:04 -03:00
Andrew Phillips
a0fcb86fce refactor: update response types to MetaPluginResponse 
Co-authored-by: aider (openai/andrew/openrouter/mistralai/mistral-medium-3.1) <aider@aider.chat>
 2025-08-26 16:17:34 -03:00
Andrew Phillips
ec5fde2771 refactor: update meta plugins to use MetaPluginResponse 
Co-authored-by: aider (openai/andrew/openrouter/mistralai/mistral-medium-3.1) <aider@aider.chat>
 2025-08-26 16:13:30 -03:00
Andrew Phillips
3021932eb6 feat: add read time tracking functionality 
Co-authored-by: aider (openai/andrew/openrouter/mistralai/mistral-medium-3.1) <aider@aider.chat>
 2025-08-26 15:37:35 -03:00
Andrew Phillips
128d98c4e3 feat: update digest and system plugins to return PluginResponse 
Co-authored-by: aider (openai/andrew/openrouter/mistralai/mistral-medium-3.1) <aider@aider.chat>
 2025-08-26 15:37:00 -03:00
Andrew Phillips
7b43827926 refactor: update meta plugins to use new trait interface 
Co-authored-by: aider (openai/andrew/openrouter/mistralai/mistral-medium-3.1) <aider@aider.chat>
 2025-08-26 15:35:57 -03:00
Andrew Phillips
449e47f721 fix: resolve stack overflow in meta plugin initialization by removing recursive calls 
Co-authored-by: aider (openai/andrew/openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
 2025-08-23 12:40:37 -03:00
Andrew Phillips
670d078eae fix: initialize final_options in meta plugin constructors 
Co-authored-by: aider (openai/andrew/openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
 2025-08-19 14:20:45 -03:00
Andrew Phillips
a3494ee831 feat: add options to meta plugins 
Co-authored-by: aider (openai/andrew/openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
 2025-08-19 14:18:59 -03:00
Andrew Phillips
73bfc064ea refactor: change meta_name to immutable reference and add debugging 
Co-authored-by: aider (openai/andrew/openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
 2025-08-19 14:12:53 -03:00