refactor: streaming, security hardening, and MCP removal

Major overhaul of server architecture and security posture:

- Streaming: Unified all I/O through PIPESIZE (8192-byte) buffers.
  POST bodies stream via MpscReader through the save pipeline. GET
  content streams from disk via decompression to client. Removed
  save_item_with_reader, get_item_content_info, ChannelReader.
  413 responses keep partial items (nonfatal by design).

- Security: XSS protection in all HTML pages via html_escape crate.
  Security headers middleware (nosniff, frame deny, referrer policy).
  CORS tightened to explicit headers. Input validation for tags
  (256 chars), metadata (128/4096), pagination (10k cap). Config
  file reads use from_utf8_lossy. Generic error messages in HTML.
  Diff endpoint has 10 MB per-item cap. max_body_size config option.

- Panics eliminated: Path unwraps → proper error propagation.
  Mutex unwraps → map_err (registries) / expect with message (local).

- MCP removed: Deleted all MCP code, rmcp dependency, mcp feature.

- Docs: Updated README, DESIGN, AGENTS to reflect all changes.
This commit is contained in:
2026-03-14 00:03:42 -03:00
parent 560ba6e20c
commit 17be6abaab
51 changed files with 876 additions and 1309 deletions

View File

@@ -29,11 +29,27 @@ TERM=dumb cargo build --features server # With server feature
- Filter plugins must implement `filter()`, `clone_box()`, and `options()`
- Meta plugins extend `BaseMetaPlugin` for boilerplate reduction
- Enum string representations: `#[strum(serialize_all = "snake_case")]`
- Lint rules: `deny(clippy::all)`, `deny(unsafe_code)` (except `libc::umask` in main.rs)
- Feature flags: `default = ["magic", "lz4", "gzip"]`; optional: `server`, `mcp`, `swagger`
- Lint rules: `deny(clippy::all)`, `deny(unsafe_code)` (except `libc::umask` in main.rs, `unsafe impl Send` in `src/meta_plugin/magic_file.rs` for `SendCookie`)
- Feature flags: `default = ["magic", "lz4", "gzip"]`; optional: `server`, `swagger`
## Testing
- Tests in `src/tests/` mirroring `src/` structure; shared helpers in `src/tests/common/test_helpers.rs`
- Key helpers: `create_temp_dir()`, `create_temp_db()`, `test_compression_engine()`
- Test naming: `test_<feature>_<scenario>`
## Streaming Constraint
**At no point should the whole file be in memory at once.** All I/O must use fixed-size buffers:
- `PIPESIZE` = 8192 bytes (`src/common/mod.rs:10`)
- Server POST body streams through `save_item_raw_streaming` via `MpscReader`
- Server GET content streams via streaming reader (not `read_to_end`)
- When `max_body_size` is exceeded, return `413` but keep the partial item (nonfatal by design)
- Filter/meta plugins use `PIPESIZE`-sized buffers
## HTML Rendering
- Use `html_escape` crate for all user-controlled data in HTML pages
- `esc()` for text content, `esc_attr()` for HTML attributes
- Security headers middleware: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: strict-origin-when-cross-origin`