refactor: decouple meta plugins from DB via SaveMetaFn callback, extract shared utilities

- Add SaveMetaFn callback pattern: meta plugins receive a closure instead of
  &Connection, enabling the same plugin code to work in local, client, and
  server contexts (collect-to-Vec, collect-to-HashMap, or direct DB write)
- Client save now runs meta plugins locally during streaming (smart client
  sets meta=false, server skips its own plugins)
- Add POST /api/item/{id}/update endpoint for re-running plugins on stored
  content without downloading compressed data
- Add client update mode (--update with --meta-plugin flags)
- Extract shared utilities: stream_copy, print_serialized, build_path_table,
  ensure_default_tag to reduce duplication across modes
- Add upsert_tag for idempotent tag addition (INSERT OR IGNORE)
- Add warn logging on save_meta lock failure in BaseMetaPlugin and MetaService
This commit is contained in:
2026-03-14 22:36:59 -03:00
parent fdc5f1d744
commit 5bad7ac7a6
39 changed files with 843 additions and 290 deletions

View File

@@ -33,7 +33,7 @@
- `modes/status.rs` - Show system status and capabilities
- `modes/server.rs` - REST HTTP/HTTPS server mode with OpenAPI documentation
- `modes/client.rs` - Client mode for remote server (streaming save, local decompression)
- `modes/common.rs` - Shared utilities for all modes
- `modes/common.rs` - Shared utilities for all modes (OutputFormat, table creation, `print_serialized`, `build_path_table`, `ensure_default_tag`, `render_item_info_table`, `render_list_table_with_format`)
### Database Module
- `db.rs` - SQLite database operations
@@ -49,24 +49,31 @@
- `compression_engine/program.rs` - External program wrapper
### Meta Plugin Module
- `meta_plugin.rs` - Trait and type definitions
- `meta_plugin.rs` - Trait and type definitions, `SaveMetaFn` callback type
- `meta_plugin/program.rs` - External program wrapper
- `meta_plugin/digest.rs` - Internal digest implementations
- `meta_plugin/system.rs` - System information metadata plugins
**SaveMetaFn Architecture**: Meta plugins are decoupled from direct DB access via a `SaveMetaFn` callback (`Arc<Mutex<dyn FnMut(&str, &str) + Send>>`). The callback is injected at `MetaService` construction and propagated to all plugins via `BaseMetaPlugin`. This enables:
- **Local mode**: Callback collects metadata into a `Vec`, written to DB after plugins finish
- **Client mode**: Callback collects into a `HashMap`, sent to server after streaming completes
- **Server mode**: Callback collects into a `Vec`, written to DB after plugins finish (same as local)
### Common Modules
- `common/is_binary.rs` - Binary file detection utilities
- `common/status.rs` - Status information generation
- `common/mod.rs` - `PIPESIZE` constant (8192), `stream_copy()` streaming utility
### Client Module
- `client.rs` - HTTP client wrapper (ureq-based, supports streaming POST)
- `modes/client/save.rs` - 3-thread streaming save (stdin → tee → compress → pipe → HTTP POST)
- `modes/client/save.rs` - 3-thread streaming save with local meta plugins (stdin → tee → compress → meta plugins → pipe → HTTP POST)
- `modes/client/get.rs` - Get with server-side raw fetch + local decompression
- `modes/client/list.rs` - List delegation to server
- `modes/client/info.rs` - Info delegation to server
- `modes/client/delete.rs` - Delete delegation to server
- `modes/client/diff.rs` - Diff delegation to server
- `modes/client/status.rs` - Status delegation to server
- `modes/client/update.rs` - Update delegation to server (sends plugin names/metadata/tags)
### Utility Modules
- `plugins.rs` - Shared plugin utilities
@@ -130,6 +137,7 @@
- `GET /api/item/` - Get a list of items as JSON. Optional params: `order=newest|oldest`, `start=0`, `count=100`, `tags=tag1,tag2`
- `POST /api/item/` - Add a new item (body: raw content, **streamed** through fixed-size 8192-byte buffers). Query params: `tags`, `metadata` (JSON), `compress=true|false`, `meta=true|false`
- `POST /api/item/<#>/meta` - Add metadata to an existing item (body: JSON object)
- `POST /api/item/<#>/update` - Re-run meta plugins on stored content. Query params: `plugins` (comma-separated), `metadata` (JSON), `tags` (comma-separated, idempotent)
- `DELETE /api/item/<#>` - Delete an item
- `GET /api/item/latest` - Return the latest item as JSON. Optional params: `tags=tag1,tag2`, `allow_binary=true|false`
- `GET /api/item/latest/meta` - Return the latest item metadata as JSON. Optional params: `tags=tag1,tag2`
@@ -148,8 +156,9 @@
- Conditional selection at startup: cert+key present → HTTPS, otherwise → HTTP
### Client/Server Protocol
- Smart clients (keep CLI) set `compress=false` and `meta=false` on POST, handling compression/metadata locally
- Smart clients (keep CLI) set `compress=false` and `meta=false` on POST, handling compression and meta plugins locally
- Dumb clients (curl) use defaults (`compress=true`, `meta=true`), server handles everything
- Smart client update: sends `plugins` param to server, server runs plugins on stored content (avoids downloading compressed data)
- GET responses include `X-Keep-Compression` header when `decompress=false`
- Streaming save uses chunked transfer encoding for constant memory usage
- **Universal streaming**: All server paths (POST, GET, diff) use `PIPESIZE` (8192) byte buffers