chore: code review cleanup — fixes, deps, docs
Fixed: - CLI help typo: "metatdata" -> "metadata" - Filter buffer OOM: check size before loading into memory Changed: - #[inline] on HTML escape helpers for hot path performance - Replaced once_cell and lazy_static with std::sync::LazyLock - Removed unused once_cell and lazy_static crate dependencies Refactored: - Added module-level doc to services/ module Documentation: - README.md: zstd is native not external, "none" -> "raw" - DESIGN.md: current schema and meta plugins section - CHANGELOG.md: Unreleased section populated
This commit is contained in:
41
DESIGN.md
41
DESIGN.md
@@ -117,7 +117,7 @@
|
||||
## Data Storage
|
||||
|
||||
### Database Schema
|
||||
- `items` table: id (primary key), ts (timestamp), size (optional), compression
|
||||
- `items` table: id (primary key), ts (timestamp), uncompressed_size (optional), compressed_size (optional), closed (boolean), compression
|
||||
- `tags` table: id (foreign key to items), name (tag name)
|
||||
- `metas` table: id (foreign key to items), name (meta key), value (meta value)
|
||||
- Indexes on tag names and meta names for faster queries
|
||||
@@ -178,26 +178,25 @@
|
||||
- None (no compression)
|
||||
|
||||
## Supported Meta Plugins
|
||||
- FileMagic - File type detection using file command
|
||||
- FileMime - MIME type detection using file command
|
||||
- FileEncoding - File encoding detection using file command
|
||||
- LineCount - Line count using wc command
|
||||
- WordCount - Word count using wc command
|
||||
- Cwd - Current working directory
|
||||
- Binary - Binary file detection
|
||||
- Uid - Current user ID
|
||||
- User - Current username
|
||||
- Gid - Current group ID
|
||||
- Group - Current group name
|
||||
- Shell - Shell path from SHELL environment variable
|
||||
- ShellPid - Shell process ID from PPID environment variable
|
||||
- KeepPid - Keep process ID
|
||||
- DigestSha256 - SHA-256 digest
|
||||
- DigestMd5 - MD5 digest using md5sum command
|
||||
- ReadTime - Time taken to read data
|
||||
- ReadRate - Rate of data reading
|
||||
- Hostname - System hostname
|
||||
- FullHostname - Fully qualified domain name
|
||||
|
||||
Meta plugins collect metadata during item save. Each plugin produces one or more key-value pairs:
|
||||
|
||||
- `magic_file` - File type detection using libmagic (when `magic` feature enabled)
|
||||
- `infer` - MIME type detection using infer crate (when `infer` feature enabled)
|
||||
- `tree_magic_mini` - MIME type detection using tree_magic_mini (when `tree_magic_mini` feature enabled)
|
||||
- `tokens` - LLM token counting using tiktoken (when `tokens` feature enabled)
|
||||
- `text` - Text analysis: line count, word count, char count, line average length
|
||||
- `digest` - SHA-256 and MD5 checksums
|
||||
- `hostname` - System hostname (full and short)
|
||||
- `cwd` - Current working directory
|
||||
- `user` - Current username and UID
|
||||
- `shell` - Shell path from SHELL environment variable
|
||||
- `shell_pid` - Shell process ID from PPID
|
||||
- `keep_pid` - Keep process ID
|
||||
- `env` - Arbitrary environment variables (via `KEEP_META_ENV_*` prefix)
|
||||
- `exec` - Execute external commands for custom metadata
|
||||
- `read_time` - Time taken to read content
|
||||
- `read_rate` - Content read rate (bytes/second)
|
||||
|
||||
## Testing Strategy
|
||||
- Unit tests for each module in `src/tests/`
|
||||
|
||||
Reference in New Issue
Block a user