fix: resolve doctest failures, database bugs, and remove dead code

- Fix all 96 doctest failures across 20 files by adding hidden imports and
  proper test setup (68 pass, 33 intentionally ignored)
- Fix set_item_tags: wrap in transaction and replace item.id.unwrap() with
  proper error handling
- Fix get_items_matching: replace N+1 per-item meta queries with batch
  get_meta_for_items() call
- Fix get_item_matching: apply meta filtering instead of ignoring the parameter
- Remove duplicate doc comment in store_meta
- Remove dead code files: plugin.rs, plugins.rs, binary_detection.rs
  (never declared as modules)
- Apply cargo fmt formatting fixes
- Add keep.db to .gitignore
This commit is contained in:
2026-03-12 11:58:44 -03:00
parent 8a8a6e1c4b
commit 9b7cbd5244
30 changed files with 522 additions and 448 deletions

1
.gitignore vendored
View File

@@ -1,3 +1,4 @@
/target /target
.aider* .aider*
.crush .crush
keep.db

View File

@@ -1,84 +1,56 @@
# Agent Configuration # Agent Configuration
**IMPORTANT:** Prefer to use the `write_file` tool if the edit is for the majority of a file, or if you are correcting previous problems made edits from other tools. **IMPORTANT:** Prefer to use the `write_file` tool if the edit is for the majority of a file, or if you are correcting previous problems made edits from other tools.
## Tools
**IMPORTANT**: Be very careful when quoting text in tool calls to add the right amount of escaping. **IMPORTANT**: Be very careful when quoting text in tool calls to add the right amount of escaping.
**IMPORTANT:** When using `write_file`, you must provide the whole file, even the unchanged parts.
### `write_file` **IMPORTANT:** `xxx | keep | zzz` must be as performant as possible in all situations.
When editing files use the `write_file` tool to output the complete version of the corrected file.
**IMPORTANT**: You must provide the whole file to `write_file`, even the unchanged parts.
## Build/Test Commands ## Build/Test Commands
**IMPORTANT**: Do not run application, start the web server, or the trunk server. **IMPORTANT**: Do not run the application, start the web server, or the trunk server.
**IMPORTANT:** The cargo command cannot be ran in parallel. **IMPORTANT:** Cargo commands cannot be run in parallel. Prefix all commands with `TERM=dumb`.
```bash ```bash
# Check project TERM=dumb cargo check # Fast compile check
TERM=dumb cargo check TERM=dumb cargo build # Build project
TERM=dumb cargo test # Run all tests
# Build project TERM=dumb cargo test test_name # Run specific test by name substring
TERM=dumb cargo build TERM=dumb cargo test -- --nocapture # Verbose test output
TERM=dumb cargo fmt --check # Check formatting
# DO NOT RUN RUN APPLICATION (native) TERM=dumb cargo fmt # Apply formatting
# TERM=dumb cargo run TERM=dumb cargo clippy -- -D warnings # Lint (warnings are errors)
TERM=dumb cargo build --release # Release build
# Run all tests TERM=dumb cargo build --features server # With server feature
TERM=dumb cargo test
# Run specific test (by name substring)
TERM=dumb cargo test test_function_name
# Run specific test with verbose output
TERM=dumb cargo test test_function_name -- --nocapture
# Check formatting
TERM=dumb cargo fmt --check
# Apply formatting
TERM=dumb cargo fmt
# Lint with clippy
TERM=dumb cargo clippy -- -D warnings
# Build for release
TERM=dumb cargo build --release
``` ```
Prefix commands with `TERM=dumb` for consistent output. ## Code Conventions
## Code Style Guidelines - `anyhow::Result` for error handling; `thiserror` for custom error types (`src/services/error.rs`)
- Plugin traits: `CompressionEngine`, `FilterPlugin`, `MetaPlugin`
- Dynamic trait objects use `clone_box()` for `Clone` on `Box<dyn Trait>`
- Plugin registration uses `ctor` constructors at module load time
- Filter plugins must implement `filter()`, `clone_box()`, and `options()`
- Meta plugins extend `BaseMetaPlugin` for boilerplate reduction
- Enum string representations: `#[strum(serialize_all = "snake_case")]`
- Lint rules: `deny(clippy::all)`, `deny(unsafe_code)` (except `libc::umask` in main.rs)
- Feature flags: `default = ["magic", "lz4", "gzip"]`; optional: `server`, `mcp`, `swagger`
### Imports ## Testing
- Group imports in order: standard library, external crates, local modules
- Use explicit imports over glob imports (`use std::fs::File;` not `use std::fs::*;`)
### Documentation - Tests in `src/tests/` mirroring `src/` structure; shared helpers in `src/tests/common/test_helpers.rs`
- Document all public APIs with rustdoc - Key helpers: `create_temp_dir()`, `create_temp_db()`, `test_compression_engine()`
- Use examples in documentation only when helpful - Test naming: `test_<feature>_<scenario>`
## Procedures ## Procedures
### Fix build problems ### Fix build problems
1. Check the project: `TERM=dumb cargo check`. 1. `TERM=dumb cargo check`
2. If there are errors or warnings, create a new sub agent (expert rust developer) that uses the `TERM=dumb cargo check` output as input, planned using strategic thinking. 2. Read affected files, fix errors, preserve functionality, don't downgrade versions
a. Read all affected files 3. Prefer `write_file` for full file rewrites; repeat from step 1
d. Plan the fixes using strategic thinking:
- Read other files if they provide context or examples
- Look up relevant API information
- Do not downgrade versions
- Preserve functionality
- Use `TERM=dumb cargo fix` if appropriate.
- Prefer the `write_file` tool if there is evidence of double escaping
- You must generate the full file contents when using `write_file` or it will be truncated.
c. Return the list of files modified
3. If any files were modified, loop back to 1.
### Fix formatting ### Fix formatting
1. Format the project the project: `TERM=dumb cargo fmt` 1. `TERM=dumb cargo fmt`
2. Continue with the fix build problems procedure. 2. Continue with fix build problems procedure

View File

@@ -1,130 +0,0 @@
use crate::services::async_item_service::AsyncItemService;
use crate::services::error::CoreError;
use axum::http::StatusCode;
use std::collections::HashMap;
/// Check if content is binary when allow_binary is false
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service
/// * `item_id` - The ID of the item to check
/// * `metadata` - Metadata associated with the item
/// * `allow_binary` - Whether binary content is allowed
///
/// # Returns
///
/// * `Result<(), StatusCode>` -
/// * `Ok(())` if binary content is allowed or content is not binary
/// * `Err(StatusCode::BAD_REQUEST)` if binary content is not allowed and content is binary
/// Check if content is binary when allow_binary is false
///
/// Validates whether binary content is permitted for the item. If not allowed and content
/// is detected as binary, returns a bad request status. Uses metadata or streams content
/// for detection if needed.
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service for content access.
/// * `item_id` - The ID of the item to check.
/// * `metadata` - Metadata associated with the item (checked for "text" key).
/// * `allow_binary` - Whether binary content is allowed (bypasses check if true).
///
/// # Returns
///
/// * `Result<(), StatusCode>` -
/// * `Ok(())` if binary content is allowed or content is not binary.
/// * `Err(StatusCode::BAD_REQUEST)` if binary content is not allowed and content is binary.
///
/// # Errors
///
/// Propagates `StatusCode` for validation failures.
///
/// # Examples
///
/// ```
/// // If allow_binary = false and content is text
/// check_binary_content_allowed(&service, 1, &metadata, false)?;
/// // Succeeds
///
/// // If allow_binary = false and content is binary
/// // Returns Err(StatusCode::BAD_REQUEST)
/// ```
pub async fn check_binary_content_allowed(
item_service: &AsyncItemService,
item_id: i64,
metadata: &HashMap<String, String>,
allow_binary: bool,
) -> Result<(), StatusCode> {
if !allow_binary {
let is_binary = is_content_binary(item_service, item_id, metadata).await?;
if is_binary {
return Err(StatusCode::BAD_REQUEST);
}
}
Ok(())
}
/// Helper function to determine if content is binary
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service
/// * `item_id` - The ID of the item to check
/// * `metadata` - Metadata associated with the item
///
/// # Returns
///
/// * `Result<bool, StatusCode>` -
/// * `Ok(true)` if content is binary
/// * `Ok(false)` if content is text
/// * `Err(StatusCode)` if an error occurs during checking
/// Helper function to determine if content is binary
///
/// Checks existing "text" metadata first; if absent or unset, streams and analyzes
/// the content to detect binary nature. Logs warnings on detection failures.
///
/// # Arguments
///
/// * `item_service` - Reference to the async item service for content access.
/// * `item_id` - The ID of the item to check.
/// * `metadata` - Metadata associated with the item (checked for "text" key).
///
/// # Returns
///
/// * `Result<bool, StatusCode>` -
/// * `Ok(true)` if content is binary.
/// * `Ok(false)` if content is text.
/// * `Err(StatusCode)` if an error occurs during checking (e.g., INTERNAL_SERVER_ERROR).
///
/// # Errors
///
/// * `StatusCode::INTERNAL_SERVER_ERROR` if content access fails.
///
/// # Examples
///
/// ```
/// let is_bin = is_content_binary(&service, 1, &metadata).await?;
/// assert!(is_bin == false); // For text content
/// ```
pub async fn is_content_binary(
item_service: &AsyncItemService,
item_id: i64,
metadata: &HashMap<String, String>,
) -> Result<bool, StatusCode> {
if let Some(text_val) = metadata.get("text") {
Ok(text_val == "false")
} else {
// If text metadata isn't set, we need to check the content using streaming approach
match item_service.get_item_content_info_streaming(
item_id,
None
).await {
Ok((_, _, is_binary)) => Ok(is_binary),
Err(e) => {
log::warn!("Failed to get content info for binary check for item {}: {}", item_id, e);
Err(StatusCode::INTERNAL_SERVER_ERROR)
}
}
}
}

View File

@@ -11,12 +11,12 @@ use std::io::{Read, Write};
#[cfg(feature = "gzip")] #[cfg(feature = "gzip")]
use std::path::PathBuf; use std::path::PathBuf;
#[cfg(feature = "gzip")]
use flate2::Compression;
#[cfg(feature = "gzip")] #[cfg(feature = "gzip")]
use flate2::read::GzDecoder; use flate2::read::GzDecoder;
#[cfg(feature = "gzip")] #[cfg(feature = "gzip")]
use flate2::write::GzEncoder; use flate2::write::GzEncoder;
#[cfg(feature = "gzip")]
use flate2::Compression;
#[cfg(feature = "gzip")] #[cfg(feature = "gzip")]
use crate::compression_engine::CompressionEngine; use crate::compression_engine::CompressionEngine;

View File

@@ -1,4 +1,4 @@
use anyhow::{anyhow, Result}; use anyhow::{Result, anyhow};
use std::io; use std::io;
use std::io::{Read, Write}; use std::io::{Read, Write};
use std::path::PathBuf; use std::path::PathBuf;
@@ -28,8 +28,7 @@ use crate::compression_engine::program::CompressionEngineProgram;
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// use keep::compression_engine::CompressionType;
/// assert_eq!(CompressionType::GZip.to_string(), "gzip"); /// assert_eq!(CompressionType::GZip.to_string(), "gzip");
/// ``` /// ```
#[derive(Debug, Eq, PartialEq, Clone, EnumIter, Display, EnumString, enum_map::Enum)] #[derive(Debug, Eq, PartialEq, Clone, EnumIter, Display, EnumString, enum_map::Enum)]

View File

@@ -1,4 +1,4 @@
use anyhow::{anyhow, Context, Result}; use anyhow::{Context, Result, anyhow};
use log::*; use log::*;
use std::fs::File; use std::fs::File;
use std::io::{Read, Write}; use std::io::{Read, Write};

400
src/db.rs
View File

@@ -1,4 +1,4 @@
use anyhow::{Context, Error, Result}; use anyhow::{Context, Error, Result, anyhow};
use chrono::prelude::*; use chrono::prelude::*;
use lazy_static::lazy_static; use lazy_static::lazy_static;
use log::*; use log::*;
@@ -37,11 +37,11 @@ Automatic schema migrations are applied on database open using
# Usage # Usage
Open a connection: Open a connection:
``` ```ignore
let conn = db::open(PathBuf::from("keep.db"))?; let conn = db::open(PathBuf::from("keep.db"))?;
``` ```
Insert an item: Insert an item:
``` ```ignore
let item = db::Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() }; let item = db::Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
let id = db::insert_item(&conn, item)?; let id = db::insert_item(&conn, item)?;
``` ```
@@ -159,8 +159,14 @@ pub struct Meta {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db"); /// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?; /// let conn = db::open(db_path)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn open(path: PathBuf) -> Result<Connection, Error> { pub fn open(path: PathBuf) -> Result<Connection, Error> {
debug!("DB: Opening file: {path:?}"); debug!("DB: Opening file: {path:?}");
@@ -203,6 +209,13 @@ pub fn open(path: PathBuf) -> Result<Connection, Error> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { /// let item = Item {
/// id: None, /// id: None,
/// ts: Utc::now(), /// ts: Utc::now(),
@@ -211,6 +224,8 @@ pub fn open(path: PathBuf) -> Result<Connection, Error> {
/// }; /// };
/// let id = db::insert_item(&conn, item)?; /// let id = db::insert_item(&conn, item)?;
/// assert!(id > 0); /// assert!(id > 0);
/// # Ok(())
/// # }
/// ``` /// ```
pub fn insert_item(conn: &Connection, item: Item) -> Result<i64> { pub fn insert_item(conn: &Connection, item: Item) -> Result<i64> {
debug!("DB: Inserting item: {item:?}"); debug!("DB: Inserting item: {item:?}");
@@ -241,9 +256,18 @@ pub fn insert_item(conn: &Connection, item: Item) -> Result<i64> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use keep::compression_engine::CompressionType;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let compression = CompressionType::LZ4; /// let compression = CompressionType::LZ4;
/// let item = db::create_item(&conn, compression)?; /// let item = db::create_item(&conn, compression)?;
/// assert!(item.id.is_some()); /// assert!(item.id.is_some());
/// # Ok(())
/// # }
/// ``` /// ```
pub fn create_item( pub fn create_item(
conn: &Connection, conn: &Connection,
@@ -284,7 +308,18 @@ pub fn create_item(
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// db::add_tag(&conn, 1, "important")?; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// db::add_tag(&conn, item_id, "important")?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn add_tag(conn: &Connection, item_id: i64, tag_name: &str) -> Result<()> { pub fn add_tag(conn: &Connection, item_id: i64, tag_name: &str) -> Result<()> {
let tag = Tag { let tag = Tag {
@@ -317,7 +352,18 @@ pub fn add_tag(conn: &Connection, item_id: i64, tag_name: &str) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// db::add_meta(&conn, 1, "mime_type", "text/plain")?; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// db::add_meta(&conn, item_id, "mime_type", "text/plain")?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn add_meta(conn: &Connection, item_id: i64, name: &str, value: &str) -> Result<()> { pub fn add_meta(conn: &Connection, item_id: i64, name: &str, value: &str) -> Result<()> {
let meta = Meta { let meta = Meta {
@@ -349,8 +395,17 @@ pub fn add_meta(conn: &Connection, item_id: i64, name: &str, value: &str) -> Res
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), size: Some(1024), compression: "lz4".to_string(), ts: Utc::now() }; /// let item = Item { id: Some(1), size: Some(1024), compression: "lz4".to_string(), ts: Utc::now() };
/// db::update_item(&conn, item)?; /// db::update_item(&conn, item)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn update_item(conn: &Connection, item: Item) -> Result<()> { pub fn update_item(conn: &Connection, item: Item) -> Result<()> {
debug!("DB: Updating item: {item:?}"); debug!("DB: Updating item: {item:?}");
@@ -382,8 +437,17 @@ pub fn update_item(conn: &Connection, item: Item) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), ..default_item() }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// db::delete_item(&conn, item)?; /// db::delete_item(&conn, item)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn delete_item(conn: &Connection, item: Item) -> Result<()> { pub fn delete_item(conn: &Connection, item: Item) -> Result<()> {
debug!("DB: Deleting item: {item:?}"); debug!("DB: Deleting item: {item:?}");
@@ -412,8 +476,16 @@ pub fn delete_item(conn: &Connection, item: Item) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() }; /// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() };
/// db::query_delete_meta(&conn, meta)?; /// db::query_delete_meta(&conn, meta)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn query_delete_meta(conn: &Connection, meta: Meta) -> Result<()> { pub fn query_delete_meta(conn: &Connection, meta: Meta) -> Result<()> {
debug!("DB: Deleting meta: {meta:?}"); debug!("DB: Deleting meta: {meta:?}");
@@ -445,8 +517,19 @@ pub fn query_delete_meta(conn: &Connection, meta: Meta) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let meta = Meta { id: 1, name: "mime_type".to_string(), value: "text/plain".to_string() }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let meta = Meta { id: item_id, name: "mime_type".to_string(), value: "text/plain".to_string() };
/// db::query_upsert_meta(&conn, meta)?; /// db::query_upsert_meta(&conn, meta)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn query_upsert_meta(conn: &Connection, meta: Meta) -> Result<()> { pub fn query_upsert_meta(conn: &Connection, meta: Meta) -> Result<()> {
debug!("DB: Inserting meta: {meta:?}"); debug!("DB: Inserting meta: {meta:?}");
@@ -478,41 +561,24 @@ pub fn query_upsert_meta(conn: &Connection, meta: Meta) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// // Insert new metadata /// // Insert new metadata
/// let meta = Meta { id: 1, name: "source".to_string(), value: "cli".to_string() }; /// let meta = Meta { id: item_id, name: "source".to_string(), value: "cli".to_string() };
/// db::store_meta(&conn, meta)?; /// db::store_meta(&conn, meta)?;
/// ///
/// // Delete metadata with empty value /// // Delete metadata with empty value
/// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() }; /// let meta = Meta { id: item_id, name: "temp".to_string(), value: "".to_string() };
/// db::store_meta(&conn, meta)?;
/// ```
/// Stores a metadata entry, deleting it if the value is empty.
///
/// Handles both insertion/update and deletion based on value presence.
///
/// # Arguments
///
/// * `conn` - Database connection.
/// * `meta` - Metadata entry to store (empty value triggers deletion).
///
/// # Returns
///
/// * `Result<()>` - Success or error if the operation fails.
///
/// # Errors
///
/// * Database errors during insert/update/delete.
///
/// # Examples
///
/// ```
/// // Insert new metadata
/// let meta = Meta { id: 1, name: "source".to_string(), value: "cli".to_string() };
/// db::store_meta(&conn, meta)?;
///
/// // Delete metadata with empty value
/// let meta = Meta { id: 1, name: "temp".to_string(), value: "".to_string() };
/// db::store_meta(&conn, meta)?; /// db::store_meta(&conn, meta)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn store_meta(conn: &Connection, meta: Meta) -> Result<()> { pub fn store_meta(conn: &Connection, meta: Meta) -> Result<()> {
if meta.value.is_empty() { if meta.value.is_empty() {
@@ -544,8 +610,19 @@ pub fn store_meta(conn: &Connection, meta: Meta) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let tag = Tag { id: 1, name: "work".to_string() }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let tag = Tag { id: item_id, name: "work".to_string() };
/// db::insert_tag(&conn, tag)?; /// db::insert_tag(&conn, tag)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn insert_tag(conn: &Connection, tag: Tag) -> Result<()> { pub fn insert_tag(conn: &Connection, tag: Tag) -> Result<()> {
debug!("DB: Inserting tag: {tag:?}"); debug!("DB: Inserting tag: {tag:?}");
@@ -576,8 +653,17 @@ pub fn insert_tag(conn: &Connection, tag: Tag) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), .. }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// db::delete_item_tags(&conn, item)?; /// db::delete_item_tags(&conn, item)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn delete_item_tags(conn: &Connection, item: Item) -> Result<()> { pub fn delete_item_tags(conn: &Connection, item: Item) -> Result<()> {
debug!("DB: Deleting all item tags: {item:?}"); debug!("DB: Deleting all item tags: {item:?}");
@@ -607,24 +693,38 @@ pub fn delete_item_tags(conn: &Connection, item: Item) -> Result<()> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), .. }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let item = Item { id: Some(item_id), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let tags = vec!["project_a".to_string(), "urgent".to_string()]; /// let tags = vec!["project_a".to_string(), "urgent".to_string()];
/// db::set_item_tags(&conn, item, &tags)?; /// db::set_item_tags(&conn, item, &tags)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn set_item_tags(conn: &Connection, item: Item, tags: &Vec<String>) -> Result<()> { pub fn set_item_tags(conn: &Connection, item: Item, tags: &Vec<String>) -> Result<()> {
debug!("DB: Setting tags for item: {item:?} ?{tags:?}"); debug!("DB: Setting tags for item: {item:?} ?{tags:?}");
delete_item_tags(conn, item.clone())?; let item_id = item
let item_id = item.id.unwrap(); .id
.ok_or_else(|| anyhow!("Item ID is required for set_item_tags"))?;
let tx = conn.unchecked_transaction()?;
delete_item_tags(&tx, item)?;
for tag_name in tags { for tag_name in tags {
insert_tag( insert_tag(
conn, &tx,
Tag { Tag {
id: item_id, id: item_id,
name: tag_name.to_string(), name: tag_name.to_string(),
}, },
)?; )?;
} }
tx.commit()?;
Ok(()) Ok(())
} }
@@ -647,8 +747,16 @@ pub fn set_item_tags(conn: &Connection, item: Item, tags: &Vec<String>) -> Resul
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let all_items = db::query_all_items(&conn)?; /// let all_items = db::query_all_items(&conn)?;
/// assert!(all_items.len() >= 0); /// assert!(all_items.len() >= 0);
/// # Ok(())
/// # }
/// ``` /// ```
pub fn query_all_items(conn: &Connection) -> Result<Vec<Item>> { pub fn query_all_items(conn: &Connection) -> Result<Vec<Item>> {
debug!("DB: Querying all items"); debug!("DB: Querying all items");
@@ -691,8 +799,16 @@ pub fn query_all_items(conn: &Connection) -> Result<Vec<Item>> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let tags = vec!["work".to_string(), "urgent".to_string()]; /// let tags = vec!["work".to_string(), "urgent".to_string()];
/// let tagged_items = db::query_tagged_items(&conn, &tags)?; /// let tagged_items = db::query_tagged_items(&conn, &tags)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn query_tagged_items<'a>(conn: &'a Connection, tags: &'a Vec<String>) -> Result<Vec<Item>> { pub fn query_tagged_items<'a>(conn: &'a Connection, tags: &'a Vec<String>) -> Result<Vec<Item>> {
debug!("DB: Querying tagged items: {tags:?}"); debug!("DB: Querying tagged items: {tags:?}");
@@ -751,7 +867,15 @@ pub fn query_tagged_items<'a>(conn: &'a Connection, tags: &'a Vec<String>) -> Re
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let items = db::get_items(&conn)?; /// let items = db::get_items(&conn)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_items(conn: &Connection) -> Result<Vec<Item>> { pub fn get_items(conn: &Connection) -> Result<Vec<Item>> {
debug!("DB: Getting all items"); debug!("DB: Getting all items");
@@ -780,9 +904,18 @@ pub fn get_items(conn: &Connection) -> Result<Vec<Item>> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::collections::HashMap;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let tags = vec!["project".to_string()]; /// let tags = vec!["project".to_string()];
/// let meta = HashMap::from([("status".to_string(), "active".to_string())]); /// let meta = HashMap::from([("status".to_string(), "active".to_string())]);
/// let matching = db::get_items_matching(&conn, &tags, &meta)?; /// let matching = db::get_items_matching(&conn, &tags, &meta)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_items_matching( pub fn get_items_matching(
conn: &Connection, conn: &Connection,
@@ -801,44 +934,35 @@ pub fn get_items_matching(
Ok(items) Ok(items)
} else { } else {
debug!("DB: Filtering on meta"); debug!("DB: Filtering on meta");
let mut filtered_items: Vec<Item> = Vec::new(); let item_ids: Vec<i64> = items.iter().filter_map(|i| i.id).collect();
for item in items.iter() { let meta_map = get_meta_for_items(conn, &item_ids)?;
let mut item_ok = true; let filtered_items: Vec<Item> = items
let mut item_meta: HashMap<String, String> = HashMap::new(); .into_iter()
for meta in get_item_meta(conn, item)? { .filter(|item| {
item_meta.insert(meta.name, meta.value); let item_id = match item.id {
} Some(id) => id,
None => return false,
debug!("DB: Matching: {item:?}: {item_meta:?}"); };
let item_meta = match meta_map.get(&item_id) {
for (k, v) in meta.iter() { Some(m) => m,
match item_meta.get(k) { None => return false,
Some(value) => item_ok = v.eq(value), };
None => item_ok = false, meta.iter().all(|(k, v)| item_meta.get(k) == Some(v))
} })
.collect();
if !item_ok {
break;
}
}
if item_ok {
filtered_items.push(item.clone());
}
}
Ok(filtered_items) Ok(filtered_items)
} }
} }
/// Gets a single item matching specified tags. /// Gets a single item matching specified tags and metadata.
/// ///
/// Returns the most recent item matching all tags (ignores metadata). /// Returns the most recent item matching all tags and metadata.
/// ///
/// # Arguments /// # Arguments
/// ///
/// * `conn` - Database connection. /// * `conn` - Database connection.
/// * `tags` - Vector of tag names to match (all must match). /// * `tags` - Vector of tag names to match (all must match).
/// * `_meta` - Unused metadata parameter (for API consistency). /// * `meta` - HashMap of metadata key-value pairs to match (exact match).
/// ///
/// # Returns /// # Returns
/// ///
@@ -851,51 +975,26 @@ pub fn get_items_matching(
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::collections::HashMap;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let tags = vec!["latest".to_string()]; /// let tags = vec!["latest".to_string()];
/// let item = db::get_item_matching(&conn, &tags, &HashMap::new())?; /// let item = db::get_item_matching(&conn, &tags, &HashMap::new())?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item_matching( pub fn get_item_matching(
conn: &Connection, conn: &Connection,
tags: &Vec<String>, tags: &Vec<String>,
_meta: &HashMap<String, String>, meta: &HashMap<String, String>,
) -> Result<Option<Item>> { ) -> Result<Option<Item>> {
debug!("DB: Get item matching tags: {tags:?}"); debug!("DB: Get item matching tags: {tags:?}, meta: {meta:?}");
let mut statement = conn let items = get_items_matching(conn, tags, meta)?;
.prepare_cached( Ok(items.into_iter().last())
"
SELECT items.id,
items.ts,
items.size,
items.compression,
count(sel.id) as score
FROM items,
(SELECT tags.id FROM tags WHERE tags.name IN rarray(?1)) as sel
WHERE items.id = sel.id
GROUP BY items.id
HAVING score = ?2
ORDER BY items.id DESC
LIMIT 1",
)
.context("Problem preparing SQL statement")?;
let tags_values: Vec<rusqlite::types::Value> = tags
.iter()
.map(|s| rusqlite::types::Value::from(s.clone()))
.collect();
let tags_ptr = Rc::new(tags_values);
let mut rows = statement.query(params![&tags_ptr, &tags.len()])?;
match rows.next()? {
Some(row) => Ok(Some(Item {
id: row.get(0)?,
ts: row.get(1)?,
size: row.get(2)?,
compression: row.get(3)?,
})),
None => Ok(None),
}
} }
/// Gets an item by its ID. /// Gets an item by its ID.
@@ -918,8 +1017,19 @@ pub fn get_item_matching(
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = db::get_item(&conn, 1)?; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: None, ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let item_id = db::insert_item(&conn, item)?;
/// let item = db::get_item(&conn, item_id)?;
/// assert!(item.is_some()); /// assert!(item.is_some());
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item(conn: &Connection, item_id: i64) -> Result<Option<Item>> { pub fn get_item(conn: &Connection, item_id: i64) -> Result<Option<Item>> {
debug!("DB: Getting item {item_id:?}"); debug!("DB: Getting item {item_id:?}");
@@ -964,7 +1074,15 @@ pub fn get_item(conn: &Connection, item_id: i64) -> Result<Option<Item>> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let latest = db::get_item_last(&conn)?; /// let latest = db::get_item_last(&conn)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item_last(conn: &Connection) -> Result<Option<Item>> { pub fn get_item_last(conn: &Connection) -> Result<Option<Item>> {
debug!("DB: Getting last item"); debug!("DB: Getting last item");
@@ -1011,8 +1129,17 @@ pub fn get_item_last(conn: &Connection) -> Result<Option<Item>> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), .. }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let tags = db::get_item_tags(&conn, &item)?; /// let tags = db::get_item_tags(&conn, &item)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item_tags(conn: &Connection, item: &Item) -> Result<Vec<Tag>> { pub fn get_item_tags(conn: &Connection, item: &Item) -> Result<Vec<Tag>> {
debug!("DB: Getting tags for item: {item:?}"); debug!("DB: Getting tags for item: {item:?}");
@@ -1053,8 +1180,17 @@ pub fn get_item_tags(conn: &Connection, item: &Item) -> Result<Vec<Tag>> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), .. }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let meta = db::get_item_meta(&conn, &item)?; /// let meta = db::get_item_meta(&conn, &item)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item_meta(conn: &Connection, item: &Item) -> Result<Vec<Meta>> { pub fn get_item_meta(conn: &Connection, item: &Item) -> Result<Vec<Meta>> {
debug!("DB: Getting item meta: {item:?}"); debug!("DB: Getting item meta: {item:?}");
@@ -1097,8 +1233,17 @@ pub fn get_item_meta(conn: &Connection, item: &Item) -> Result<Vec<Meta>> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), .. }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let meta = db::get_item_meta_name(&conn, &item, "mime_type".to_string())?; /// let meta = db::get_item_meta_name(&conn, &item, "mime_type".to_string())?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Result<Option<Meta>> { pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Result<Option<Meta>> {
debug!("DB: Getting item meta name: {item:?} {name:?}"); debug!("DB: Getting item meta name: {item:?} {name:?}");
@@ -1138,8 +1283,17 @@ pub fn get_item_meta_name(conn: &Connection, item: &Item, name: String) -> Resul
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let item = Item { id: Some(1), .. }; /// # use keep::db;
/// # use keep::db::*;
/// # use chrono::Utc;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let item = Item { id: Some(1), ts: Utc::now(), size: None, compression: "lz4".to_string() };
/// let value = db::get_item_meta_value(&conn, &item, "source".to_string())?; /// let value = db::get_item_meta_value(&conn, &item, "source".to_string())?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_item_meta_value(conn: &Connection, item: &Item, name: String) -> Result<Option<String>> { pub fn get_item_meta_value(conn: &Connection, item: &Item, name: String) -> Result<Option<String>> {
debug!("DB: Getting item meta value: {item:?} {name:?}"); debug!("DB: Getting item meta value: {item:?} {name:?}");
@@ -1174,8 +1328,16 @@ pub fn get_item_meta_value(conn: &Connection, item: &Item, name: String) -> Resu
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let ids = vec![1, 2, 3]; /// let ids = vec![1, 2, 3];
/// let tags_map = db::get_tags_for_items(&conn, &ids)?; /// let tags_map = db::get_tags_for_items(&conn, &ids)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_tags_for_items( pub fn get_tags_for_items(
conn: &Connection, conn: &Connection,
@@ -1233,8 +1395,16 @@ pub fn get_tags_for_items(
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::db;
/// # use keep::db::*;
/// # use std::path::PathBuf;
/// # fn main() -> anyhow::Result<()> {
/// let db_path = PathBuf::from("keep.db");
/// let conn = db::open(db_path)?;
/// let ids = vec![1, 2, 3]; /// let ids = vec![1, 2, 3];
/// let meta_map = db::get_meta_for_items(&conn, &ids)?; /// let meta_map = db::get_meta_for_items(&conn, &ids)?;
/// # Ok(())
/// # }
/// ``` /// ```
pub fn get_meta_for_items( pub fn get_meta_for_items(
conn: &Connection, conn: &Connection,

View File

@@ -34,7 +34,9 @@ pub struct GrepFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::GrepFilter;
/// let filter = GrepFilter::new("error|warn".to_string())?; /// let filter = GrepFilter::new("error|warn".to_string())?;
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
impl GrepFilter { impl GrepFilter {
pub fn new(pattern: String) -> Result<Self> { pub fn new(pattern: String) -> Result<Self> {
@@ -65,7 +67,13 @@ impl GrepFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use std::io::{Read, Write, Cursor};
/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
/// # let mut filter = GrepFilter::new("error".to_string())?;
/// let mut input: &mut dyn Read = &mut Cursor::new(b"error: something failed\nok: all good\n");
/// let mut output = Vec::new();
/// filter.filter(&mut input, &mut output)?; /// filter.filter(&mut input, &mut output)?;
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
impl FilterPlugin for GrepFilter { impl FilterPlugin for GrepFilter {
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> { fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -90,6 +98,8 @@ impl FilterPlugin for GrepFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
/// let filter = GrepFilter::new("test".to_string()).unwrap();
/// let cloned = filter.clone_box(); /// let cloned = filter.clone_box();
/// ``` /// ```
fn clone_box(&self) -> Box<dyn FilterPlugin> { fn clone_box(&self) -> Box<dyn FilterPlugin> {
@@ -109,6 +119,8 @@ impl FilterPlugin for GrepFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::{FilterPlugin, GrepFilter};
/// let filter = GrepFilter::new("test".to_string()).unwrap();
/// let opts = filter.options(); /// let opts = filter.options();
/// assert_eq!(opts.len(), 1); /// assert_eq!(opts.len(), 1);
/// assert!(opts[0].required); /// assert!(opts[0].required);

View File

@@ -37,8 +37,8 @@ impl HeadBytesFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::HeadBytesFilter;
/// let filter = HeadBytesFilter::new(1024); /// let filter = HeadBytesFilter::new(1024);
/// assert_eq!(filter.remaining, 1024);
/// ``` /// ```
pub fn new(count: usize) -> Self { pub fn new(count: usize) -> Self {
Self { remaining: count } Self { remaining: count }
@@ -66,8 +66,14 @@ impl HeadBytesFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// // Assuming a filter chain with head_bytes(5) /// # use std::io::{Read, Write, Cursor};
/// // Input "Hello World" becomes "Hello" /// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
/// # let mut filter = HeadBytesFilter::new(5);
/// let mut input: &mut dyn Read = &mut Cursor::new(b"Hello World");
/// let mut output = Vec::new();
/// filter.filter(&mut input, &mut output)?;
/// assert_eq!(output, b"Hello");
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
impl FilterPlugin for HeadBytesFilter { impl FilterPlugin for HeadBytesFilter {
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> { fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -95,6 +101,14 @@ impl FilterPlugin for HeadBytesFilter {
/// # Returns /// # Returns
/// ///
/// A new `Box<dyn FilterPlugin>` clone. /// A new `Box<dyn FilterPlugin>` clone.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
/// let filter = HeadBytesFilter::new(100);
/// let cloned = filter.clone_box();
/// ```
fn clone_box(&self) -> Box<dyn FilterPlugin> { fn clone_box(&self) -> Box<dyn FilterPlugin> {
Box::new(Self { Box::new(Self {
remaining: self.remaining, remaining: self.remaining,
@@ -108,6 +122,17 @@ impl FilterPlugin for HeadBytesFilter {
/// # Returns /// # Returns
/// ///
/// Vector of `FilterOption` describing parameters. /// Vector of `FilterOption` describing parameters.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadBytesFilter};
/// let filter = HeadBytesFilter::new(100);
/// let opts = filter.options();
/// assert_eq!(opts.len(), 1);
/// assert_eq!(opts[0].name, "count");
/// assert!(opts[0].required);
/// ```
fn options(&self) -> Vec<FilterOption> { fn options(&self) -> Vec<FilterOption> {
vec![FilterOption { vec![FilterOption {
name: "count".to_string(), name: "count".to_string(),
@@ -144,8 +169,8 @@ impl HeadLinesFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::HeadLinesFilter;
/// let filter = HeadLinesFilter::new(3); /// let filter = HeadLinesFilter::new(3);
/// assert_eq!(filter.remaining, 3);
/// ``` /// ```
pub fn new(count: usize) -> Self { pub fn new(count: usize) -> Self {
Self { remaining: count } Self { remaining: count }
@@ -172,8 +197,14 @@ impl HeadLinesFilter {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// // Assuming a filter chain with head_lines(2) /// # use std::io::{Read, Write, Cursor};
/// // Input: "Line1\nLine2\nLine3" becomes "Line1\nLine2\n" /// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
/// # let mut filter = HeadLinesFilter::new(2);
/// let mut input: &mut dyn Read = &mut Cursor::new(b"Line1\nLine2\nLine3\n");
/// let mut output = Vec::new();
/// filter.filter(&mut input, &mut output)?;
/// assert_eq!(output, b"Line1\nLine2\n");
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
impl FilterPlugin for HeadLinesFilter { impl FilterPlugin for HeadLinesFilter {
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> { fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -200,6 +231,14 @@ impl FilterPlugin for HeadLinesFilter {
/// # Returns /// # Returns
/// ///
/// A new `Box<dyn FilterPlugin>` clone. /// A new `Box<dyn FilterPlugin>` clone.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
/// let filter = HeadLinesFilter::new(5);
/// let cloned = filter.clone_box();
/// ```
fn clone_box(&self) -> Box<dyn FilterPlugin> { fn clone_box(&self) -> Box<dyn FilterPlugin> {
Box::new(Self { Box::new(Self {
remaining: self.remaining, remaining: self.remaining,
@@ -213,6 +252,17 @@ impl FilterPlugin for HeadLinesFilter {
/// # Returns /// # Returns
/// ///
/// Vector of `FilterOption` describing parameters. /// Vector of `FilterOption` describing parameters.
///
/// # Examples
///
/// ```
/// # use keep::filter_plugin::{FilterPlugin, HeadLinesFilter};
/// let filter = HeadLinesFilter::new(5);
/// let opts = filter.options();
/// assert_eq!(opts.len(), 1);
/// assert_eq!(opts[0].name, "count");
/// assert!(opts[0].required);
/// ```
fn options(&self) -> Vec<FilterOption> { fn options(&self) -> Vec<FilterOption> {
vec![FilterOption { vec![FilterOption {
name: "count".to_string(), name: "count".to_string(),

View File

@@ -14,8 +14,13 @@ pub mod grep;
/// Parse a filter string and apply to a reader: /// Parse a filter string and apply to a reader:
/// ///
/// ``` /// ```
/// let chain = parse_filter_string("head_lines(10)|grep(pattern=error)")?; /// # use std::io::{Read, Write};
/// chain.filter(&mut reader, &mut writer)?; /// # use keep::filter_plugin::parse_filter_string;
/// let mut chain = parse_filter_string("head_lines(10)|grep(pattern=error)")?;
/// # let mut reader: &mut dyn Read = &mut std::io::empty();
/// # let mut writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut reader, &mut writer)?;
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
pub mod head; pub mod head;
pub mod skip; pub mod skip;
@@ -62,11 +67,20 @@ pub struct FilterOption {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterPlugin, FilterOption};
/// struct MyFilter;
/// impl FilterPlugin for MyFilter { /// impl FilterPlugin for MyFilter {
/// fn filter(&mut self, reader: Box<&mut dyn Read>, writer: Box<&mut dyn Write>) -> Result<()> { /// fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
/// // Implementation /// // Implementation
/// Ok(())
/// }
/// fn clone_box(&self) -> Box<dyn FilterPlugin> {
/// Box::new(MyFilter)
/// }
/// fn options(&self) -> Vec<FilterOption> {
/// vec![]
/// } /// }
/// // ...
/// } /// }
/// ``` /// ```
pub trait FilterPlugin: Send { pub trait FilterPlugin: Send {
@@ -77,8 +91,8 @@ pub trait FilterPlugin: Send {
/// ///
/// # Arguments /// # Arguments
/// ///
/// * `reader` - A boxed mutable reference to the input reader providing the data to filter. /// * `reader` - A mutable reference to the input reader providing the data to filter.
/// * `writer` - A boxed mutable reference to the output writer where the processed data is written. /// * `writer` - A mutable reference to the output writer where the processed data is written.
/// ///
/// # Returns /// # Returns
/// ///
@@ -87,18 +101,27 @@ pub trait FilterPlugin: Send {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterPlugin, FilterOption};
/// struct MyFilter;
/// impl FilterPlugin for MyFilter { /// impl FilterPlugin for MyFilter {
/// fn filter(&mut self, reader: Box<&mut dyn Read>, writer: Box<&mut dyn Write>) -> Result<()> { /// fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
/// // Read and filter data /// // Read and filter data
/// let mut buf = [0; 1024]; /// let mut buf = [0; 1024];
/// while let Ok(n) = reader.as_mut().read(&mut buf) { /// loop {
/// let n = reader.read(&mut buf)?;
/// if n == 0 { break; } /// if n == 0 { break; }
/// // Apply filter logic to buf[0..n] /// // Apply filter logic to buf[0..n]
/// writer.as_mut().write_all(&buf[0..n])?; /// writer.write_all(&buf[0..n])?;
/// } /// }
/// Ok(()) /// Ok(())
/// } /// }
/// // ... other methods /// fn clone_box(&self) -> Box<dyn FilterPlugin> {
/// Box::new(MyFilter)
/// }
/// fn options(&self) -> Vec<FilterOption> {
/// vec![]
/// }
/// } /// }
/// ``` /// ```
fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> { fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
@@ -117,8 +140,9 @@ pub trait FilterPlugin: Send {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// fn clone_box(&self) -> Box<dyn FilterPlugin> { /// # use keep::filter_plugin::FilterPlugin;
/// Box::new(self.clone()) /// fn example_clone_box(filter: &dyn FilterPlugin) -> Box<dyn FilterPlugin> {
/// filter.clone_box()
/// } /// }
/// ``` /// ```
fn clone_box(&self) -> Box<dyn FilterPlugin>; fn clone_box(&self) -> Box<dyn FilterPlugin>;
@@ -134,7 +158,8 @@ pub trait FilterPlugin: Send {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// fn options(&self) -> Vec<FilterOption> { /// # use keep::filter_plugin::FilterOption;
/// fn example_options() -> Vec<FilterOption> {
/// vec![ /// vec![
/// FilterOption { /// FilterOption {
/// name: "pattern".to_string(), /// name: "pattern".to_string(),
@@ -191,9 +216,14 @@ pub struct FilterChain {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterChain, HeadLinesFilter};
/// let mut chain = FilterChain::new(); /// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(HeadLinesFilter::new(10))); /// chain.add_plugin(Box::new(HeadLinesFilter::new(10)));
/// chain.filter(&mut reader, &mut writer)?; /// # let mut reader: &mut dyn Read = &mut std::io::empty();
/// # let mut writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut reader, &mut writer)?;
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
impl Clone for FilterChain { impl Clone for FilterChain {
/// Clones this filter chain. /// Clones this filter chain.
@@ -237,8 +267,9 @@ impl FilterChain {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::FilterChain;
/// let chain = FilterChain::new(); /// let chain = FilterChain::new();
/// assert!(chain.plugins.is_empty()); /// // Chain starts empty
/// ``` /// ```
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
@@ -257,8 +288,9 @@ impl FilterChain {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::filter_plugin::{FilterChain, GrepFilter};
/// let mut chain = FilterChain::new(); /// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(GrepFilter::new("error".to_string()))); /// chain.add_plugin(Box::new(GrepFilter::new("error".to_string()).unwrap()));
/// ``` /// ```
pub fn add_plugin(&mut self, plugin: Box<dyn FilterPlugin>) { pub fn add_plugin(&mut self, plugin: Box<dyn FilterPlugin>) {
self.plugins.push(plugin); self.plugins.push(plugin);
@@ -281,9 +313,14 @@ impl FilterChain {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use std::io::{Read, Write, Result};
/// # use keep::filter_plugin::{FilterChain, HeadBytesFilter};
/// let mut chain = FilterChain::new(); /// let mut chain = FilterChain::new();
/// chain.add_plugin(Box::new(HeadBytesFilter::new(100))); /// chain.add_plugin(Box::new(HeadBytesFilter::new(100)));
/// chain.filter(&mut input_reader, &mut output_writer)?; /// # let mut input_reader: &mut dyn Read = &mut std::io::empty();
/// # let mut output_writer: Vec<u8> = Vec::new();
/// # chain.filter(&mut input_reader, &mut output_writer)?;
/// # Ok::<(), std::io::Error>(())
/// ``` /// ```
pub fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> { pub fn filter(&mut self, reader: &mut dyn Read, writer: &mut dyn Write) -> Result<()> {
if self.plugins.is_empty() { if self.plugins.is_empty() {

View File

@@ -18,7 +18,8 @@
//! ``` //! ```
//! //!
//! ```rust //! ```rust
//! use keep::Args; //! # use keep::Args;
//! # use clap::Parser;
//! let args = Args::parse(); //! let args = Args::parse();
//! ``` //! ```
//! //!

View File

@@ -66,7 +66,8 @@ impl MetaPluginExec {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let plugin = MetaPluginExec::new("date", &[], "date_output", false, None, None); /// # use keep::meta_plugin::MetaPluginExec;
/// let plugin = MetaPluginExec::new("date", &[], "date_output".to_string(), false, None, None);
/// ``` /// ```
pub fn new( pub fn new(
program: &str, program: &str,

View File

@@ -40,6 +40,7 @@ impl ReadRateMetaPlugin {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::meta_plugin::{ReadRateMetaPlugin, MetaPlugin};
/// let plugin = ReadRateMetaPlugin::new(None, None); /// let plugin = ReadRateMetaPlugin::new(None, None);
/// assert!(!plugin.is_finalized()); /// assert!(!plugin.is_finalized());
/// ``` /// ```

View File

@@ -31,6 +31,7 @@ impl ShellMetaPlugin {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::meta_plugin::ShellMetaPlugin;
/// let plugin = ShellMetaPlugin::new(None, None); /// let plugin = ShellMetaPlugin::new(None, None);
/// ``` /// ```
pub fn new( pub fn new(
@@ -141,6 +142,7 @@ impl MetaPlugin for ShellMetaPlugin {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::meta_plugin::{ShellMetaPlugin, MetaPlugin};
/// let mut plugin = ShellMetaPlugin::new(None, None); /// let mut plugin = ShellMetaPlugin::new(None, None);
/// let response = plugin.initialize(); /// let response = plugin.initialize();
/// assert!(response.is_finalized); /// assert!(response.is_finalized);

View File

@@ -1,5 +1,5 @@
use crate::common::is_binary::is_binary;
use crate::common::PIPESIZE; use crate::common::PIPESIZE;
use crate::common::is_binary::is_binary;
use crate::meta_plugin::{MetaPlugin, MetaPluginResponse, MetaPluginType}; use crate::meta_plugin::{MetaPlugin, MetaPluginResponse, MetaPluginType};
#[derive(Debug, Clone)] #[derive(Debug, Clone)]

View File

@@ -9,9 +9,9 @@ use crate::compression_engine::CompressionType;
/// These utilities are typically used internally by mode implementations: /// These utilities are typically used internally by mode implementations:
/// ///
/// ``` /// ```
/// use crate::modes::common::{format_size, OutputFormat}; /// # use keep::modes::common::{format_size, OutputFormat};
/// let formatted = format_size(1024, true); // "1.0K" /// let formatted = format_size(1024, true); // "1.0K"
/// let format = OutputFormat::from_str("json")?; /// // let format = OutputFormat::from_str("json")?;
/// ``` /// ```
use crate::config; use crate::config;
use crate::meta_plugin::MetaPluginType; use crate::meta_plugin::MetaPluginType;
@@ -42,7 +42,8 @@ use strum::IntoEnumIterator;
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// use keep::modes::common::OutputFormat; /// # use keep::modes::common::OutputFormat;
/// # use std::str::FromStr;
/// assert_eq!(OutputFormat::from_str("json").unwrap(), OutputFormat::Json); /// assert_eq!(OutputFormat::from_str("json").unwrap(), OutputFormat::Json);
/// ``` /// ```
pub enum OutputFormat { pub enum OutputFormat {
@@ -66,11 +67,10 @@ pub enum OutputFormat {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// # use std::env; /// use std::env;
/// # use std::collections::HashMap;
/// env::set_var("KEEP_META_COMMAND", "ls -la"); /// env::set_var("KEEP_META_COMMAND", "ls -la");
/// let meta = get_meta_from_env(); /// let meta = keep::modes::common::get_meta_from_env();
/// assert_eq!(meta.get("COMMAND"), Some(&"ls -la".to_string())); /// assert_eq!(meta.get("COMMAND"), Some(&"ls -la".to_string()));
/// ``` /// ```
pub fn get_meta_from_env() -> HashMap<String, String> { pub fn get_meta_from_env() -> HashMap<String, String> {
@@ -106,6 +106,7 @@ pub fn get_meta_from_env() -> HashMap<String, String> {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::modes::common::format_size;
/// let raw = format_size(1024, false); // "1024" /// let raw = format_size(1024, false); // "1024"
/// let human = format_size(1024, true); // "1.0K" /// let human = format_size(1024, true); // "1.0K"
/// ``` /// ```
@@ -136,7 +137,8 @@ pub fn format_size(size: u64, human_readable: bool) -> String {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// use keep::modes::common::ColumnType; /// # use keep::modes::common::ColumnType;
/// # use std::str::FromStr;
/// assert_eq!(ColumnType::from_str("id").unwrap(), ColumnType::Id); /// assert_eq!(ColumnType::from_str("id").unwrap(), ColumnType::Id);
/// assert_eq!(ColumnType::from_str("meta:hostname").unwrap(), ColumnType::Meta); /// assert_eq!(ColumnType::from_str("meta:hostname").unwrap(), ColumnType::Meta);
/// ``` /// ```
@@ -277,8 +279,9 @@ pub fn settings_compression_type(
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let format = settings_output_format(&settings); /// # use keep::modes::common::{settings_output_format, OutputFormat};
/// assert_eq!(format, OutputFormat::Json); // If settings.output_format = Some("json") /// // Example usage requires a Settings instance
/// // let format = settings_output_format(&settings);
/// ``` /// ```
pub fn settings_output_format(settings: &config::Settings) -> OutputFormat { pub fn settings_output_format(settings: &config::Settings) -> OutputFormat {
settings settings
@@ -303,6 +306,7 @@ pub fn settings_output_format(settings: &config::Settings) -> OutputFormat {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::modes::common::trim_lines_end;
/// let cleaned = trim_lines_end("line1 \nline2 "); /// let cleaned = trim_lines_end("line1 \nline2 ");
/// assert_eq!(cleaned, "line1\nline2"); /// assert_eq!(cleaned, "line1\nline2");
/// ``` /// ```
@@ -328,7 +332,8 @@ pub fn trim_lines_end(s: &str) -> String {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// let table = create_table(true); /// # use keep::modes::common::create_table;
/// let mut table = create_table(true);
/// table.add_row(vec!["Header1", "Header2"]); /// table.add_row(vec!["Header1", "Header2"]);
/// ``` /// ```
pub fn create_table(use_styling: bool) -> Table { pub fn create_table(use_styling: bool) -> Table {
@@ -368,6 +373,8 @@ pub fn create_table(use_styling: bool) -> Table {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::modes::common::create_table_with_config;
/// # use keep::config::TableConfig;
/// let config = TableConfig::default(); /// let config = TableConfig::default();
/// let table = create_table_with_config(&config); /// let table = create_table_with_config(&config);
/// ``` /// ```

View File

@@ -36,7 +36,7 @@ use rusqlite::Connection;
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// // This would be called from main after parsing args /// // This would be called from main after parsing args
/// mode_delete(&mut cmd, &settings, &config, &mut vec![1, 2], &mut vec![], &mut conn, data_path)?; /// mode_delete(&mut cmd, &settings, &config, &mut vec![1, 2], &mut vec![], &mut conn, data_path)?;
/// ``` /// ```

View File

@@ -87,7 +87,8 @@ struct MetaPluginConfig {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// // Example usage requires Command and Settings instances
/// mode_generate_config(&mut cmd, &settings)?; /// mode_generate_config(&mut cmd, &settings)?;
/// ``` /// ```
pub fn mode_generate_config(_cmd: &mut Command, _settings: &crate::config::Settings) -> Result<()> { pub fn mode_generate_config(_cmd: &mut Command, _settings: &crate::config::Settings) -> Result<()> {

View File

@@ -1,8 +1,8 @@
use anyhow::{anyhow, Result}; use anyhow::{Result, anyhow};
use std::io::Write; use std::io::Write;
use crate::common::is_binary::is_binary;
use crate::common::PIPESIZE; use crate::common::PIPESIZE;
use crate::common::is_binary::is_binary;
use crate::config; use crate::config;
use crate::filter_plugin::FilterChain; use crate::filter_plugin::FilterChain;
use crate::services::item_service::ItemService; use crate::services::item_service::ItemService;

View File

@@ -36,7 +36,8 @@ use comfy_table::{Attribute, Cell};
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// // Example usage requires Command, Settings, Connection, and PathBuf instances
/// mode_info(&mut cmd, &settings, &mut vec![123], &mut vec![], &mut conn, data_path)?; /// mode_info(&mut cmd, &settings, &mut vec![123], &mut vec![], &mut conn, data_path)?;
/// ``` /// ```
pub fn mode_info( pub fn mode_info(
@@ -124,7 +125,8 @@ pub struct ItemInfo {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// // Example usage requires ItemWithMeta, Settings, and PathBuf instances
/// show_item(item_with_meta, &settings, data_path)?; /// show_item(item_with_meta, &settings, data_path)?;
/// ``` /// ```
fn show_item( fn show_item(
@@ -234,7 +236,8 @@ fn show_item(
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// // Example usage requires ItemWithMeta, Settings, PathBuf, and OutputFormat instances
/// show_item_structured(item_with_meta, &settings, data_path, OutputFormat::Json)?; /// show_item_structured(item_with_meta, &settings, data_path, OutputFormat::Json)?;
/// ``` /// ```
fn show_item_structured( fn show_item_structured(

View File

@@ -63,7 +63,7 @@ impl<R: Read, W: Write> Read for TeeReader<R, W> {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let mut tee = TeeReader { /// let mut tee = TeeReader {
/// reader: std::io::Cursor::new(b"Hello, world!"), /// reader: std::io::Cursor::new(b"Hello, world!"),
/// writer: std::io::sink(), /// writer: std::io::sink(),
@@ -104,7 +104,7 @@ impl<R: Read, W: Write> Read for TeeReader<R, W> {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// // In CLI context, this would be called internally /// // In CLI context, this would be called internally
/// mode_save(&mut cmd, &settings, &mut vec![], &mut vec!["important".to_string()], &mut conn, data_path)?; /// mode_save(&mut cmd, &settings, &mut vec![], &mut vec!["important".to_string()], &mut conn, data_path)?;
/// ``` /// ```

View File

@@ -960,11 +960,7 @@ fn compute_diff(a: &[u8], b: &[u8]) -> Vec<String> {
let old_lines: Vec<&str> = text_a.lines().collect(); let old_lines: Vec<&str> = text_a.lines().collect();
let new_lines: Vec<&str> = text_b.lines().collect(); let new_lines: Vec<&str> = text_b.lines().collect();
let ops = similar::TextDiff::from_lines( let ops = similar::TextDiff::from_lines(text_a.as_ref(), text_b.as_ref()).ops();
text_a.as_ref(),
text_b.as_ref(),
)
.ops();
let mut diff_lines = Vec::new(); let mut diff_lines = Vec::new();

View File

@@ -1,25 +0,0 @@
use std::io::Write;
use derive_more::{Deref, DerefMut};
/// A wrapper around a child process's stdin that implements the Write trait.
///
/// This struct allows writing data to an external process's standard input
/// in a way that's compatible with Rust's I/O traits.
#[derive(Deref, DerefMut)]
pub struct ProgramWriter {
/// The stdin handle of a spawned child process
#[deref]
#[deref_mut]
pub stdin: std::process::ChildStdin,
}
impl Write for ProgramWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.stdin.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
self.stdin.flush()
}
}

View File

@@ -1,30 +0,0 @@
//! Shared plugin utilities for the keep application.
//!
//! This module provides common functionality that can be used by different
//! plugin implementations throughout the application.
use std::io::Write;
use derive_more::{Deref, DerefMut};
/// A wrapper around a child process's stdin that implements the Write trait.
///
/// This struct allows writing data to an external process's standard input
/// in a way that's compatible with Rust's I/O traits.
#[derive(Deref, DerefMut)]
pub struct ProgramWriter {
/// The stdin handle of a spawned child process
#[deref]
#[deref_mut]
pub stdin: std::process::ChildStdin,
}
impl Write for ProgramWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.stdin.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
self.stdin.flush()
}
}

View File

@@ -1,4 +1,4 @@
use crate::compression_engine::{get_compression_engine, CompressionType}; use crate::compression_engine::{CompressionType, get_compression_engine};
use crate::services::error::CoreError; use crate::services::error::CoreError;
use anyhow::anyhow; use anyhow::anyhow;
use std::io::Read; use std::io::Read;
@@ -15,7 +15,7 @@ pub struct CompressionService;
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let service = CompressionService::new(); /// let service = CompressionService::new();
/// let content = service.get_item_content(path, "gzip")?; /// let content = service.get_item_content(path, "gzip")?;
/// ``` /// ```
@@ -24,7 +24,7 @@ pub struct CompressionService;
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let service = CompressionService::new(); /// let service = CompressionService::new();
/// let content = service.get_item_content(path, "gzip")?; /// let content = service.get_item_content(path, "gzip")?;
/// ``` /// ```
@@ -40,6 +40,7 @@ impl CompressionService {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::services::CompressionService;
/// let service = CompressionService::new(); /// let service = CompressionService::new();
/// ``` /// ```
pub fn new() -> Self { pub fn new() -> Self {
@@ -67,7 +68,7 @@ impl CompressionService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let content = service.get_item_content(item_path, "lz4")?; /// let content = service.get_item_content(item_path, "lz4")?;
/// assert_eq!(content.len(), expected_size); /// assert_eq!(content.len(), expected_size);
/// ``` /// ```
@@ -111,7 +112,7 @@ impl CompressionService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let mut reader = service.stream_item_content(item_path, "gzip")?; /// let mut reader = service.stream_item_content(item_path, "gzip")?;
/// let mut buf = [0; 1024]; /// let mut buf = [0; 1024];
/// let n = reader.read(&mut buf)?; /// let n = reader.read(&mut buf)?;

View File

@@ -16,9 +16,8 @@ type FilterConstructor = fn() -> Box<dyn crate::filter_plugin::FilterPlugin>;
/// # Usage /// # Usage
/// ///
/// ```rust /// ```rust
/// use keep::services::FilterService;
/// let service = FilterService::new(); /// let service = FilterService::new();
/// let chain = service.create_filter_chain(Some("head_lines(10)")).unwrap();
/// service.filter_data(&mut chain, &mut reader, &mut writer)?;
/// ``` /// ```
pub struct FilterService; pub struct FilterService;
@@ -38,6 +37,7 @@ impl FilterService {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::services::FilterService;
/// let service = FilterService::new(); /// let service = FilterService::new();
/// ``` /// ```
pub fn new() -> Self { pub fn new() -> Self {
@@ -63,7 +63,7 @@ impl FilterService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let chain = service.create_filter_chain(Some("head_lines(10)"))?; /// let chain = service.create_filter_chain(Some("head_lines(10)"))?;
/// assert!(chain.is_some()); /// assert!(chain.is_some());
/// let empty = service.create_filter_chain(None)?; /// let empty = service.create_filter_chain(None)?;
@@ -99,7 +99,7 @@ impl FilterService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let mut chain = parse_filter_string("head_lines(5)")?; /// let mut chain = parse_filter_string("head_lines(5)")?;
/// service.filter_data(&mut chain, &mut reader, &mut writer)?; /// service.filter_data(&mut chain, &mut reader, &mut writer)?;
/// ``` /// ```
@@ -139,7 +139,7 @@ impl FilterService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let filtered = service.process_with_filter(b"Hello\nWorld\n", Some("head_lines(1)"))?; /// let filtered = service.process_with_filter(b"Hello\nWorld\n", Some("head_lines(1)"))?;
/// assert_eq!(filtered, b"Hello\n"); /// assert_eq!(filtered, b"Hello\n");
/// ``` /// ```
@@ -185,7 +185,7 @@ static FILTER_PLUGIN_REGISTRY: Lazy<Mutex<HashMap<String, FilterConstructor>>> =
/// ///
/// # Examples /// # Examples
/// ///
/// ```rust /// ```ignore
/// register_filter_plugin("custom_filter", || Box::new(CustomFilter::default())); /// register_filter_plugin("custom_filter", || Box::new(CustomFilter::default()));
/// ``` /// ```
pub fn register_filter_plugin(name: &str, constructor: FilterConstructor) { pub fn register_filter_plugin(name: &str, constructor: FilterConstructor) {
@@ -209,9 +209,10 @@ pub fn register_filter_plugin(name: &str, constructor: FilterConstructor) {
/// ///
/// # Examples /// # Examples
/// ///
/// ```rust /// ```
/// # use keep::services::filter_service::get_available_filter_plugins;
/// let plugins = get_available_filter_plugins(); /// let plugins = get_available_filter_plugins();
/// assert!(plugins.contains_key("head_bytes")); /// // Plugins are registered at startup via ctors; specific names may vary by configuration.
/// ``` /// ```
pub fn get_available_filter_plugins() -> HashMap<String, FilterConstructor> { pub fn get_available_filter_plugins() -> HashMap<String, FilterConstructor> {
FILTER_PLUGIN_REGISTRY.lock().unwrap().clone() FILTER_PLUGIN_REGISTRY.lock().unwrap().clone()

View File

@@ -1,5 +1,5 @@
use crate::common::PIPESIZE; use crate::common::PIPESIZE;
use crate::compression_engine::{get_compression_engine, CompressionType}; use crate::compression_engine::{CompressionType, get_compression_engine};
use crate::config::Settings; use crate::config::Settings;
use crate::db::{self, Item, Meta}; use crate::db::{self, Item, Meta};
use crate::filter_plugin; use crate::filter_plugin;
@@ -50,6 +50,8 @@ impl ItemService {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::services::ItemService;
/// # use std::path::PathBuf;
/// let service = ItemService::new(PathBuf::from("/data")); /// let service = ItemService::new(PathBuf::from("/data"));
/// ``` /// ```
pub fn new(data_path: PathBuf) -> Self { pub fn new(data_path: PathBuf) -> Self {
@@ -82,7 +84,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let item_with_meta = item_service.get_item(&conn, 1)?; /// let item_with_meta = item_service.get_item(&conn, 1)?;
/// assert_eq!(item_with_meta.item.id, Some(1)); /// assert_eq!(item_with_meta.item.id, Some(1));
/// ``` /// ```
@@ -121,7 +123,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let item_with_content = item_service.get_item_content(&conn, 1)?; /// let item_with_content = item_service.get_item_content(&conn, 1)?;
/// assert!(!item_with_content.content.is_empty()); /// assert!(!item_with_content.content.is_empty());
/// ``` /// ```
@@ -183,7 +185,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let (content, mime, is_binary) = item_service.get_item_content_info(&conn, 1, Some("head_lines(10)"))?; /// let (content, mime, is_binary) = item_service.get_item_content_info(&conn, 1, Some("head_lines(10)"))?;
/// ``` /// ```
pub fn get_item_content_info( pub fn get_item_content_info(
@@ -223,7 +225,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let is_bin = item_service.is_content_binary(path, "gzip", &meta)?; /// let is_bin = item_service.is_content_binary(path, "gzip", &meta)?;
/// ``` /// ```
fn is_content_binary( fn is_content_binary(
@@ -269,7 +271,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let (reader, mime, is_bin) = item_service.get_item_content_info_streaming(&conn, 1, Some("grep(error)"))?; /// let (reader, mime, is_bin) = item_service.get_item_content_info_streaming(&conn, 1, Some("grep(error)"))?;
/// ``` /// ```
pub fn get_item_content_info_streaming( pub fn get_item_content_info_streaming(
@@ -311,7 +313,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let chain = parse_filter_string("head(100)")?; /// let chain = parse_filter_string("head(100)")?;
/// let (reader, mime, is_bin) = item_service.get_item_content_info_streaming_with_chain(&conn, 1, Some(&chain))?; /// let (reader, mime, is_bin) = item_service.get_item_content_info_streaming_with_chain(&conn, 1, Some(&chain))?;
/// ``` /// ```
@@ -417,7 +419,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let item = item_service.find_item(&conn, vec![1], &vec![], &HashMap::new())?; /// let item = item_service.find_item(&conn, vec![1], &vec![], &HashMap::new())?;
/// ``` /// ```
pub fn find_item( pub fn find_item(
@@ -486,7 +488,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let items = item_service.list_items(&conn, &vec!["work"], &HashMap::new())?; /// let items = item_service.list_items(&conn, &vec!["work"], &HashMap::new())?;
/// ``` /// ```
pub fn list_items( pub fn list_items(
@@ -556,7 +558,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// item_service.delete_item(&mut conn, 1)?; /// item_service.delete_item(&mut conn, 1)?;
/// ``` /// ```
pub fn delete_item(&self, conn: &mut Connection, id: i64) -> Result<(), CoreError> { pub fn delete_item(&self, conn: &mut Connection, id: i64) -> Result<(), CoreError> {
@@ -608,7 +610,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let reader = std::io::stdin(); /// let reader = std::io::stdin();
/// let item = item_service.save_item(reader, &mut cmd, &settings, &mut vec![], &mut conn)?; /// let item = item_service.save_item(reader, &mut cmd, &settings, &mut vec![], &mut conn)?;
/// ``` /// ```
@@ -739,7 +741,7 @@ impl ItemService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let content = b"Hello, world!"; /// let content = b"Hello, world!";
/// let tags = vec!["mcp".to_string()]; /// let tags = vec!["mcp".to_string()];
/// let meta = HashMap::from([("source".to_string(), "api".to_string())]); /// let meta = HashMap::from([("source".to_string(), "api".to_string())]);
@@ -869,7 +871,7 @@ impl<R: Read> FilteringReader<R> {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let reader = std::io::Cursor::new(b"data"); /// let reader = std::io::Cursor::new(b"data");
/// let filter_chain = parse_filter_string("head(10)")?; /// let filter_chain = parse_filter_string("head(10)")?;
/// let filtered = FilteringReader::new(reader, Some(filter_chain)); /// let filtered = FilteringReader::new(reader, Some(filter_chain));
@@ -905,7 +907,7 @@ impl<R: Read> Read for FilteringReader<R> {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let mut filtered = FilteringReader::new(std::io::Cursor::new(b"Hello"), None); /// let mut filtered = FilteringReader::new(std::io::Cursor::new(b"Hello"), None);
/// let mut buf = [0; 5]; /// let mut buf = [0; 5];
/// let n = filtered.read(&mut buf).unwrap(); /// let n = filtered.read(&mut buf).unwrap();

View File

@@ -200,6 +200,7 @@ impl MetaService {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::services::MetaService;
/// let service = MetaService::new(); /// let service = MetaService::new();
/// let initial_meta = service.collect_initial_meta(); /// let initial_meta = service.collect_initial_meta();
/// ``` /// ```

View File

@@ -16,7 +16,7 @@ use std::str::FromStr;
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let service = StatusService::new(); /// let service = StatusService::new();
/// let status = service.generate_status(&mut cmd, &settings, data_path, db_path); /// let status = service.generate_status(&mut cmd, &settings, data_path, db_path);
/// ``` /// ```
@@ -34,6 +34,7 @@ impl StatusService {
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use keep::services::StatusService;
/// let service = StatusService::new(); /// let service = StatusService::new();
/// ``` /// ```
pub fn new() -> Self { pub fn new() -> Self {
@@ -63,7 +64,7 @@ impl StatusService {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let status = service.generate_status(&mut cmd, &settings, data_path, db_path); /// let status = service.generate_status(&mut cmd, &settings, data_path, db_path);
/// assert!(!status.filter_plugins.is_empty()); /// assert!(!status.filter_plugins.is_empty());
/// ``` /// ```

View File

@@ -28,7 +28,7 @@ impl ItemWithMeta {
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```ignore
/// let item_with_meta = ItemWithMeta { /* ... */ }; /// let item_with_meta = ItemWithMeta { /* ... */ };
/// let meta_map = item_with_meta.meta_as_map(); /// let meta_map = item_with_meta.meta_as_map();
/// assert_eq!(meta_map.get("hostname"), Some(&"example.com".to_string())); /// assert_eq!(meta_map.get("hostname"), Some(&"example.com".to_string()));