feat: add LLM token counting meta plugin and token filters
Add tiktoken-based token counting via new 'tokens' feature flag. New components: - Shared tokenizer module wrapping tiktoken CoreBPE (cl100k_base, o200k_base) - TokensMetaPlugin: streaming token counter, tokenizes each chunk independently - head_tokens(N): stream first N tokens, split at exact boundary when mid-chunk - skip_tokens(N): skip first N tokens, stream the rest - tail_tokens(N): bounded ring buffer (~16KB), outputs last N tokens at finalize All filters are fully streaming — no full-stream buffering. Meta plugin accuracy: exact for normal text, ±1-2 tokens if long whitespace sequence spans a chunk boundary. Also: add 'client' and 'tokens' to default features, add curl to Dockerfile builder stage.
This commit is contained in:
@@ -3,6 +3,7 @@ FROM rust:1.88-slim AS builder
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
cmake \
|
||||
curl \
|
||||
make \
|
||||
gcc \
|
||||
musl-tools \
|
||||
@@ -16,7 +17,6 @@ WORKDIR /app
|
||||
# Copy manifests and fetch dependencies (cached layer)
|
||||
COPY Cargo.toml Cargo.lock ./
|
||||
RUN mkdir src && echo 'fn main() {}' > src/main.rs && echo '' > src/lib.rs
|
||||
|
||||
RUN cargo fetch --target x86_64-unknown-linux-musl
|
||||
|
||||
# Copy real source and build static binary
|
||||
|
||||
Reference in New Issue
Block a user