Engineering Decisions

Strict constraints breed creative architectures. Here are the core trade-offs and offline-first design choices that shape the system.

01. Zero Dependencies & Offline-First

Context & Trade-off

Deploying in air-gapped or high-security Kubernetes environments means we cannot rely on external APIs (like AWS Macie or external ML APIs). Everything must run entirely on-cluster.

The Single Binary / Sidecar Model

Instead of deploying a centralized secret-scanning service that requires log forwarding (and potential leak in transit), we chose a decentralized sidecar model. The LogShield engine is compiled down to a highly optimized artifact that intercepts logs via shared volumes.

// TLS Certificates for the Webhook are self-signed internally 
// using a Go-based cert generator on startup.
// We DO NOT require cert-manager.

func GenerateSelfSignedCerts() error {
    log.Println("[BOOT] Generating ephemeral Webhook TLS...")
    // In-memory generation avoids relying on PKI infrastructure
}

02. Aho-Corasick over Raw Regex

Performance Critical

Running 600+ complex regular expressions on every single log line in a high-throughput microservice would cause severe CPU throttling and latency.

Algorithmic Fast-Path

We implemented a fast-path filter using the Aho-Corasick string matching algorithm. This automaton searches for a large dictionary of high-entropy prefixes (e.g., \`sk_live_\`, \`xoxb-\`, \`BEGIN RSA\`) in a single pass over the log line. Time complexity drops to O(N + M + Z) where N is text length.

Only if the automaton triggers a match does the line advance to the expensive Regex Engine and Context Scorer. This reduces CPU load by ~92% on typical application logs.

03. Deterministic Over Generative AI

Accuracy vs Hype

While LLMs are excellent at semantic understanding, they are non-deterministic, slow, and computationally expensive for per-line log analysis.

Statistical Machine Learning

We opted for a lightweight, deterministic ML model (utilizing Random Forests / Gradient Boosting trained on SecretBench). It runs inference in <1ms per line. The model calculates the Secret Confidence Score (SCS) based on hard features:

Shannon Entropy Density

Character Set Distribution (Z-Score outliers)

Contextual keywords (e.g., proximity to "Bearer")

def calculate_scs(log_line):
    features = [
        shannon_entropy(log_line),
        distance_to_keyword(log_line, "password"),
        regex_tier_weight(log_line)
    ]
    return rf_model.predict_proba(features)[0] * 100