Velum Docs

⚡ Quick Start

Prerequisites

• Go 1.23+
• PostgreSQL
• LLM API key (optional — enables AI features) — Groq (default), OpenAI, or any OpenAI-compatible provider

Install & Run

$ git clone https://github.com/gokulnair2001/Velum.git
$ cd velum
$ go mod tidy
$ cp example.config.yaml config.yaml   # edit with your Postgres credentials
$ go run cmd/velum/main.go
✓ velum listening on :8080

Docker

$ docker network create velum-network
$ docker compose up --build

🧪 Try It

Send a few test events to see Velum detect patterns in real time:

$ curl -X POST http://localhost:8080/api/v1/analyze \
  -H "Content-Type: application/json" \
  -H "X-Project-ID: my-app" \
  -d '{
    "events": [
      {
        "event": "checkout_page_view",
        "ts": 1707500000000,
        "user_id": "usr-101",
        "session_id": "sess-abc",
        "device": "mobile"
      },
      {
        "event": "checkout_payment_click",
        "ts": 1707500015000,
        "user_id": "usr-101",
        "session_id": "sess-abc",
        "error_code": "card_declined"
      },
      {
        "event": "checkout_payment_click",
        "ts": 1707500045000,
        "user_id": "usr-101",
        "session_id": "sess-abc",
        "error_code": "card_declined"
      }
    ]
  }'

Optional: Build a Baseline

Want trend comparisons ("retry storms increased 21% vs. last 28 days")? Feed historical events to /api/v1/baseline first. Without this step, /analyze still detects all patterns — you just won't get trend data.

$ curl -X POST http://localhost:8080/api/v1/baseline \
  -H "Content-Type: application/json" \
  -H "X-Project-ID: my-app" \
  -d @historical_events.json

🔬 Processing Pipeline

Events flow through an 8-layer sequential pipeline. Each layer implements a Layer interface and processes the output of the previous layer. One API call triggers the full pipeline.

🧬 Layer Deep Dives

// Product A
{"event_name": "checkout", "uid": "u1", "time": 170850000}

// Product B
{"action": "purchase", "user": "u1", "timestamp": "2024-02-21T10:00:00Z"}

// Velum detects both automatically — zero config needed

CanonicalEvent{
  Flow:       "payment",       // from Surface token
  Action:     "payment",       // from Surface token
  Status:     "failed",        // from Status token
  UserID:     "u1",
  SessionID:  "s1",
  Timestamp:  1708500015000,
  RawProperties: { ... }       // original JSON preserved
}

{
  "pattern": "retry_storm",
  "flow": "booking",
  "affected_users": 3,
  "total_flows": 4,
  "impact_ratio": 0.75,
  "description": "High frequency of retry attempts in booking flow"
}

// Current observation
retry_storm on booking: impact_ratio = 0.75

// Historical baseline (last 7 observations)
average = 0.35, σ = 0.08

// Deviation calculation
(0.75 - 0.35) / 0.08 = 5.0 standard deviations
threshold = 2.0

→ SIGNIFICANT INCREASE

{
  "summary": "Retry storm affecting 42% of playback users...",
  "details": ["retry_storm in playback: 2/12 users. Errors: buffer_timeout (2), drm_license_failed (3)"],
  "hypotheses": ["DRM licensing may be broken for IN region — 3 of 5 failures are drm_license_failed on mobile"],
  "confidence_note": "These are hypotheses based on observed behavioral changes."
}

🎯 Detected Patterns

Velum detects seven behavioral anti-patterns across users. Detection is cross-instance — patterns are found across multiple flow instances per user for maximum accuracy.

Severity & Significance

Pattern severity is weighted by pattern type and flow intent:

Transactional flows get a 1.5× multiplier; browse flows get 0.7×.

🔌 API Reference

Endpoints

Headers

Request Body

{
  "events": [
    {
      "event": "checkout_payment_click",     // required — event name
      "ts": 1707500000000,                  // required — epoch milliseconds
      "user_id": "usr-123",                 // required — user identifier
      "session_id": "sess-abc",             // optional — session identifier
      "device": "mobile",                   // auto-classified as dimension
      "error_code": "card_declined"          // auto-classified as condition
    }
  ]
}

Response Shape

{
  "success": true,
  "message": "Behavioral analysis complete",
  "request_id": "...",
  "data": {
    "ai_analysis": {
      "summary": "Booking flow shows 75% retry storm rate...",
      "details": ["..."],
      "hypotheses": ["Driver supply insufficient for long-distance rides..."],
      "confidence_note": "Hypotheses based on observed behavioral changes."
    }
  }
}

When AI is disabled, data.patterns is returned instead of data.ai_analysis, containing the raw detected patterns array.

⚙️ Configuration

Config is loaded from config.yaml → config.yml → /etc/velum/config.yaml (first found wins), then overridden by VELUM_* environment variables.

Minimal Config

server:
  port: "8080"
  environment: "development"

storage:
  type: "postgres"
  postgres:
    host: "localhost"
    port: 5432
    database: "velum"
    user: "velum_user"
    password: "your_password"

security:
  enabled: false

Config Sections

Security

security:
  enabled: true
  api_key_hash: "<sha256-hash-of-your-key>"

Generate a hash:

$ printf "my-secret-key" | shasum -a 256

Then pass X-Infra-Key: my-secret-key on every request. Velum compares using constant-time SHA-256 comparison.

AI Features

All three AI layers support any OpenAI-compatible API — Groq (default), OpenAI, Together, Mistral, Fireworks, and more. The API URL is auto-resolved from the provider name.

Baseline Detection

baseline:
  window_days: 28               # Days of history for baseline computation
  min_days: 7                   # Minimum days before baseline is valid
  min_affected_users: 5         # Below this, significance is capped at "low"
  computation_mode: "daily"     # "daily" (cached) or "always" (per-request)
  trend_threshold: 0.10         # 10% delta to flag increasing/decreasing
  high_significance_threshold: 0.15  # 15% delta for high significance
  std_deviation_multiplier: 2.0      # Multiplier for std-based significance

min_affected_users prevents low-volume patterns (e.g., 1 user with 100% impact ratio) from being flagged as high significance. Set to 1 to disable the guard.

How Baseline Works

Every analysis request:

1. Detects patterns in the current batch (stateless, works for any time window)
2. Compares each pattern's impact ratio against the stored historical average (last window_days)
3. Stores the current snapshot via upsert — keyed on (date, pattern_type, flow, context_key)

Trend is classified by delta percentage: ≥10% increase → increasing, ≥10% decrease → decreasing, otherwise stable.

Significance uses standard deviation when available (delta ≥ 2×stddev → high), falls back to absolute threshold (delta ≥ 0.15 → high). Capped at low when affected users < min_affected_users.

Data Ingestion Guidelines

• Consistent windows: For meaningful baseline comparisons, send the same time window each ingestion (e.g., always a full day). Inconsistent window sizes produce different denominators, making ratio comparisons noisy.
• No overlap: Avoid sending overlapping event batches for the same day. The last batch overwrites the snapshot (upsert), so overlapping batches cause the stored ratio to reflect only the last batch.
• Re-processing: Sending the same complete batch again is safe — the upsert overwrites with identical values.
• Ad-hoc analysis: The /api/v1/analyze endpoint never writes to baseline history, so investigative queries with non-standard windows are always safe.

Retention & Cleanup

Snapshots are auto-deleted after retention_days (default: 90 days). A background goroutine runs cleanup on startup and every 24 hours.

storage:
  retention_days: 90    # Snapshots older than this are deleted

ai_analyzer:
  enabled: true
  provider: "groq"              # URL auto-resolved (default if omitted)
  api_key: "gsk_..."             # or env: VELUM_AI_API_KEY
  model: "llama-3.1-8b-instant"

ai_analyzer:
  enabled: true
  provider: "openai"             # URL auto-resolved
  api_key: "sk-..."              # or env: VELUM_AI_API_KEY
  model: "gpt-4o-mini"

ai_analyzer:
  enabled: true
  provider: "custom"
  base_url: "http://localhost:11434/v1/chat/completions"
  model: "llama3"

Data Mapping

If your events use a non-standard schema, map fields declaratively:

data_mapping:
  enabled: true
  mapping:
    event:
      paths: ["payload.event.action", "event_name"]
      required: true
    ts:
      paths: ["meta.time", "timestamp"]
      format: "epoch_ms"
      required: true
    user_id:
      paths: ["context.user.id", "user_id"]
      required: true
    session_id:
      paths: ["context.session.id", "session_id"]

Supports dot-notation paths with fallback order. Extra properties pass through automatically.

Environment Variables

📋 Full Example: Ride-Hailing App

Here's a complete walkthrough showing what each pipeline layer does with a real ride-hailing scenario.

Input Events

{
  "events": [
    {"event":"app_opened","ts":1708700000000,"user_id":"u1","session_id":"s1","device":"mobile","country":"IN"},
    {"event":"booking_requested","ts":1708700045000,"user_id":"u1","session_id":"s1","fare":790},
    {"event":"driver_assigned","ts":1708700060000,"user_id":"u1","session_id":"s1","driver_id":"d01"},
    {"event":"driver_cancelled","ts":1708700090000,"user_id":"u1","session_id":"s1","cancel_reason":"too_far"},
    {"event":"booking_requested","ts":1708700095000,"user_id":"u1","session_id":"s1","fare":790},
    {"event":"driver_assigned","ts":1708700110000,"user_id":"u1","session_id":"s1","driver_id":"d02"},
    {"event":"ride_started","ts":1708700180000,"user_id":"u1","session_id":"s1"},
    {"event":"ride_completed","ts":1708701000000,"user_id":"u1","session_id":"s1"}
  ]
}

Layer-by-Layer Trace

Layer 0 (Context Enricher):
  Schema detected: event="event", timestamp="ts", user_id="user_id"

Layer 2 (Event Adapter):
  "booking_requested" → {Flow:"booking", Status:"request"}
  "driver_assigned"   → {Flow:"driver",  Status:"assigned"}
  "driver_cancelled"  → {Flow:"driver",  Status:"cancelled"}
  "ride_completed"    → {Flow:"ride",    Status:"completed"}

Layer 3 (Session Flow):
  u1 → Flow 1: booking [booking_requested, driver_*, booking_requested, driver_*]
       ↑ driver events folded in, retry cycle merged
       Flow 2: ride [ride_started, ride_completed]

Layer 4 (Behavior):
  booking: [attempt, retry, progress]
  ride:    [attempt, succeed]

Layer 5 (Patterns — across all users):
  retry_storm on booking: 3/4 users (75%) → DETECTED
  masked_failure on booking: 2/4 users → DETECTED

Layer 6 (Baseline):
  retry_storm: FIRST OBSERVATION → establishing baseline
  masked_failure: FIRST OBSERVATION → establishing baseline

Layer 7 (AI):
  "75% retry storm in booking — driver_cancelled with cancel_reason=too_far
   in 2 of 3 cases. Driver supply may be insufficient for long-distance rides."

🏢 Multi-Tenancy

Each X-Project-ID gets isolated storage:

• Pattern baselines are stored in per-project tables (pattern_snapshots_{project_id})
• Vocabulary and property registry are shared across projects
• A background goroutine runs daily cleanup of snapshots older than retention_days

Database Tables

🛡️ Resiliency

Velum is built for production reliability with multiple layers of protection:

🔧 Tech Stack

External Dependencies

Build & Test

$ go test ./...           # All tests
$ go test ./... -v        # Verbose
$ go test ./... -cover    # With coverage
$ go build -o velum cmd/velum/main.go  # Build binary