CivStart Intelligence System — Proposal & Technical Research

Date: April 24, 2026 Proposed by: Bijo Mathew Jose (Developer) Status: Research proposal — for team review Purpose: Research and technical design for a continuously running, multi-modular scraping and analysis system that could identify gov-tech gaps, emerging trends, and product opportunities for CivStart. Intended as a revenue-enabling intelligence layer.

Note: This document is a developer's research proposal exploring how CivStart could expand its strategic capabilities. It is not an approved specification. Ownership, priorities, and build decisions belong to CivStart's product leadership.

Vision
High-Level Architecture
Core Module Specifications
- 3.1 Government Scraper System
- 3.2 Vendor Scraper System
- 3.3 Data Consumption System
- 3.4 Categorization System
- 3.5 Connection Engine
Cross-Cutting Concerns
- 4.1 Observability
- 4.2 Human Review
- 4.3 Internal Signal Integration
- 4.4 Historical Layer
- 4.5 Outcome Feedback
- 4.6 Compliance & Ethics
- 4.7 Cost Control
- 4.8 Export & Integration
Data Model
Implementation Roadmap
Risk Register
Ownership & Operations

1. Vision

The Problem

CivStart's product strategy is currently informed by episodic research — conversations with pilot users, occasional competitor audits, ad-hoc market analysis. Insights go stale quickly. By the time a trend is noticed, competitors may already be moving.

The Solution

A continuously running intelligence system that:

Scrapes government and vendor sources daily
Categorizes and extracts keywords from raw data
Feeds learned keywords back to scrapers (self-improving loop)
Detects patterns, gaps, and emerging opportunities
Delivers actionable insights to CivStart's product team via daily digests, weekly syntheses, and monthly strategic briefs

Key Principles

Feedback-loop, not linear pipeline — categorization informs scraping
Modular & independently deployable — each module can be replaced without touching others
Evolution over static config — scrapers and categories grow with the system
Internal + external signals — CivStart's own data is part of the analysis
Cost-conscious — tiered processing, cheap first then expensive

Success Criteria

Month 3: System produces weekly briefs that the product team actually reads
Month 6: At least one product decision traced back to a system-surfaced insight
Month 12: Trend predictions validated by reality (e.g., "X category will grow" → it did)

2. High-Level Architecture

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│    [1. Gov Scraper]      [2. Vendor Scraper]                    │
│         │                      │                                │
│         ▼                      ▼                                │
│                                                                 │
│    [3. Data Consumption System]                                 │
│         │                                                       │
│         ▼                                                       │
│                                                                 │
│    [4. Categorization System] ──┐                               │
│         │                       │ keyword feedback              │
│         │                       └──► to scrapers                │
│         ▼                                                       │
│                                                                 │
│    [5. Connection Engine]                                       │
│         │                                                       │
│         ▼                                                       │
│    [Export: Dashboard, Slack, Email, Linear]                    │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   CROSS-CUTTING (apply to all modules):                         │
│   • Observability • Human Review • Internal Signals             │
│   • Historical Layer • Outcome Feedback • Compliance            │
│   • Cost Control • Export & Integration                         │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Technology Stack

Runtime: NestJS backend (matches existing CivStart stack)
Scraping: Playwright (already integrated via MCP)
Storage: PostgreSQL + Prisma (existing)
AI: OpenAI + Gemini (existing integrations) — mix for cost control
Vector search: Existing SignalEmbedding infrastructure
Scheduling: NestJS cron jobs
Queue: Bull/BullMQ on Redis (Redis already in Docker setup)
Frontend: Admin dashboard extension (existing Next.js admin app)

3. Core Module Specifications

3.1 Government Scraper System

Responsibility: Continuously pull data from government sources to detect emerging demand, procurement patterns, and policy shifts.

Sub-modules

3.1.1 Source Registry

Database table: scraper_sources
Fields: id, name, type (gov|vendor), url, scrape_frequency, last_scraped_at, last_success_at, status (active|paused|failed), keywords (JSON array), config (JSON)
Admin UI to add/pause/configure sources

3.1.2 Scraper Workers Each source type has its own worker implementing a common interface:

interface ScraperWorker {
  sourceId: string;
  run(): Promise<ScrapeResult>;
  healthCheck(): Promise<HealthStatus>;
}

Initial sources:

SAM.gov — federal RFP portal (has API, not just scraping)
State RFP portals — BidNet, state procurement sites
Council minutes — starts with top 50 US cities
Reddit — r/govtech, r/publicadministration, r/procurement (official API)
GovTech / StateScoop / Route Fifty — RSS feeds where available
LinkedIn — gov job postings (via public posting URLs, no auth-walled content)
GitHub — civic tech repositories, gov-adjacent repos

3.1.3 Source Discovery

Periodically (weekly) search for new government sources
LLM-powered: "Given this list of active gov sources, suggest 5 more that might yield similar signals"
Admin reviews suggestions before adding to registry

3.1.4 Adaptation Layer

Detects when a scrape returns unexpected structure (e.g., HTML layout changed)
Alerts admin instead of silently failing
Stores schema fingerprint per source, compares each run

3.1.5 Rate Limiter + Scheduler

Respects robots.txt per source
Configurable delays (default: 2 seconds between requests per domain)
Uses BullMQ for distributed scheduling
Backoff on failures (exponential, max 24 hours)

Inputs

Source registry entries
Keyword watchlist (from Categorization System feedback loop)

Outputs

Raw scrape results written to raw_scrape_results table
Queue message to Data Consumption System per successful scrape

Edge Cases

Site returns 429 (rate limited) → backoff + alert
Site structure changes → adaptation layer catches it
Authentication required → mark source as requiring credentials, admin must configure
Duplicate content across sources → handled in Data Consumption, not here

3.2 Vendor Scraper System

Responsibility: Track gov-tech vendors, competitors, and emerging startups.

Sub-modules

3.2.1 Vendor Registry

Database table: tracked_vendors
Fields: id, name, website, category (competitor|adjacent|startup|incumbent), track_depth (light|medium|deep), status (active|dead|acquired)
Seed with: Tyler, OpenGov, Granicus, CivicPlus, Accela, Bonfire, PublicInput, Polco, Zencity, etc.

3.2.2 Source Workers

Crunchbase — funding rounds, new companies (uses API)
Product Hunt — new gov-tech product launches
LinkedIn — company pages (public content), employee count changes
Company websites — blog posts, product page changes, pricing changes
GovTech magazine — news about vendors
YC / accelerator batch lists — new gov-tech startups

3.2.3 New Entrant Detection

Scans Crunchbase, Product Hunt, YC for companies tagged gov-tech, civic-tech, public-sector
LLM evaluates: "Is this gov-tech? Is it relevant to CivStart's space?"
High-relevance entrants auto-added to registry at track_depth: light

3.2.4 Change Tracker

Diffs vendor websites over time (weekly snapshots)
Flags meaningful changes: new products, pricing changes, new customer logos, leadership changes
Uses LLM for semantic diff, not just text diff

Inputs

Vendor registry
Keyword watchlist (from Categorization feedback)
Tracked companies mentioned in government signals (cross-feed)

Outputs

Raw scrape results to raw_scrape_results table
Vendor update events to vendor_changes table

3.3 Data Consumption System

Responsibility: Transform raw scraped data into clean, structured records. This is the bridge between scraping and analysis.

Sub-modules

3.3.1 Normalizer

Input types: HTML, PDF, JSON, plain text
Output: unified NormalizedSignal schema
Handles:
- HTML → extract main content (strip nav, footer)
- PDF → text extraction (use pdf-parse or similar)
- Tables → structured JSON
- Dates → ISO 8601
- Currency → normalized to USD

3.3.2 Deduplicator

Same RFP posted on SAM.gov + state portal = one record
Uses content hashing + fuzzy matching on titles/orgs
Merges metadata from multiple sources

3.3.3 Entity Extractor

LLM-powered extraction:
- Organizations (government entity, vendor)
- People (titles, not necessarily names — PII handling)
- Financial amounts (budget, contract value)
- Dates (deadline, fiscal year)
- Products/services mentioned
- Locations (state, city, county)
Stored as structured JSON on normalized signal

3.3.4 Metadata Enricher

Timestamps: scraped_at, published_at (from source), processed_at
Source attribution: which scraper, which URL, what version
Processing status: pending_categorization | categorized | failed

3.3.5 Queue Manager

Hands off normalized records to Categorization System
Uses BullMQ
Priority queue: high-value sources processed first

Inputs

Raw scrape results from Gov/Vendor scrapers
Internal signals from CivStart platform (Discovery sessions, Connection outcomes)

Outputs

normalized_signals table records
Queue messages to Categorization System

3.4 Categorization System

Responsibility: Label, classify, and extract keywords from every signal. Emits keyword feedback to scrapers.

Sub-modules

3.4.1 Category Classifier

Input: Normalized signal
Output: Category tags (from existing 24 CivStart categories + any new dynamic categories)
Implementation:
- Generate embedding for signal text
- Compare to embeddings of existing category descriptions
- Top-K matching categories with confidence scores
- Fallback: LLM classification for low-confidence results

3.4.2 New Category Detector (DBSCAN/HDBSCAN-based)

Signals that don't fit existing categories with confidence > 0.65 get flagged
Clustering approach: Use HDBSCAN (density-based clustering) on embeddings of unfit signals
- HDBSCAN is chosen over K-means because we don't know the number of categories in advance
- It handles arbitrary cluster shapes and marks truly isolated signals as noise (not forced into clusters)
- Works on the vector embeddings already generated by the Categorization System
Workflow:
1. Periodically (weekly) run HDBSCAN over embeddings of signals flagged as "unfit" in past 30 days
2. Clusters with ≥10 signals and high density → candidate new categories
3. LLM summarizes each dense cluster into a proposed category name + description
4. Admin reviews and approves/rejects proposals
5. Approved categories get added to taxonomy, scrapers start watching for them
Alternative library: BERTopic (built on HDBSCAN + BERT embeddings) simplifies this significantly — may be preferable for v1

3.4.3 Keyword Extractor

LLM extracts:
- Key phrases (3-5 word chunks that define the signal)
- Acronyms (FedRAMP, VPAT, NIGP, etc.)
- Emerging terminology (new words not seen before)
- Problem descriptors (what's broken, what's needed)
Stored per-signal for later analysis

3.4.4 Metadata Tagger

Urgency: inferred from language (emergency, immediate, FY26, etc.)
Geography: state, region, urban/rural
Org size: small (<50K pop), mid (50K-500K), large (>500K)
Budget range: if mentioned
Signal type: RFP | complaint | question | announcement | trend

3.4.5 Keyword Feedback Emitter — The self-improvement loop (phased rollout)

Reality check: This module is the hardest part of the system to get right. Plan to ship it disabled initially, run the keyword watchlist manually for the first 2-3 months, and enable automated feedback only once categorization is stable and human-reviewed corrections have tuned the classifier.

Approach (once enabled):

Weekly job: analyze extracted keywords across all signals from past 7 days
Uses density-based clustering (DBSCAN/HDBSCAN) on keyword embeddings:
- Groups semantically similar keywords together (e.g., "data residency," "data sovereignty," "data localization" cluster into one concept)
- Prevents the feedback loop from treating every synonym as a new keyword
- Identifies true density hotspots vs. noise
Identifies within each keyword cluster:
- Clusters growing in frequency week-over-week (trending)
- Clusters appearing mostly in recent data but not historical (emerging)
- Clusters concentrating in specific categories (category-specific signals)
Guard rails (critical to prevent garbage):
- Minimum cluster density threshold before promotion
- Noise filter (isolated one-off keywords never get promoted)
- Human approval queue before watchlist changes take effect (during beta)
- Velocity cap — no more than N new keywords added per week
Writes approved keywords to scraper watchlists
Scrapers pick up watchlist updates on next run

Inputs

Normalized signals from Data Consumption
Existing category taxonomy
Historical keyword database

Outputs

categorized_signals table (signal → categories + keywords + metadata)
Updates to scraper_sources.keywords (feedback to scrapers)
Proposed new categories for admin review

3.5 Connection Engine

Responsibility: Cross-reference all signals, find patterns, generate insights. This is where CivStart's intelligence lives.

Sub-modules

3.5.1 Gov ↔ Vendor Matcher

Finds vendor capabilities that match government demand signals
"3 cities need water quality monitoring, Startup X offers it → recommend outreach"
Uses existing RAG matching infrastructure

3.5.2 Cross-Org Clustering

Groups similar problems across government organizations
"5 cities in different states posted similar cybersecurity RFPs this month"
Vector similarity on signal content + category matching

3.5.3 Supply-Demand Analyzer

Per category, computes:
- Demand signals count (from gov scrapers)
- Supply vendor count (from vendor scrapers + CivStart's registered startups)
- Gap score (high demand + low supply = opportunity)
Tracks over time — is the gap growing or shrinking?

3.5.4 Trend Detector

Time-series analysis per category/keyword
Metrics: volume, velocity (rate of change), sentiment, geography spread
Detects: new trends, accelerating trends, declining trends
Seasonal baseline — adjusts for fiscal year cycles

3.5.5 Competitor Watcher

Monitors vendor changes tracked by Vendor Scraper
Flags when a competitor:
- Launches a product overlapping CivStart's roadmap
- Enters a category CivStart is targeting
- Gets funded (signals scale ambitions)
- Loses a major customer
Semantic comparison to CivStart's current capabilities

3.5.6 Opportunity Scorer

Score = urgency × frequency × strategic_fit × feasibility
Where:
- Urgency: how fast is this trend moving?
- Frequency: how many signals corroborate it?
- Strategic fit: how close to CivStart's current capabilities?
- Feasibility: can CivStart realistically build this?
Ranks opportunities for the daily digest

3.5.7 Report Generator

Daily digest (automated, 5-min read):
- Top 3 new signals
- 1-2 competitor moves worth noting
- Any urgent alerts
Weekly synthesis (automated, deeper analysis):
- Trend summary
- New opportunity scores
- Category heat map changes
Monthly strategic brief (LLM-generated from weekly syntheses):
- Recommended product bets
- Validation of prior predictions
- Strategic landscape shifts

Inputs

Categorized signals from past 7/30/90 days
CivStart's internal data (signals, connections, outcomes)
Historical trend data

Outputs

Insights written to market_insights table
Reports to Export & Integration layer
Alerts for urgent signals

4. Cross-Cutting Concerns

4.1 Observability

What: Metrics, logs, alerts across all modules.

Metrics to track:

Per scraper: last success timestamp, success rate (7d), records produced per run
Pipeline: queue depth, processing latency per stage
Categorization: confidence score distribution, % uncategorized
Cost: LLM tokens used per module per day
Storage: table sizes, growth rate

Alerting rules:

Scraper hasn't succeeded in 48 hours → alert
Queue depth > 1000 records → alert (pipeline stuck)
LLM spend > $X/day → alert
Categorization confidence drops below baseline → alert

Implementation:

Log all events with structured JSON (correlation IDs for tracing)
Store metrics in Postgres (simple) or Prometheus (if scaling)
Alerts via existing Slack integration

4.2 Human Review

What: Interfaces for humans to correct the system and feed corrections back.

Review queues:

Low-confidence categorizations — admin reviews and corrects
Proposed new categories — admin approves/rejects
New source suggestions — admin approves before adding to registry
Competitor change alerts — admin confirms relevance before escalating

Correction feedback:

Admin corrections stored as training data
Weekly: re-train classification prompts using corrections
Track accuracy over time (is the system getting better?)

4.3 Internal Signal Integration

What: Pipe CivStart's own platform data into the same analysis layer.

Sources:

Discovery session submissions (what problems are gov users describing?)
Failed matches (what problems have no good solution?)
Declined startup applications (what patterns trigger rejection?)
Connection outcomes (what kinds of matches succeed/fail?)
Support tickets (what is broken about CivStart itself?)

Integration: These flow into Data Consumption System just like external scraped data, then through Categorization and Connection Engine. First-party signals get higher weight in scoring.

4.4 Historical Layer

What: Time-series storage so trends can be detected.

Approach:

Don't overwrite — append
Each signal has a first_seen_at and updates have observed_at
Weekly snapshots of category-level aggregates for fast time-series queries
Data retention: raw scrapes for 90 days, categorized signals indefinitely

4.5 Outcome Feedback

What: Track which surfaced opportunities CivStart acted on, and what happened.

Mechanism:

Opportunities surfaced in reports get an opportunity_id
Admin can mark opportunity as: acted_on | deferred | dismissed
Follow-up (60/90 days later): did the opportunity play out as predicted?
Outcomes feed back into Opportunity Scorer — tunes future predictions

4.6 Compliance & Ethics

Rules:

Respect robots.txt for every scrape
Rate limits: default 2s between requests per domain, configurable per source
Terms of service: Reddit uses API (not scraping), LinkedIn only public posts (no authenticated scraping)
PII handling: redact personal names, emails, phone numbers from scraped content unless explicitly public and relevant
Data retention: raw scrapes 90 days, anonymized insights indefinitely
Source attribution: every insight must cite its sources (internal audit trail)

Legal posture: Only scrape genuinely public data. Never authenticated content. Never behind paywalls without licenses.

4.7 Cost Control

Tiered processing:

Cheap first: embedding similarity for classification (uses existing embeddings, no LLM call)
Expensive only when needed: GPT-4 only for strategic analysis, not routine classification
Gemini Flash for high-volume extraction
GPT-4o for strategic briefs

Caching:

Don't re-process content that hasn't changed (hash-based cache)
Don't re-embed text that hasn't changed
Cache LLM responses for identical inputs (short TTL)

Budget limits:

Per-module daily budget caps
System pauses and alerts if approaching monthly cap
Admin dashboard shows spend in real-time

Target budget: $500-1000/month total LLM spend at steady state.

4.8 Export & Integration

Outputs:

Admin Dashboard (Next.js admin app) — live view of insights, trends, competitor moves
Slack — daily digest, urgent alerts (use existing Slack integration)
In-app notifications — users return to platform to view reports (no report content in emails)
Linear — auto-create issues for high-scored opportunities (tagged for triage)
API — other internal tools can query insights

4.9 Computational Strategy: What Runs Where

A practical guide to how different parts of the system should be built. The principle: use the cheapest, fastest, most reliable tool that can do the job. Paid AI APIs are the last resort, not the default.

Three tiers of work

Tier 1 — Math and algorithms (no AI, no training needed)

Pure computation. Runs on your server. Free. Instant.

Task	Tool
Grouping similar items	HDBSCAN / DBSCAN clustering
Finding similar items	Cosine similarity on vector embeddings (pgvector)
Detecting trends over time	Statistical methods (Prophet, z-scores, moving averages)
Deduplication	Content hashing + fuzzy string matching
Keyword extraction (basic)	YAKE, RAKE (statistical)

Tier 2 — Pre-built models you download and run locally

Someone else trained these. You install a library and use them. Free. Fast. No API fees.

Task	Tool
Generating embeddings	sentence-transformers (`all-MiniLM-L6-v2` or `bge-small-en`)
Extracting entities (orgs, people, money, dates)	spaCy (`en_core_web_lg`) or GLiNER
Classifying signals into categories	sentence-transformers + cosine similarity to category descriptions
PII detection and redaction	Microsoft Presidio
Keyword extraction (advanced)	KeyBERT (uses BERT embeddings)
Topic modeling / new category detection	BERTopic (wraps HDBSCAN + embeddings)

Tier 3 — Paid AI APIs (use sparingly)

For tasks that genuinely need reasoning, world knowledge, or natural-language synthesis.

Task	Why AI
Writing strategic briefs and reports	Natural-language synthesis
Confirming uncertain classifications	Judgment on edge cases
Filling data gaps when structured extraction fails	Reasoning about missing information
Naming newly-detected categories	Creative synthesis
Explaining anomalies ("why is this unusual?")	Context-aware reasoning

Rough allocation

At steady-state volume:

~85% of system work done in Tier 1 + Tier 2 (free, local)
~15% of system work done in Tier 3 (paid AI API)

This keeps costs low and predictable while preserving AI for the parts where it genuinely adds value.

No model training required

For clarity: this plan does not require training any custom models from scratch. All Tier 2 tools are pre-trained and publicly available. You download them and use them.

Fine-tuning (adjusting pre-trained models on your own data) becomes an option later — after 12+ months of accumulated corrections — but is not needed to launch. Start with off-the-shelf pre-trained models plus LLM fallback.

Cost comparison at 100K documents/month

Approach	Monthly cost	Why
LLM for everything (GPT-4o)	$800-1,500	Every task hits paid API
Mixed (OpenAI embeddings + GPT)	$200-400	Still paying per embedding call
Tier 1 + 2 + 3 as described above	$50-150	Local models do 85% of work; LLM only for reports + judgment

Running local models is 5-10x cheaper than LLM-heavy architectures at scale.

Infrastructure implications

Local model inference needs some resources:

Small models (sentence-transformers MiniLM, spaCy) run fine on CPU
Larger models benefit from a small GPU but aren't required for MVP
Recommended: a dedicated ML worker service (Python + FastAPI or ONNX Runtime in Node.js) that NestJS calls for inference
Additional infrastructure cost: ~$30-60/month for the ML worker on Fargate

Implementation note

Python vs Node.js: Most good ML libraries are Python-first. Options:

Python microservice: Small FastAPI service running the ML libraries, called from NestJS over HTTP. Simpler, cleaner boundaries.
ONNX Runtime in Node.js: Export models to ONNX format, run directly in Node. Tighter integration, steeper learning curve.

Recommendation for MVP: Option 1 (Python microservice). Deploy as a sidecar container or separate Fargate service.

4.10 Scale & Concurrency Architecture

Long-running scrapes create an unusual operational challenge: you can never match concurrent demand with concurrent execution. Even a modest burst of user requests creates problems if the system is designed naively. This section captures the architectural principles, real-world problems, and cost tradeoffs — not a capacity plan for a specific user count.

Why Naive Concurrency Fails

Scraping jobs in this system are fundamentally different from typical web requests:

They take hours, not milliseconds. A Playwright-based scrape of a government RFP portal may run 1-8 hours due to rate limiting on the target site, pagination depth, and extraction of nested documents.
They hold resources throughout. A running Chromium process consumes 300-500 MB of memory and fractional CPU for the entire duration. You cannot release these resources mid-job.
They cannot be safely parallelized against a single source. Ten workers scraping the same portal will get rate-limited, banned, or both.
They fail unpredictably. Target sites change structure, timeouts occur, CAPTCHAs appear. Recovery logic matters more than raw speed.

Given these constraints, trying to run N scrapes for N simultaneous users is the wrong mental model. The right model is:

Users submit requests → system absorbs and deduplicates them → a small pool of workers processes work in priority order → users are notified when their result is ready.

This is the same pattern used by video rendering services, data warehouse ETL tools, and any system that processes long-running jobs on behalf of many users.

Core Architectural Principles

1. Queue everything — never execute synchronously.

Every user trigger writes to a durable queue (SQS, BullMQ, or similar). Workers pull from the queue. This:

Absorbs bursts without scaling resources
Provides natural backpressure
Enables retries and failure recovery
Makes cost predictable

2. Cache aggressively — most requests are not unique.

Government data doesn't change minute-to-minute. California's procurement patterns from 3 days ago are still relevant today. A well-tuned cache can serve 50-80% of requests from existing work:

Exact cache hits: same query submitted recently → return instantly
Semantic cache hits: similar query served by existing research → serve with minor customization
Partial cache hits: 60% of the answer already exists → only scrape the gap

Cache TTLs should be topic-specific: pricing data might have a 30-day TTL, active RFP listings a 1-day TTL.

3. Concurrency caps — small and deliberate.

Limit the number of simultaneous scrapes regardless of demand. This protects:

Your infrastructure budget
Your IP reputation (too many concurrent requests from one IP range triggers bans)
Target site stability (don't DDoS the resources you depend on)
LLM API quotas
Database connection pools

Reasonable defaults:

MVP / Phase 2: max 3-5 concurrent workers
Growth / Phase 3: max 10-15 concurrent workers
Mature / Phase 4+: max 20-30 concurrent workers

4. Users wait — set expectations, deliver asynchronously.

Users don't need instant results for intelligence work. They need reliable results. "Your report will be ready in 1-4 hours" is a completely acceptable UX when:

The wait time is communicated upfront
Related cached content is shown while they wait
They receive a notification when ready
The final result is worth the wait

5. Work is shared — one scrape serves many.

When multiple users ask for overlapping research, don't run it multiple times. Detect the overlap, run once, serve all requesters from the single result. This is where cache architecture pays off as the user base grows.

6. Checkpointing — long jobs must be resumable.

A scrape that crashes at hour 3 should resume from where it left off, not restart from zero. Store incremental progress; design scrapers to pick up mid-job.

7. Cost attribution — per-customer tracking from day one.

Even in beta, track the LLM tokens, compute time, and cache hits attributable to each customer. Without this, you cannot price fairly or identify unprofitable customers later.

Real-World Problems You Will Encounter

These are not theoretical. Any non-trivial scraping system hits all of them.

IP blocking by target sites

AWS IP ranges are well-known to anti-scraping tools. Gov sites increasingly block cloud provider IP ranges aggressively. This happens at low concurrency, not just high — some sites flag 5-10 concurrent requests from the same IP range.

Mitigations, in order of escalation:

Respect robots.txt and rate limits (always)
Use realistic user agents and request timing
Distribute across multiple NAT gateways (IP diversity within AWS)
Use residential proxy services (Bright Data, Smartproxy, Oxylabs) — $300-2,000/month depending on volume
For especially hostile sites, direct human fetching may be the only option

LLM API rate limits

Provider rate limits exist at every tier. At small scale, Gemini's free tier (360 requests/minute) may be sufficient. As volume grows:

OpenAI tier 1 starts at ~500 requests/min for most models
Anthropic enforces similar limits
Bursts during peak usage will hit these limits

Mitigations:

Queue LLM calls through a rate-limited service layer
Spend commitments to unlock higher tiers
Multi-provider failover (if OpenAI rate-limits, fall back to Gemini)
Cache LLM outputs aggressively

Database connection exhaustion

Postgres default max_connections is typically 100. Between your application, admin dashboard, scheduled jobs, and scraper workers, you exhaust this quickly.

Mitigations:

PgBouncer or RDS Proxy for connection pooling
Higher-tier RDS instance with more connections
Design workers to hold connections only while actively writing

Work loss on failure

A scraper that crashes at hour 3 of a 4-hour job has wasted 3 hours of compute and LLM budget. Without checkpointing, this happens frequently in production.

Mitigations:

Save progress after each page/section
Persist intermediate state in database, not memory
Design scrapers to detect their own checkpoints on restart
Use durable job orchestration (Temporal, Step Functions) for complex workflows

Observability at depth

Debugging why one specific scrape failed 3 hours in, among 20 other concurrent jobs, requires structured logging, correlation IDs, and per-job trace aggregation. This is not optional at scale — without it, failures become unfixable.

Mitigations:

Structured JSON logging from day one
Per-scrape correlation IDs threaded through every log line
CloudWatch Logs Insights queries for common failure patterns
Consider OpenTelemetry for distributed tracing

Cost surprises

It is easy to run up unexpected bills:

NAT gateway traffic charges ($0.045 per GB processed) add up fast
CloudWatch log ingestion at $0.50/GB becomes the third-largest line item
LLM spending can 10x overnight if caching breaks
Fargate tasks left running after a bug can burn hundreds of dollars before anyone notices

Mitigations:

Hard spending caps per module
Billing alerts at 50%, 75%, 100% of expected monthly
Daily cost dashboard reviewed by an owner
Circuit breakers on LLM spend — halt if daily budget exceeded

Operational Cost Ranges

These are operational ranges, not capacity math. Costs vary significantly based on actual usage patterns, cache hit rates, and source mix.

Small-scale (MVP / Phase 2)

3-5 concurrent workers
Single jurisdiction or small multi-source mix
Caching in early stages (low hit rate)
No proxy service yet
Monthly range: $150-400 infrastructure + LLM

Growth-stage (Phase 3)

10-15 concurrent workers
Multiple jurisdictions
Caching mature (40-60% hit rate)
Residential proxy service added
Per-customer cost tracking in place
Monthly range: $500-1,500 infrastructure + LLM + proxies

Mature (Phase 4+)

20-30 concurrent workers
Broad source coverage
Caching very mature (60-80% hit rate)
Multi-region scraping for IP diversity
Dedicated LLM API tier
Monthly range: $1,500-4,000 all-in

Always-bigger-than-expected items to budget for:

Data transfer (NAT gateway + egress): 15-25% of compute cost
Observability (CloudWatch, monitoring): $50-300/month at any real scale
Proxy services: $300-2,000/month if needed
Variable LLM costs during bursts: can 3-5x baseline temporarily
One-time IP reputation recovery if you get flagged: $100-500 for premium proxy access

Recommended Baseline Architecture

For the Intelligence System, a minimum viable production architecture:

┌──────────────────────────────────────────────────────────────┐
│                                                              │
│   User submits trigger (API, UI, or scheduled)               │
│        ↓                                                     │
│   [Deduplicator] → hits cache? Yes → serve immediately       │
│        ↓                                                     │
│        No → [SQS queue with priority]                        │
│                 ↓                                             │
│   [Worker pool — N=3-20 Fargate tasks]                       │
│       - Each worker pulls one job                            │
│       - Runs scraping + categorization + synthesis           │
│       - Writes checkpoints to Postgres                       │
│       - Writes final result to cache + DB                    │
│                 ↓                                             │
│   [Notification service] → user receives "report ready" link │
│        ↓                                                     │
│   User visits platform → views report on dashboard           │
│                                                              │
│   Cross-cutting:                                             │
│   - Rate limiter (per target site + per LLM provider)        │
│   - Cost tracker (per customer + per job)                    │
│   - Observability (structured logs + correlation IDs)        │
│   - Spending circuit breakers                                │
│                                                              │
└──────────────────────────────────────────────────────────────┘

AWS service mapping:

Queue: SQS (or BullMQ on ElastiCache if preferring existing Redis)
Workers: ECS Fargate tasks (or EC2 with auto-scaling group)
Cache: ElastiCache Redis for semantic cache, S3 for finished reports
Rate limiting: Lambda + DynamoDB for distributed rate limiter state
Notifications: SNS → SES (emails) and platform in-app delivery
Orchestration (if jobs are complex): Step Functions for checkpoint workflows

What Success Looks Like

A well-architected system should handle arbitrary user burst without catastrophic failure. Concrete targets:

No job loss during bursts: SQS absorbs all incoming work
Cost scales sublinearly with users: because cache absorbs most repeat requests
No dependence on synchronous user-facing performance: async delivery is the norm
Failures are recoverable: checkpointing means crashes cost minutes, not hours
Per-customer economics visible: we always know who cost us what

If those properties hold, the system can grow from its first beta customer to many without architectural rewrites — only scaling knobs.

Build-Order Implication

Because queue and cache architecture are hard to retrofit later, they should be in place from Phase 1 even if workload is tiny. Specifically:

Phase 1 MVP should already route through SQS (even if queue depth is usually zero)
Phase 1 MVP should already have per-source caching (even if hit rate is low)
Phase 1 MVP should already track per-job costs (even if costs are trivial)

Retrofitting these patterns later is expensive. Including them from day one is cheap and prevents the "we built a naive system and now we're rewriting it" trap.

5. Data Model

New Prisma Models

// Source registry — tracks where data comes from
model ScraperSource {
  id              String   @id @default(uuid())
  name            String
  type            String   // "government" | "vendor"
  url             String
  scraperKey      String   // identifier for the worker class
  config          Json     // source-specific config
  keywords        Json     // current watchlist (updated by feedback loop)
  scrapeFrequency String   // "hourly" | "daily" | "weekly"
  lastScrapedAt   DateTime?
  lastSuccessAt   DateTime?
  status          String   // "active" | "paused" | "failed"
  failureCount    Int      @default(0)
  createdAt       DateTime @default(now())
  updatedAt       DateTime @updatedAt

  rawResults      RawScrapeResult[]
}

// Raw scrape results — before processing
model RawScrapeResult {
  id          String   @id @default(uuid())
  sourceId    String
  source      ScraperSource @relation(fields: [sourceId], references: [id])
  content     String   @db.Text
  contentType String   // "html" | "pdf" | "json" | "text"
  url         String
  contentHash String   // for dedup
  scrapedAt   DateTime @default(now())
  processed   Boolean  @default(false)

  normalizedSignals NormalizedSignal[]

  @@index([sourceId, scrapedAt])
  @@index([contentHash])
}

// Normalized signals — cleaned, entity-extracted
model NormalizedSignal {
  id              String   @id @default(uuid())
  rawResultId     String
  rawResult       RawScrapeResult @relation(fields: [rawResultId], references: [id])
  title           String
  content         String   @db.Text
  entities        Json     // extracted orgs, people, amounts, etc.
  publishedAt     DateTime?
  sourceUrl       String
  contentHash     String
  processingStatus String  // "pending_categorization" | "categorized" | "failed"
  createdAt       DateTime @default(now())

  categorization  CategorizedSignal?

  @@index([processingStatus])
  @@index([contentHash])
}

// Categorized signals — labeled, keyworded
model CategorizedSignal {
  id                String   @id @default(uuid())
  normalizedSignalId String  @unique
  normalizedSignal  NormalizedSignal @relation(fields: [normalizedSignalId], references: [id])
  categories        String[] // maps to existing 24 solution categories
  keywords          String[] // extracted phrases
  signalType        String   // "rfp" | "complaint" | "question" | "announcement" | "trend"
  urgency           String?  // "low" | "medium" | "high" | "emergency"
  geography         Json?    // { state, region, city, orgSize }
  budgetRange       Json?    // { min, max, currency }
  confidence        Float
  embedding         Float[]  // for similarity search
  categorizedAt     DateTime @default(now())

  @@index([categorizedAt])
}

// Tracked vendors
model TrackedVendor {
  id            String   @id @default(uuid())
  name          String
  website       String
  category      String   // "competitor" | "adjacent" | "startup" | "incumbent"
  trackDepth    String   // "light" | "medium" | "deep"
  status        String   // "active" | "dead" | "acquired"
  metadata      Json     // funding, employee count, etc.
  createdAt     DateTime @default(now())
  updatedAt     DateTime @updatedAt

  changes       VendorChange[]
}

// Vendor changes detected over time
model VendorChange {
  id          String   @id @default(uuid())
  vendorId    String
  vendor      TrackedVendor @relation(fields: [vendorId], references: [id])
  changeType  String   // "new_product" | "pricing_change" | "new_customer" | "leadership" | "funding"
  description String
  details     Json
  detectedAt  DateTime @default(now())
  relevanceScore Float
}

// Market insights — the output of the Connection Engine
model MarketInsight {
  id              String   @id @default(uuid())
  insightType     String   // "gap" | "trend" | "competitor_move" | "opportunity" | "alert"
  title           String
  description     String   @db.Text
  categories      String[]
  relatedSignalIds String[]
  score           Float
  urgency         String
  supportingData  Json
  status          String   // "new" | "reviewed" | "acted_on" | "deferred" | "dismissed"
  createdAt       DateTime @default(now())
  reviewedAt      DateTime?
  outcomeNote     String?  @db.Text

  @@index([status, createdAt])
  @@index([score])
}

// Keyword tracking — for feedback loop
model TrackedKeyword {
  id              String   @id @default(uuid())
  keyword         String   @unique
  firstSeenAt     DateTime
  lastSeenAt      DateTime
  occurrenceCount Int      @default(1)
  velocityScore   Float    // rate of change
  isWatched       Boolean  @default(false) // in scraper watchlists
  relatedCategories String[]
}

// Observability
model SystemMetric {
  id          String   @id @default(uuid())
  metricName  String
  value       Float
  dimensions  Json     // { module, source, etc. }
  recordedAt  DateTime @default(now())

  @@index([metricName, recordedAt])
}

Relationships to Existing CivStart Models

Signal (existing) — first-party signals feed into NormalizedSignal pipeline
Connection (existing) — outcome data feeds MarketInsight validation
StartupContact (existing) — cross-referenced with TrackedVendor for gov ↔ vendor matching

6. Implementation Roadmap (Realistic)

Reality note: Original estimates were optimistic. These revised estimates assume a solo developer with AI assistance, part-time hours, and real-world issues (scraper breakages, prompt tuning, edge cases). Expect 2x overrun on any given phase. That is normal, not failure.

Phase 1: Foundation (6-10 weeks, not 3)

Goal: End-to-end thin slice for ONE jurisdiction and ONE source type.

Set up Prisma models (ScraperSource, RawScrapeResult, NormalizedSignal, CategorizedSignal, MarketInsight)
Build Data Consumption System skeleton (normalizer + entity extractor + queue)
Build Categorization System v1 (reuse existing embeddings, classify against 24 categories — no feedback loop yet)
Build one Government Scraper (SAM.gov is easiest — has an API)
Build basic admin dashboard page (list of categorized signals)
Set up cost tracking from day one
Manually operate for 2-3 weeks — read every output, fix issues

Milestone: Scrape SAM.gov daily → clean → categorize → admin reviews in dashboard. No automation yet, no customers yet.

What will likely break in this phase: PDF parsing, entity extraction accuracy, classification confidence thresholds. Budget time to fix.

Phase 2: Expansion (8-12 weeks)

Goal: Add sources carefully; prove pipeline handles variation.

Add 1-2 more gov sources (e.g., one state portal + Reddit)
Add deduplication in Data Consumption (fuzzy match across sources)
Observability: metrics dashboard, basic alerting
Compliance: robots.txt checks, rate limiting, logging
Beta sale validation — hand-sell to 3-5 CivStart startup customers as "early access beta" (see Revenue Model doc)
First Vendor Scraper (Product Hunt gov-tech tag)

Milestone: 3-4 sources feeding pipeline. First 3-5 paying beta customers.

Phase 3: Intelligence Layer (3-4 months)

Goal: Turn data into something worth paying for.

Build Connection Engine v1 (Gov ↔ Vendor Matcher, Cross-Org Clustering, Supply-Demand Analyzer)
Trend Detector (simple velocity metrics per category — no DBSCAN yet)
Opportunity Scorer (basic; expect months of tuning)
Daily digest email
Human review queue and correction interface (essential for accuracy)

Milestone: First useful weekly insights. Beta customers report real value. Graduate to higher beta pricing.

Phase 4: Self-Improvement (3-6 months)

Goal: Feedback loop — but only once categorization is stable enough to trust.

Keyword Extractor in Categorization System (KeyBERT or YAKE)
DBSCAN/HDBSCAN clustering for keyword grouping and new category detection (BERTopic simplifies this significantly)
Keyword Feedback Emitter with guard rails (human approval queue)
Internal Signal Integration (CivStart data into pipeline)
Outcome Feedback tracking

Milestone: System gradually improves itself. Scraper coverage expands automatically based on detected trends.

Phase 5: Maturation (6-12 months total cumulative)

Goal: Production-grade, generally available.

Vendor Scraper expansion (Crunchbase, company websites, news sources)
Change Tracker for vendors (semantic diff)
Historical Layer (snapshots, time-series queries, seasonality adjustment)
Monthly strategic brief generator (LLM-assisted)
Graduate from beta pricing to full pricing
Linear auto-issue creation for high-scored opportunities

Milestone: System runs itself with minimal intervention. Revenue meaningfully contributes.

Phase 6: Scale & Polish (Year 2+)

Add sources as customer demand dictates
Refine prompts based on human corrections
Expand category taxonomy via validated HDBSCAN clusters
Build predictive analytics
Validate predictions against reality (outcome feedback loop)
Consider API access and white-label offerings

Honest Timeline Summary

Phase	Optimistic	Realistic	What It Delivers
Phase 1	3 weeks	6-10 weeks	MVP, 1 source, manual operation
Phase 2	3 weeks	8-12 weeks	Multi-source + first beta customers
Phase 3	4 weeks	3-4 months	Useful insights, growing beta
Phase 4	4 weeks	3-6 months	Self-improving system
Phase 5	6 weeks	Cumulative 6-12 months	Production-grade, general availability
Phase 6	Ongoing	Year 2+	Scale and maturity

Total realistic time to production-grade system: 9-15 months of dedicated work. Probably closer to 18 months with part-time work.

7. Risk Register

Risk	Probability	Impact	Mitigation
Scrapers break when sites change	High	Medium	Adaptation layer detects + alerts; budget engineering time for repairs
LLM costs spiral	Medium	High	Tiered processing, caching, hard budget caps
Legal issues from scraping	Low	High	Strict compliance rules, public data only, TOS respect
Signal overload — too much noise	High	Medium	Strong filtering, opportunity scoring, human review queue
Categorization accuracy poor	Medium	Medium	Human review feedback loop, regular prompt tuning
System becomes abandonware	Medium	High	Clear ownership, automated digests that require engagement
Scrapers banned (IP-level)	Low	Medium	Rate limiting, rotating IPs if needed, respect 429 responses
Data privacy / PII leak	Low	High	Redaction rules, audit trail, minimal PII retention
Competitors detect scraping	Low	Low	Only public data; no competitive intelligence that isn't legally accessible
Indefinite storage growth	Medium	Low	90-day retention for raw scrapes, aggregation for older data

8. Ownership & Operations

Roles (to be assigned by CivStart leadership if approved)

Product Owner: TBD (decides what matters, reviews digests, acts on insights)
Engineering: TBD (constructs and maintains the system) — Bijo available for implementation work
Reviewer: TBD (corrects AI mistakes, approves new categories)

Operating Model

Daily: Read digest (5 min)
Weekly: Review system metrics, triage human review queue (30 min)
Monthly: Strategic brief review, adjust scoring weights if needed (1 hr)
Quarterly: Evaluate outcomes — did last quarter's predictions play out? (2 hrs)

On-Call

Critical alerts (scraper down 48+ hours, LLM budget exceeded) → on-call engineer
Non-critical (low-confidence signals, new category proposals) → review queue

Handoff Plan

All runbooks, prompts, and configs checked into repo
System should be operable by any engineer with 2-day onboarding
Documentation updated quarterly

9. Open Questions

What's the MVP scope? Should Phase 1 include Vendor Scraper or defer to Phase 2?
Which state RFP portals are highest-signal? Need research to prioritize.
Should LinkedIn be scraped at all? TOS is restrictive — maybe limit to public posts only.
Historical data backfill? Do we scrape 6 months of archives on launch, or start from day zero?
Who approves new categories? As the system grows, category taxonomy decisions affect strategy — needs a designated product owner.
What's the escalation path? If system surfaces a critical competitor move, who gets paged?

Appendix A: Existing Infrastructure to Leverage

Playwright MCP — already installed for scraping
OpenAI + Gemini — already integrated
Prisma + PostgreSQL — primary data store
SignalEmbedding model — pattern for vector storage (reuse approach)
24-category taxonomy — seed for classification
NestJS cron scheduling — standard pattern in codebase
Admin dashboard (Next.js) — extend for new views
Slack integration — for alerts
Redis/BullMQ — Redis already in Docker Compose

Appendix B: Source Priority List

Tier 1 (build first):

SAM.gov (highest-volume gov signal source)
Reddit r/govtech (real complaints in real language)
Product Hunt (vendor signals, API available)
GovTech magazine (curated news)

Tier 2 (add in Phase 2):

State RFP portals (California, Texas, New York, Florida first)
Council minutes for top 50 cities (scripted PDF extraction)
Crunchbase (vendor funding signals)
LinkedIn (gov job postings only, public content)

Tier 3 (Phase 5+):

Individual city budget documents
GitHub (civic tech repos)
Conference agendas
International sources (UK GDS, Estonia e-gov, Singapore)

End of Specification

This document is a living specification. Update as the system evolves.

Table of Contents​

1. Vision​

The Problem​

The Solution​

Key Principles​

Success Criteria​

2. High-Level Architecture​

Technology Stack​

3. Core Module Specifications​

3.1 Government Scraper System​

Sub-modules​

Inputs​

Outputs​

Edge Cases​

3.2 Vendor Scraper System​

Sub-modules​

Inputs​

Outputs​

3.3 Data Consumption System​

Sub-modules​

Inputs​

Outputs​

3.4 Categorization System​

Sub-modules​

Inputs​

Outputs​

3.5 Connection Engine​

Sub-modules​

Inputs​

Outputs​

4. Cross-Cutting Concerns​

4.1 Observability​

4.2 Human Review​

4.3 Internal Signal Integration​

4.4 Historical Layer​

4.5 Outcome Feedback​

4.6 Compliance & Ethics​

4.7 Cost Control​

4.8 Export & Integration​

4.9 Computational Strategy: What Runs Where​

Three tiers of work​

Rough allocation​

No model training required​

Cost comparison at 100K documents/month​

Infrastructure implications​

Implementation note​

4.10 Scale & Concurrency Architecture​

Why Naive Concurrency Fails​

Core Architectural Principles​

Real-World Problems You Will Encounter​

Operational Cost Ranges​

Recommended Baseline Architecture​

What Success Looks Like​

Build-Order Implication​

5. Data Model​

New Prisma Models​

Relationships to Existing CivStart Models​

6. Implementation Roadmap (Realistic)​

Phase 1: Foundation (6-10 weeks, not 3)​

Phase 2: Expansion (8-12 weeks)​

Phase 3: Intelligence Layer (3-4 months)​

Phase 4: Self-Improvement (3-6 months)​

Phase 5: Maturation (6-12 months total cumulative)​

Phase 6: Scale & Polish (Year 2+)​

Honest Timeline Summary​

7. Risk Register​

8. Ownership & Operations​

Roles (to be assigned by CivStart leadership if approved)​

Operating Model​

On-Call​

Handoff Plan​

9. Open Questions​

Appendix A: Existing Infrastructure to Leverage​

Appendix B: Source Priority List​

Table of Contents

1. Vision

The Problem

The Solution

Key Principles

Success Criteria

2. High-Level Architecture

Technology Stack

3. Core Module Specifications

3.1 Government Scraper System

Sub-modules

Inputs

Outputs

Edge Cases

3.2 Vendor Scraper System

Sub-modules

Inputs

Outputs

3.3 Data Consumption System

Sub-modules

Inputs

Outputs

3.4 Categorization System

Sub-modules

Inputs

Outputs

3.5 Connection Engine

Sub-modules

Inputs

Outputs

4. Cross-Cutting Concerns

4.1 Observability

4.2 Human Review

4.3 Internal Signal Integration

4.4 Historical Layer

4.5 Outcome Feedback

4.6 Compliance & Ethics

4.7 Cost Control

4.8 Export & Integration

4.9 Computational Strategy: What Runs Where

Three tiers of work

Rough allocation

No model training required

Cost comparison at 100K documents/month

Infrastructure implications

Implementation note

4.10 Scale & Concurrency Architecture

Why Naive Concurrency Fails

Core Architectural Principles

Real-World Problems You Will Encounter

Operational Cost Ranges

Recommended Baseline Architecture

What Success Looks Like

Build-Order Implication

5. Data Model

New Prisma Models

Relationships to Existing CivStart Models

6. Implementation Roadmap (Realistic)

Phase 1: Foundation (6-10 weeks, not 3)

Phase 2: Expansion (8-12 weeks)

Phase 3: Intelligence Layer (3-4 months)

Phase 4: Self-Improvement (3-6 months)

Phase 5: Maturation (6-12 months total cumulative)

Phase 6: Scale & Polish (Year 2+)

Honest Timeline Summary

7. Risk Register

8. Ownership & Operations

Roles (to be assigned by CivStart leadership if approved)

Operating Model

On-Call

Handoff Plan

9. Open Questions

Appendix A: Existing Infrastructure to Leverage

Appendix B: Source Priority List