Skip to main content

WebSocket & Real-Time Enhancements Plan

Executive Summary​

This document outlines the plan to enhance the CivStart platform's real-time capabilities by reviewing and improving the existing WebSocket implementation and ensuring all critical pages receive live updates.

Created: 2025-11-05 Branch: feature/websocket-realtime-enhancements Status: Planning Phase


Current Architecture Review​

✅ What's Already Implemented​

Backend Infrastructure​

  1. WebSocket Gateways (apps/backend/src/core/)

    • AdminWebSocketGateway - Handles admin panel connections (/admin namespace)
    • ClientWebSocketGateway - Handles startup/client connections (/client namespace)
    • Both use Socket.IO with proper CORS configuration
  2. WebSocket Service (websocket.service.ts)

    • Centralized service for emitting events
    • Supports both admin and client-side broadcasts
    • Event types covered:
      • Connection events (created, updated)
      • Signal events (created, updated, assigned)
      • Match events (generated)
      • Credit events (updated)
      • Assignment events (created, responseChanged, viewed)
      • Metrics updates
      • Notifications
  3. Notification Service (notification.service.ts)

    • Multi-channel notification system (email, SMS, push, in-app)
      • Template-based content generation
      • Scheduled delivery with retry logic
      • Comprehensive statistics tracking
      • Currently focuses on scheduled notifications, not real-time broadcasts
  4. Airtable Webhook Integration (airtable-sync.service.ts)

    • Handles webhook payloads from Airtable
    • Syncs organizations, contacts, signals, startups
    • Maps Airtable changes to Prisma models
    • Gap: Not integrated with WebSocket broadcasts

Frontend Infrastructure​

  1. WebSocket Client (apps/frontend/lib/websocket.ts & apps/admin/lib/websocket.ts)

    • Socket.IO client wrapper
    • Type-safe event handling
    • Auto-reconnection logic
    • Startup-specific room joining
  2. WebSocket Provider (WebSocketProvider.tsx)

    • React context for WebSocket state
    • Auto-connects when user is authenticated
    • Prevents duplicate connections with global state management
    • Subscription management for event listeners
    • Currently live on frontend: Credit updates
  3. Admin WebSocket Provider (apps/admin/lib/websocket-provider.tsx)

    • Similar to client provider but for admin panel
    • Auto-connects authenticated admins
    • Gap: Limited event subscriptions currently active
  4. Notification Listeners

    • NotificationListener.tsx components in both admin and frontend
    • Display toast notifications for real-time events
    • Gap: Only partially implemented across all pages

Identified Gaps & Enhancement Areas​

🔴 Critical Gaps​

  1. Webhook → WebSocket Bridge Missing

    • Airtable webhook changes don't trigger WebSocket broadcasts
    • Manual data sync required to see changes from Airtable
    • Impact: Admins don't see real-time updates from Airtable changes
  2. Incomplete Event Coverage

    • Many backend services emit events, but not all are consumed on frontend
    • Signal status changes not always broadcast in real-time
    • Startup assignments to signals not reflected immediately
  3. Missing Real-Time Pages

    • Several critical admin pages don't subscribe to WebSocket events
    • Dashboard metrics not updating in real-time
    • Signal pool page requires manual refresh
  4. No Notification Queue Processing

    • NotificationService has scheduled processing, but no real-time trigger
    • User inbox updates aren't pushed via WebSocket
    • Impact: Users must refresh to see new notifications

🟡 Enhancement Opportunities​

  1. Enhanced Notification System

    • Integrate NotificationService with WebSocket for instant delivery
    • Add in-app notification queue with real-time updates
    • Implement notification badges with live counts
  2. Better Error Handling

    • No fallback strategy when WebSocket connection fails
    • Limited retry logic on client-side reconnections
    • Missing offline state indicators
  3. Optimistic UI Updates

    • Current implementation waits for server confirmation
    • Could implement optimistic updates with rollback on error
  4. Event Filtering & Targeting

    • All admins receive all events (potential performance issue)
    • Could implement event filtering based on admin permissions/interests
    • Startup-specific filtering already implemented for clients

Pages Requiring Real-Time Updates​

Admin Panel (apps/admin/)​

PageCurrent StateEvents NeededPriority
/dashboard❌ Staticmetrics:updated, connection:created, signal:created🔴 High
/signals⚠️ Partialsignal:created, signal:updated, match:generated🔴 High
/signal-pool❌ Staticsignal:created, signal:updated, signal:statusChanged🔴 High
/startups❌ Staticstartup:created, startup:updated, credits:changed🟡 Medium
/startups/[id]✅ Goodcredits:updated, assignment:created, connection:updated🟢 Low
/connections⚠️ Partialconnection:created, connection:updated, connection:statusChanged🔴 High
/contact-requests❌ StaticcontactRequest:created, contactRequest:updated🟡 Medium
/analytics❌ Staticmetrics:updated, analytics:updated🟢 Low
/organizations❌ Staticorganization:synced (from Airtable)🟢 Low

Client/Startup Portal (apps/frontend/)​

PageCurrent StateEvents NeededPriority
/govintel/inbox⚠️ Partialassignment:created, signal:assigned, notification:new🔴 High
/govintel/pool❌ Staticsignal:published, signal:removed🔴 High
/govintel/connections❌ Staticconnection:statusChanged, connection:updated🟡 Medium
Dashboard (overview)⚠️ Partialcredits:updated, assignment:new, metrics:updated🟡 Medium

Enhanced WebSocket Event Architecture​

Proposed Event Categories​

1. Data Sync Events (from Airtable webhooks)​

// When Airtable data changes
"airtable:organization:synced";
"airtable:contact:synced";
"airtable:signal:synced";
"airtable:interaction:synced";

2. Signal Lifecycle Events​

"signal:created"; // New signal generated
"signal:updated"; // Signal data changed
"signal:published"; // Signal made public in pool
"signal:removed"; // Signal removed from pool
"signal:assigned"; // Signal assigned to startup(s)
"signal:statusChanged"; // Approval status changed
"signal:matchGenerated"; // AI matches created for signal

3. Assignment & Response Events​

"assignment:created"; // Signal assigned to specific startup
"assignment:viewed"; // Startup viewed the assignment
"assignment:interested"; // Startup marked as interested
"assignment:declined"; // Startup declined
"assignment:responseChanged"; // Response status updated

4. Connection Events​

"connection:created"; // New connection request
"connection:updated"; // Connection details changed
"connection:statusChanged"; // Status transition (pending → approved, etc.)

5. Credit Events (already implemented)​

"credits:updated"; // Credit balance changed
"credits:deducted"; // Credits used
"credits:allocated"; // New credits added
"credits:tierChanged"; // Tier upgraded/downgraded

6. Notification Events (new)​

"notification:new"; // New notification created
"notification:read"; // Notification marked as read
"notification:deleted"; // Notification dismissed
"notification:countChanged"; // Unread count updated

7. Metrics & Analytics Events​

"metrics:updated"; // Dashboard metrics refreshed
"analytics:updated"; // Analytics data updated

Implementation Phases​

Phase 1: Foundation Enhancement (Week 1)​

Goal: Establish robust foundation and fix critical gaps

  1. ✅ Create feature branch

  2. ✅ Document current architecture

  3. Webhook → WebSocket Bridge

    • Modify AirtableSyncService to emit WebSocket events after successful syncs
    • Add new event types for Airtable sync operations
    • Test with actual Airtable webhook triggers
  4. Enhanced Notification Service Integration

    • Add WebSocket emission to NotificationService.processNotification()
    • Create real-time notification event for in-app delivery
    • Update UserInbox creation to trigger WebSocket events
  5. Client-Side Event Listeners

    • Create reusable hooks for common event subscriptions
    • useSignalUpdates() - Subscribe to signal events
    • useConnectionUpdates() - Subscribe to connection events
    • useNotifications() - Subscribe to notification events
    • useMetrics() - Subscribe to metrics updates

Phase 2: Admin Panel Real-Time Updates (Week 1-2)​

Goal: Make all admin pages live-updating

  1. Dashboard Page (/dashboard)

    • Subscribe to metrics:updated for live stat updates
    • Subscribe to signal:created to show recent signals
    • Subscribe to connection:created for activity feed
    • Implement auto-refresh with smooth transitions
  2. Signals Page (/signals)

    • Subscribe to signal:created, signal:updated
    • Add real-time filtering when signals change status
    • Show live match generation progress
    • Optimistic UI for signal approvals
  3. Signal Pool Page (/signal-pool)

    • Subscribe to signal:published, signal:removed
    • Real-time pool updates as signals are added/removed
    • Live startup assignment count updates
  4. Connections Page (/connections)

    • Subscribe to connection:created, connection:updated
    • Live status badge updates
    • Real-time filtering when status changes
  5. Startups Page (/startups and /startups/[id])

    • Subscribe to credits:updated for live credit displays
    • Subscribe to assignment:created for new assignments
    • Real-time tier change indicators

Phase 3: Client Portal Real-Time Updates (Week 2)​

Goal: Make startup portal fully real-time

  1. Inbox Page (/govintel/inbox)

    • Subscribe to assignment:created for instant notifications
    • Subscribe to notification:new for in-app alerts
    • Live unread count badge
    • Sound/visual notification for new assignments
  2. Signal Pool Page (/govintel/pool)

    • Subscribe to signal:published for new public signals
    • Subscribe to signal:removed to remove from list
    • Real-time cost display updates
  3. Connections Page (/govintel/connections)

    • Subscribe to connection:statusChanged
    • Live connection request status updates
    • Optimistic UI for connection requests

Phase 4: Advanced Features (Week 2-3)​

Goal: Polish and optimize

  1. Notification Badge System

    • Persistent unread notification count
    • Real-time badge updates across all pages
    • Notification center dropdown with live updates
  2. Optimistic UI Updates

    • Implement optimistic updates for admin actions
    • Add rollback logic for failed operations
    • Visual feedback for pending operations
  3. Offline Support & Reconnection

    • Better offline state indicators
    • Queue events when offline, replay on reconnect
    • Automatic data refresh after reconnection
  4. Performance Optimizations

    • Event debouncing for high-frequency updates
    • Pagination-aware event handling
    • Selective event subscription based on current page
  5. Admin Event Filtering

    • Allow admins to customize which events they want to see
    • Implement event preferences in admin settings
    • Reduce unnecessary re-renders

Technical Implementation Details​

Backend Changes Needed​

  1. Modify services to emit WebSocket events:
// Example: Signal Assignment Service
async assignSignalToStartups(signalId: string, startupIds: string[]) {
// ... existing logic ...

// NEW: Emit WebSocket event
this.websocketService.emitSignalUpdate({
id: signal.id,
signalIdentifier: signal.signalIdentifier,
approvalStatus: 'assigned',
assignedStartups: startupIds,
});

// NEW: Emit to each assigned startup
for (const startupId of startupIds) {
this.websocketService.emitSignalAssignmentToClient({
startupId,
signalId: signal.id,
signalIdentifier: signal.signalIdentifier,
assignmentId: assignment.id,
});
}
}
  1. Add WebSocket emissions to Airtable webhook handler:
// In AirtableSyncService
async handleWebhookChange(change: AirtableChange) {
// ... existing sync logic ...

// NEW: Broadcast sync event
this.websocketService.emitAirtableSync({
table: change.tableId,
action: change.action,
recordId: change.recordId,
data: syncedData,
});
}

Frontend Changes Needed​

  1. Create custom hooks for common patterns:
// hooks/useRealtimeSignals.ts
export function useRealtimeSignals(filters?: SignalFilters) {
const { subscribe, unsubscribe } = useWebSocket();
const [signals, setSignals] = useState<Signal[]>([]);

useEffect(() => {
const handleSignalCreated = (data: SignalCreatedEvent) => {
setSignals((prev) => [data.signal, ...prev]);
};

const handleSignalUpdated = (data: SignalUpdatedEvent) => {
setSignals((prev) =>
prev.map((s) => (s.id === data.id ? { ...s, ...data.updates } : s)),
);
};

subscribe("signal:created", handleSignalCreated);
subscribe("signal:updated", handleSignalUpdated);

return () => {
unsubscribe("signal:created", handleSignalCreated);
unsubscribe("signal:updated", handleSignalUpdated);
};
}, [subscribe, unsubscribe]);

return signals;
}
  1. Update page components to use real-time hooks:
// app/signals/page.tsx
export default function SignalsPage() {
const [signals, setSignals] = useState<Signal[]>([]);
const { subscribe, unsubscribe, isConnected } = useWebSocket();

// Fetch initial data
useEffect(() => {
fetchSignals().then(setSignals);
}, []);

// Subscribe to real-time updates
useEffect(() => {
if (!isConnected) return;

const handleNewSignal = (data: SignalCreatedEvent) => {
setSignals(prev => [data.signal, ...prev]);
toast.success(`New signal: ${data.signal.signalIdentifier}`);
};

subscribe('signal:created', handleNewSignal);
return () => unsubscribe('signal:created', handleNewSignal);
}, [isConnected, subscribe, unsubscribe]);

return (
<div>
{!isConnected && <LiveUpdateIndicator offline />}
{/* ... rest of UI ... */}
</div>
);
}

Testing Strategy​

Unit Tests​

  • Test WebSocket event emissions in services
  • Test event subscriptions in custom hooks
  • Test reconnection logic

Integration Tests​

  • Test webhook → WebSocket flow end-to-end
  • Test multi-client scenarios (admin + multiple startups)
  • Test event delivery across different namespaces

Manual Testing Checklist​

  • Airtable webhook triggers real-time update in admin panel
  • Signal assignment shows immediately in startup inbox
  • Credit deduction reflects instantly across all sessions
  • Connection status change visible to both admin and startup
  • Notifications appear without page refresh
  • Offline/reconnect handling works smoothly
  • Multiple admins see same events simultaneously
  • No duplicate events or memory leaks

Success Criteria​

Must Have​

✅ All admin dashboard metrics update in real-time ✅ Signal assignments appear instantly in startup inbox ✅ Airtable changes broadcast to admin panel ✅ Connection status changes visible immediately ✅ No page refreshes needed for data updates

Should Have​

✅ Notification badge system with live counts ✅ Optimistic UI for common actions ✅ Offline state indicators ✅ Event filtering options for admins

Nice to Have​

✅ Sound notifications for critical events ✅ Event history/replay capability ✅ Admin event preferences ✅ Performance metrics dashboard for WebSocket health


Risk Mitigation​

RiskMitigation Strategy
Too many events causing performance issuesImplement event debouncing and selective subscriptions
WebSocket disconnectionsRobust reconnection logic with exponential backoff
Stale data after reconnectionAuto-refresh data on reconnection
Memory leaks from event listenersProper cleanup in useEffect return functions
Testing webhook integrationsCreate mock webhook payload generator for testing

Next Steps​

  1. ✅ Review and approve this plan
  2. Begin Phase 1 implementation
  3. Set up testing environment for webhooks
  4. Implement Airtable → WebSocket bridge
  5. Create reusable real-time hooks
  6. Update admin dashboard with live metrics

References​