WebSocket & Real-Time Enhancements Plan

Executive Summary

This document outlines the plan to enhance the CivStart platform's real-time capabilities by reviewing and improving the existing WebSocket implementation and ensuring all critical pages receive live updates.

Created: 2025-11-05 Branch: feature/websocket-realtime-enhancements Status: Planning Phase

---

Current Architecture Review

✅ What's Already Implemented

Backend Infrastructure

1. WebSocket Gateways (apps/backend/src/core/)

   • AdminWebSocketGateway - Handles admin panel connections (/admin namespace)
   • ClientWebSocketGateway - Handles startup/client connections (/client namespace)
   • Both use Socket.IO with proper CORS configuration

2. WebSocket Service (websocket.service.ts)

   • Centralized service for emitting events
   • Supports both admin and client-side broadcasts
   • Event types covered:
     • Connection events (created, updated)
     • Signal events (created, updated, assigned)
     • Match events (generated)
     • Credit events (updated)
     • Assignment events (created, responseChanged, viewed)
     • Metrics updates
     • Notifications

3. Notification Service (notification.service.ts)

   • Multi-channel notification system (email, SMS, push, in-app)
     • Template-based content generation
     • Scheduled delivery with retry logic
     • Comprehensive statistics tracking
     • Currently focuses on scheduled notifications, not real-time broadcasts

4. Airtable Webhook Integration (airtable-sync.service.ts)

   • Handles webhook payloads from Airtable
   • Syncs organizations, contacts, signals, startups
   • Maps Airtable changes to Prisma models
   • Gap: Not integrated with WebSocket broadcasts

Frontend Infrastructure

1. WebSocket Client (apps/frontend/lib/websocket.ts & apps/admin/lib/websocket.ts)

   • Socket.IO client wrapper
   • Type-safe event handling
   • Auto-reconnection logic
   • Startup-specific room joining

2. WebSocket Provider (WebSocketProvider.tsx)

   • React context for WebSocket state
   • Auto-connects when user is authenticated
   • Prevents duplicate connections with global state management
   • Subscription management for event listeners
   • Currently live on frontend: Credit updates

3. Admin WebSocket Provider (apps/admin/lib/websocket-provider.tsx)

   • Similar to client provider but for admin panel
   • Auto-connects authenticated admins
   • Gap: Limited event subscriptions currently active

4. Notification Listeners

   • NotificationListener.tsx components in both admin and frontend
   • Display toast notifications for real-time events
   • Gap: Only partially implemented across all pages

---

Identified Gaps & Enhancement Areas

🔴 Critical Gaps

1. Webhook → WebSocket Bridge Missing

   • Airtable webhook changes don't trigger WebSocket broadcasts
   • Manual data sync required to see changes from Airtable
   • Impact: Admins don't see real-time updates from Airtable changes

2. Incomplete Event Coverage

   • Many backend services emit events, but not all are consumed on frontend
   • Signal status changes not always broadcast in real-time
   • Startup assignments to signals not reflected immediately

3. Missing Real-Time Pages

   • Several critical admin pages don't subscribe to WebSocket events
   • Dashboard metrics not updating in real-time
   • Signal pool page requires manual refresh

4. No Notification Queue Processing

   • NotificationService has scheduled processing, but no real-time trigger
   • User inbox updates aren't pushed via WebSocket
   • Impact: Users must refresh to see new notifications

🟡 Enhancement Opportunities

1. Enhanced Notification System

   • Integrate NotificationService with WebSocket for instant delivery
   • Add in-app notification queue with real-time updates
   • Implement notification badges with live counts

2. Better Error Handling

   • No fallback strategy when WebSocket connection fails
   • Limited retry logic on client-side reconnections
   • Missing offline state indicators

3. Optimistic UI Updates

   • Current implementation waits for server confirmation
   • Could implement optimistic updates with rollback on error

4. Event Filtering & Targeting

   • All admins receive all events (potential performance issue)
   • Could implement event filtering based on admin permissions/interests
   • Startup-specific filtering already implemented for clients

---

Pages Requiring Real-Time Updates

Admin Panel (apps/admin/)

Page	Current State	Events Needed	Priority
/dashboard	❌ Static	metrics:updated, connection:created, signal:created	🔴 High
/signals	⚠️ Partial	signal:created, signal:updated, match:generated	🔴 High
/signal-pool	❌ Static	signal:created, signal:updated, signal:statusChanged	🔴 High
/startups	❌ Static	startup:created, startup:updated, credits:changed	🟡 Medium
/startups/[id]	✅ Good	credits:updated, assignment:created, connection:updated	🟢 Low
/connections	⚠️ Partial	connection:created, connection:updated, connection:statusChanged	🔴 High
/contact-requests	❌ Static	contactRequest:created, contactRequest:updated	🟡 Medium
/analytics	❌ Static	metrics:updated, analytics:updated	🟢 Low
/organizations	❌ Static	organization:synced (from Airtable)	🟢 Low

Client/Startup Portal (apps/frontend/)

Page	Current State	Events Needed	Priority
/govintel/inbox	⚠️ Partial	assignment:created, signal:assigned, notification:new	🔴 High
/govintel/pool	❌ Static	signal:published, signal:removed	🔴 High
/govintel/connections	❌ Static	connection:statusChanged, connection:updated	🟡 Medium
Dashboard (overview)	⚠️ Partial	credits:updated, assignment:new, metrics:updated	🟡 Medium

---

Enhanced WebSocket Event Architecture

Proposed Event Categories

1. Data Sync Events (from Airtable webhooks)

// When Airtable data changes
"airtable:organization:synced";
"airtable:contact:synced";
"airtable:signal:synced";
"airtable:interaction:synced";

2. Signal Lifecycle Events

"signal:created"; // New signal generated
"signal:updated"; // Signal data changed
"signal:published"; // Signal made public in pool
"signal:removed"; // Signal removed from pool
"signal:assigned"; // Signal assigned to startup(s)
"signal:statusChanged"; // Approval status changed
"signal:matchGenerated"; // AI matches created for signal

3. Assignment & Response Events

"assignment:created"; // Signal assigned to specific startup
"assignment:viewed"; // Startup viewed the assignment
"assignment:interested"; // Startup marked as interested
"assignment:declined"; // Startup declined
"assignment:responseChanged"; // Response status updated

4. Connection Events

"connection:created"; // New connection request
"connection:updated"; // Connection details changed
"connection:statusChanged"; // Status transition (pending → approved, etc.)

5. Credit Events (already implemented)

"credits:updated"; // Credit balance changed
"credits:deducted"; // Credits used
"credits:allocated"; // New credits added
"credits:tierChanged"; // Tier upgraded/downgraded

6. Notification Events (new)

"notification:new"; // New notification created
"notification:read"; // Notification marked as read
"notification:deleted"; // Notification dismissed
"notification:countChanged"; // Unread count updated

7. Metrics & Analytics Events

"metrics:updated"; // Dashboard metrics refreshed
"analytics:updated"; // Analytics data updated

---

Implementation Phases

Phase 1: Foundation Enhancement (Week 1)

Goal: Establish robust foundation and fix critical gaps

1. ✅ Create feature branch

2. ✅ Document current architecture

3. Webhook → WebSocket Bridge

   • Modify AirtableSyncService to emit WebSocket events after successful syncs
   • Add new event types for Airtable sync operations
   • Test with actual Airtable webhook triggers

4. Enhanced Notification Service Integration

   • Add WebSocket emission to NotificationService.processNotification()
   • Create real-time notification event for in-app delivery
   • Update UserInbox creation to trigger WebSocket events

5. Client-Side Event Listeners

   • Create reusable hooks for common event subscriptions
   • useSignalUpdates() - Subscribe to signal events
   • useConnectionUpdates() - Subscribe to connection events
   • useNotifications() - Subscribe to notification events
   • useMetrics() - Subscribe to metrics updates

Phase 2: Admin Panel Real-Time Updates (Week 1-2)

Goal: Make all admin pages live-updating

1. Dashboard Page (/dashboard)

   • Subscribe to metrics:updated for live stat updates
   • Subscribe to signal:created to show recent signals
   • Subscribe to connection:created for activity feed
   • Implement auto-refresh with smooth transitions

2. Signals Page (/signals)

   • Subscribe to signal:created, signal:updated
   • Add real-time filtering when signals change status
   • Show live match generation progress
   • Optimistic UI for signal approvals

3. Signal Pool Page (/signal-pool)

   • Subscribe to signal:published, signal:removed
   • Real-time pool updates as signals are added/removed
   • Live startup assignment count updates

4. Connections Page (/connections)

   • Subscribe to connection:created, connection:updated
   • Live status badge updates
   • Real-time filtering when status changes

5. Startups Page (/startups and /startups/[id])

   • Subscribe to credits:updated for live credit displays
   • Subscribe to assignment:created for new assignments
   • Real-time tier change indicators

Phase 3: Client Portal Real-Time Updates (Week 2)

Goal: Make startup portal fully real-time

1. Inbox Page (/govintel/inbox)

   • Subscribe to assignment:created for instant notifications
   • Subscribe to notification:new for in-app alerts
   • Live unread count badge
   • Sound/visual notification for new assignments

2. Signal Pool Page (/govintel/pool)

   • Subscribe to signal:published for new public signals
   • Subscribe to signal:removed to remove from list
   • Real-time cost display updates

3. Connections Page (/govintel/connections)

   • Subscribe to connection:statusChanged
   • Live connection request status updates
   • Optimistic UI for connection requests

Phase 4: Advanced Features (Week 2-3)

Goal: Polish and optimize

1. Notification Badge System

   • Persistent unread notification count
   • Real-time badge updates across all pages
   • Notification center dropdown with live updates

2. Optimistic UI Updates

   • Implement optimistic updates for admin actions
   • Add rollback logic for failed operations
   • Visual feedback for pending operations

3. Offline Support & Reconnection

   • Better offline state indicators
   • Queue events when offline, replay on reconnect
   • Automatic data refresh after reconnection

4. Performance Optimizations

   • Event debouncing for high-frequency updates
   • Pagination-aware event handling
   • Selective event subscription based on current page

5. Admin Event Filtering

   • Allow admins to customize which events they want to see
   • Implement event preferences in admin settings
   • Reduce unnecessary re-renders

---

Technical Implementation Details

Backend Changes Needed

1. Modify services to emit WebSocket events:

// Example: Signal Assignment Service
async assignSignalToStartups(signalId: string, startupIds: string[]) {
  // ... existing logic ...

  // NEW: Emit WebSocket event
  this.websocketService.emitSignalUpdate({
    id: signal.id,
    signalIdentifier: signal.signalIdentifier,
    approvalStatus: 'assigned',
    assignedStartups: startupIds,
  });

  // NEW: Emit to each assigned startup
  for (const startupId of startupIds) {
    this.websocketService.emitSignalAssignmentToClient({
      startupId,
      signalId: signal.id,
      signalIdentifier: signal.signalIdentifier,
      assignmentId: assignment.id,
    });
  }
}

2. Add WebSocket emissions to Airtable webhook handler:

// In AirtableSyncService
async handleWebhookChange(change: AirtableChange) {
  // ... existing sync logic ...

  // NEW: Broadcast sync event
  this.websocketService.emitAirtableSync({
    table: change.tableId,
    action: change.action,
    recordId: change.recordId,
    data: syncedData,
  });
}

Frontend Changes Needed

1. Create custom hooks for common patterns:

// hooks/useRealtimeSignals.ts
export function useRealtimeSignals(filters?: SignalFilters) {
  const { subscribe, unsubscribe } = useWebSocket();
  const [signals, setSignals] = useState<Signal[]>([]);

  useEffect(() => {
    const handleSignalCreated = (data: SignalCreatedEvent) => {
      setSignals((prev) => [data.signal, ...prev]);
    };

    const handleSignalUpdated = (data: SignalUpdatedEvent) => {
      setSignals((prev) =>
        prev.map((s) => (s.id === data.id ? { ...s, ...data.updates } : s)),
      );
    };

    subscribe("signal:created", handleSignalCreated);
    subscribe("signal:updated", handleSignalUpdated);

    return () => {
      unsubscribe("signal:created", handleSignalCreated);
      unsubscribe("signal:updated", handleSignalUpdated);
    };
  }, [subscribe, unsubscribe]);

  return signals;
}

2. Update page components to use real-time hooks:

// app/signals/page.tsx
export default function SignalsPage() {
  const [signals, setSignals] = useState<Signal[]>([]);
  const { subscribe, unsubscribe, isConnected } = useWebSocket();

  // Fetch initial data
  useEffect(() => {
    fetchSignals().then(setSignals);
  }, []);

  // Subscribe to real-time updates
  useEffect(() => {
    if (!isConnected) return;

    const handleNewSignal = (data: SignalCreatedEvent) => {
      setSignals(prev => [data.signal, ...prev]);
      toast.success(`New signal: ${data.signal.signalIdentifier}`);
    };

    subscribe('signal:created', handleNewSignal);
    return () => unsubscribe('signal:created', handleNewSignal);
  }, [isConnected, subscribe, unsubscribe]);

  return (
    <div>
      {!isConnected && <LiveUpdateIndicator offline />}
      {/* ... rest of UI ... */}
    </div>
  );
}

---

Testing Strategy

Unit Tests

• Test WebSocket event emissions in services
• Test event subscriptions in custom hooks
• Test reconnection logic

Integration Tests

• Test webhook → WebSocket flow end-to-end
• Test multi-client scenarios (admin + multiple startups)
• Test event delivery across different namespaces

Manual Testing Checklist

• Airtable webhook triggers real-time update in admin panel
• Signal assignment shows immediately in startup inbox
• Credit deduction reflects instantly across all sessions
• Connection status change visible to both admin and startup
• Notifications appear without page refresh
• Offline/reconnect handling works smoothly
• Multiple admins see same events simultaneously
• No duplicate events or memory leaks

---

Success Criteria

Must Have

✅ All admin dashboard metrics update in real-time ✅ Signal assignments appear instantly in startup inbox ✅ Airtable changes broadcast to admin panel ✅ Connection status changes visible immediately ✅ No page refreshes needed for data updates

Should Have

✅ Notification badge system with live counts ✅ Optimistic UI for common actions ✅ Offline state indicators ✅ Event filtering options for admins

Nice to Have

✅ Sound notifications for critical events ✅ Event history/replay capability ✅ Admin event preferences ✅ Performance metrics dashboard for WebSocket health

---

Risk Mitigation

Risk	Mitigation Strategy
Too many events causing performance issues	Implement event debouncing and selective subscriptions
WebSocket disconnections	Robust reconnection logic with exponential backoff
Stale data after reconnection	Auto-refresh data on reconnection
Memory leaks from event listeners	Proper cleanup in useEffect return functions
Testing webhook integrations	Create mock webhook payload generator for testing

---

Next Steps

1. ✅ Review and approve this plan
2. Begin Phase 1 implementation
3. Set up testing environment for webhooks
4. Implement Airtable → WebSocket bridge
5. Create reusable real-time hooks
6. Update admin dashboard with live metrics

---

References

• Socket.IO Documentation
• React WebSocket Best Practices
• Current Implementation:
  • apps/backend/src/core/websocket.gateway.ts
  • apps/backend/src/core/websocket.service.ts
  • apps/frontend/lib/WebSocketProvider.tsx
  • apps/admin/lib/websocket-provider.tsx