Skip to main content

CivStart Deployment Guide

This guide explains the CivStart deployment architecture and how deployments are managed.

Table of Contents​

Architecture Overview​

CivStart uses a separation of concerns approach for CI/CD:

🔍 CI Pipeline (GitHub Actions)​

  • Purpose: Quality gates before merging code
  • Runs on: Every push and pull request
  • Includes:
    • Code quality checks (linting, type checking, formatting)
    • Test suite with coverage
    • Build verification
    • Database migration validation
    • Docker build verification
    • Security scans

🚀 Deployment (External Systems)​

  • Backend: CloudFormation/AWS handles deployment and infrastructure
  • Frontend & Admin: Vercel handles automatic deployment
  • Separation: Deployments happen outside GitHub Actions

📊 Deployment Tracking​

CI Pipeline (GitHub Actions)​

Workflow: .github/workflows/ci.yml​

The CI pipeline runs automatically on:

  • Pushes to main, develop, dev, staging branches
  • Pull requests to these branches

Jobs​

  1. Code Quality - Linting, type checking, formatting
  2. Test Suite - Unit tests with PostgreSQL and Redis
  3. Build Verification - Backend build validation
  4. Migration Validation - Prisma migration checks
  5. Docker Build - Backend Docker image verification
  6. Security Scan - Dependency audits and Trivy scans

Purpose​

The CI pipeline acts as a quality gate - it ensures code meets standards before merging, but does not deploy the application.

Backend Deployment (CloudFormation/AWS)​

Deployment Method​

Backend services are deployed via AWS CloudFormation, which:

  • Provisions infrastructure (ECS, RDS, ElastiCache, etc.)
  • Builds and deploys Docker containers
  • Manages database migrations
  • Handles health checks and rollbacks

Triggering Deployments​

Deployments are triggered outside GitHub Actions, typically:

  • Via AWS CodePipeline (automatic on code changes)
  • Via CloudFormation CLI (manual)
  • Via AWS Console (manual)

Deployment Architecture​

Code Merge → CloudFormation Stack Update
    ├─ Build Docker image (backend)
    ├─ Push to ECR
    ├─ Update ECS task definition
    ├─ Run database migrations
    └─ Deploy to ECS cluster

Manual Deployment​

# Deploy backend to staging
aws cloudformation deploy \
  --template-file infrastructure/backend.yaml \
  --stack-name civstart-backend-staging \
  --parameter-overrides Environment=staging \
  --capabilities CAPABILITY_IAM

# Deploy backend to production
aws cloudformation deploy \
  --template-file infrastructure/backend.yaml \
  --stack-name civstart-backend-production \
  --parameter-overrides Environment=production \
  --capabilities CAPABILITY_IAM

Frontend & Admin Deployment (Vercel)​

Deployment Method​

Frontend and Admin applications are deployed via Vercel, which:

  • Automatically builds on git push
  • Deploys to global CDN
  • Provides preview URLs for PRs
  • Handles automatic rollbacks

Automatic Deployment​

Production deployments (triggered automatically):

Preview deployments (triggered automatically):

Vercel Configuration​

Each app has a vercel.json configuration:

Frontend (apps/frontend/vercel.json):

{
  "github": {
    "enabled": true,
    "silent": false
  },
  "buildCommand": "cd ../.. && pnpm run build --filter=@civstart/frontend",
  "installCommand": "cd ../.. && pnpm install"
}

Admin (apps/admin/vercel.json):

{
  "github": {
    "enabled": true,
    "silent": false
  },
  "buildCommand": "cd ../.. && pnpm run build --filter=@civstart/admin",
  "installCommand": "cd ../.. && pnpm install"
}

Manual Deployment (if needed)​

# Install Vercel CLI
pnpm add -g vercel

# Deploy frontend to production
cd apps/frontend
vercel --prod

# Deploy admin to production
cd apps/admin
vercel --prod

Tracking Deployments on GitHub​

To see deployment status directly on GitHub PRs and commits, see the comprehensive guide:

📖 DEPLOYMENT-TRACKING.md

This guide covers:

  • ✅ Automatic Vercel deployment tracking (already enabled)
  • 🔧 CloudFormation deployment reporting via GitHub API
  • 📊 Viewing deployment history and status

Manual Deployment (Legacy)​

Note: This section describes the legacy manual deployment process. Current deployments are handled by CloudFormation (backend) and Vercel (frontend/admin).

Initial Setup​

1. Configure GitHub Secrets (Legacy)​

Add the following secrets to your GitHub repository:

Go to: Settings → Secrets and variables → Actions → New repository secret

Secret NameDescriptionExample
PRODUCTION_SSH_KEYPrivate SSH key for deployment-----BEGIN OPENSSH PRIVATE KEY----
PRODUCTION_HOSTProduction server hostname or IPyour-server.com or 192.168.1.100
PRODUCTION_USERSSH username for deploymentdeploy or ubuntu
PRODUCTION_PATHAbsolute path to CivStart project on server/home/deploy/civstart

2. Generate SSH Key Pair​

On your local machine or CI server:

# Generate a new SSH key pair for deployment
ssh-keygen -t ed25519 -C "civstart-deploy" -f ~/.ssh/civstart_deploy_key

# Display the private key (add this to GitHub Secrets)
cat ~/.ssh/civstart_deploy_key

# Display the public key (add this to server's authorized_keys)
cat ~/.ssh/civstart_deploy_key.pub

3. Configure Production Server​

On your production server:

# Add the public key to authorized_keys
echo "YOUR_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys

# Set correct permissions
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh

# Make deployment script executable
cd /path/to/civstart
chmod +x scripts/deploy-production.sh

4. Configure PM2 for Backend (First Time Setup)​

Note: PM2 is only used for the backend service. Frontend and Admin are deployed via Vercel (see below).

If you haven't set up PM2 yet, create an ecosystem.config.js in your project root:

module.exports = {
  apps: [
    {
      name: "civstart-backend",
      cwd: "/path/to/civstart/apps/backend",
      script: "dist/main.js",
      instances: 1,
      exec_mode: "fork",
      env: {
        NODE_ENV: "production",
        PORT: 3800,
      },
    },
  ],
};

Start PM2 process:

pm2 start ecosystem.config.js
pm2 save
pm2 startup  # Follow instructions to enable PM2 on system boot

Deploy to Production​

Backend Deployment (Automated)​

Simply push to the main branch:

git push origin main

GitHub Actions will automatically:

  1. Run code quality checks (linting, type checking, formatting)
  2. Run test suite with coverage
  3. Build and verify backend application
  4. Build Docker image for backend
  5. SSH to production server
  6. Execute deployment script for backend
  7. Restart backend service (Docker or PM2)
  8. Run health checks
  9. Notify on success or failure

Note: Frontend and Admin are NOT deployed via GitHub Actions. They are automatically deployed by Vercel when changes are pushed to main.

Frontend & Admin Deployment (Vercel)​

Frontend and Admin are automatically deployed by Vercel when you push to main. No additional configuration needed.

Monitor Deployment​

View deployment progress:

  1. Go to GitHub → Actions
  2. Click on the latest workflow run
  3. Watch the "Deploy to Production" job

Manual Deployment​

If you need to deploy manually (for testing or emergency deployments):

1. SSH to Production Server​

ssh user@your-production-server.com
cd /path/to/civstart

2. Run Deployment Script​

Standard deployment:

./scripts/deploy-production.sh

Skip tests (faster, use with caution):

./scripts/deploy-production.sh --skip-tests

Skip backup creation:

./scripts/deploy-production.sh --no-backup

3. Verify Deployment​

Check PM2 backend status:

pm2 status
pm2 logs civstart-backend --lines 50

Test backend health endpoint:

curl http://localhost:3800/health

Frontend and Admin verification:

  • Frontend: Check your Vercel dashboard or visit your production URL
  • Admin: Check your Vercel dashboard or visit your admin production URL

Configuration​

Environment Variables​

Ensure the following environment variables are set on your production server:

Backend Server (Required):

  • DATABASE_URL - PostgreSQL connection string
  • CLERK_SECRET_KEY - Clerk authentication secret
  • PORT - Backend port (default: 3800)
  • NODE_ENV - Environment (production)

Backend Server (Optional):

  • BACKEND_URL - Override backend URL for health checks (default: http://localhost:3800)
  • REDIS_URL - Redis connection string (if using Redis)
  • PINECONE_API_KEY - For vector search features
  • GOOGLE_GEMINI_API_KEY - For AI features
  • AIRTABLE_API_KEY - For Airtable integration

Vercel (Frontend/Admin):

  • NEXT_PUBLIC_API_URL - Backend API URL (your production backend URL)
  • NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY - Clerk public key
  • CLERK_SECRET_KEY - Clerk secret key

Set these in your Vercel project settings under Environment Variables.

Deployment Script Options​

Customize the deployment script by editing scripts/deploy-production.sh:

# PM2 Process Name (backend only)
readonly PM2_BACKEND="civstart-backend"

# Health Check Configuration
readonly HEALTH_CHECK_RETRIES=5
readonly HEALTH_CHECK_INTERVAL=10

Note: Frontend and Admin are not managed by this script as they are deployed on Vercel.

Troubleshooting​

Deployment Fails During Build​

Symptoms: Build step fails with errors

Solutions:

  1. Check Node.js version compatibility (requires Node 18+)
  2. Ensure all dependencies are installed: pnpm install
  3. Check for TypeScript errors: pnpm typecheck
  4. Verify Prisma schema is valid: pnpm db:generate

PM2 Processes Won't Restart​

Symptoms: PM2 restart command fails

Solutions:

# Check PM2 process status
pm2 list

# View PM2 logs for errors
pm2 logs civstart-backend --err --lines 100

# Delete and recreate process
pm2 delete civstart-backend
pm2 start ecosystem.config.js --only civstart-backend
pm2 save

Health Checks Fail​

Symptoms: Deployment rolls back due to failed health checks

Solutions:

  1. Verify application is listening on correct port:

    netstat -tlnp | grep 3800  # Check backend port

  2. Check application logs:

    pm2 logs civstart-backend --lines 100

  3. Test health endpoint manually:

    curl -v http://localhost:3800/health

  4. Ensure database is accessible:

    psql "$DATABASE_URL" -c "SELECT 1"

Database Migration Errors​

Symptoms: Migration step fails

Solutions:

# Check database connection
pnpm --filter=@repo/database db:pull

# View migration status
pnpm --filter=@repo/database prisma migrate status

# Force push schema (development only!)
pnpm db:push --accept-data-loss

SSH Connection Issues​

Symptoms: GitHub Actions can't SSH to server

Solutions:

  1. Verify SSH key in GitHub Secrets matches server's authorized_keys

  2. Test SSH connection manually:

    ssh -i ~/.ssh/deploy_key user@server

  3. Check server SSH logs:

    sudo tail -f /var/log/auth.log  # Ubuntu/Debian
    sudo tail -f /var/log/secure    # CentOS/RHEL

  4. Verify firewall allows SSH (port 22)

Rollback Procedures​

Automatic Rollback​

The deployment script automatically creates backup tags and rolls back on failure.

Manual Rollback​

If you need to manually rollback to a previous version:

1. Find Backup Tag​

# List recent backup tags
git tag -l "backup-*" --sort=-creatordate | head -n 10

2. Checkout Backup​

# Rollback to specific backup
git checkout backup-20250121-143022

# Or rollback to previous commit
git checkout HEAD~1

3. Redeploy​

./scripts/deploy-production.sh

Emergency Rollback (Fast)​

For critical issues, use the last backup tag:

# Find last backup
LAST_BACKUP=$(git tag -l "backup-*" --sort=-creatordate | head -n 1)

# Rollback
git checkout $LAST_BACKUP
pnpm install --frozen-lockfile
pnpm db:generate
pnpm build
pm2 restart all

Best Practices​

Before Deploying​

  • Run tests locally: pnpm test
  • Check TypeScript types: pnpm typecheck
  • Run linter: pnpm lint
  • Test build locally: pnpm build
  • Review database migrations
  • Update changelog/version if needed

After Deploying​

  • Monitor PM2 logs: pm2 logs --lines 100
  • Check application health endpoints
  • Verify critical user flows
  • Monitor error tracking (if configured)
  • Check database performance

Production Checklist​

  • Environment variables are set
  • Database backups are automated
  • SSL/TLS certificates are valid
  • Firewall rules are configured
  • PM2 is configured for auto-restart
  • Monitoring/alerting is set up
  • Log rotation is configured

Additional Resources​

Support​

For deployment issues:

  1. Check PM2 logs: pm2 logs civstart-backend
  2. Review GitHub Actions logs in the Actions tab
  3. Check server system logs: journalctl -u pm2-*
  4. Verify environment variables are set correctly

For critical production issues, follow the Emergency Rollback procedure first, then investigate.