Business Continuity & Disaster Recovery Plan

1. Recovery Objectives

2. System Architecture Overview

Each client runs Cadence on its own dedicated hardware — an EMStool-provided appliance or the client's own server, in the client's own facility — fully isolated from every other client. There is no shared multi-tenant database.

• Application Server: per-client hardware — Unraid or equivalent, SWAG reverse proxy, PHP/Docker
• Database: one dedicated MariaDB container per client deployment
• Tunnel: Cloudflare Zero Trust tunnel — routes external traffic to that client's hardware, wherever it is physically plugged in
• Backups: encrypted .sql.gpg; destination is the client's own choice (Backblaze B2, local NAS/rsync, etc.)
• Code: GitHub repo mrjacobarussell/mih-scheduler, deployed via deploy.sh

EMStool also operates its own non-client host (Bender, 192.168.0.13) for software development and a public demo/trial instance (scheduler.emstool.com, seed/dummy data only). The scenarios below apply identically to any Cadence deployment, client or EMStool's own.

3. Backup Procedures

Database Backups

• Daily automated mysqldump of Cadence DB
• Encrypted with GPG before upload (key stored offline)
• Uploaded to Backblaze B2 bucket (cadence-backups)
• Retention: 30 days of daily backups
• Backup verification: monthly restore test to confirm backup integrity

Code Backups

• All code maintained in GitHub with full git history
• Production state is always deployable from latest main branch

4. Disaster Scenarios and Response Procedures

Scenario A: Database Corruption or Loss

1. Stop SWAG container to prevent further writes
2. Download latest encrypted backup from Backblaze B2
3. Decrypt: gpg --decrypt backup.sql.gpg | mysql -u root -p cadence_db
4. Start MariaDB container and verify data integrity
5. Run any pending migrations: ./deploy.sh migrate
6. Restart SWAG container; verify application operational
7. Notify affected customers of data recovery and any data loss window

Scenario B: Server Hardware Failure

Applies to Bender (EMStool's own dev/demo host) or any single client's own hardware — each deployment is independent, so a hardware failure on one never affects another.

1. Provision a replacement Unraid server or Docker host
2. Install SWAG and the MariaDB container with same configuration
3. Restore the database from the configured backup (Backblaze B2, local NAS, or whatever destination that deployment uses — see Scenario A)
4. Clone code from GitHub and run deploy.sh
5. Update Cloudflare tunnel connector to point to new server
6. Verify DNS and SSL function correctly

Scenario C: Cloudflare Tunnel Failure

1. SSH to Bender (192.168.0.13) and check cloudflared container status
2. Restart tunnel container: docker restart Unraid-Cloudflared-Cadence
3. If tunnel token expired, regenerate at Cloudflare dashboard and update container env
4. If tunnel service is globally degraded, monitor Cloudflare status page and wait

Scenario D: Security Incident (Breach/Ransomware)

1. Immediately disconnect affected systems from network
2. Follow Incident Response Runbook (IRP-2026-01)
3. Restore from last known-clean backup
4. Rebuild compromised containers from scratch
5. Rotate all credentials before bringing systems back online

5. Communication During Outage

6. Recovery Test Schedule

• Backup restore test: Monthly — restore a backup to a test DB and verify table counts
• Full DR drill: Annually — simulate a complete server failure and perform full recovery
• Tunnel failover test: Quarterly — restart tunnel container and confirm recovery within 5 minutes

7. Restore Test Log