Encrypted Google Backup Rollout Change Record

Change metadataCopy Anchor Link

• Change id: CR-2026-08-13-encrypted-google-backup-rollout
• Prepared date: 2026-08-13
• Environment: VPS / production
• Status: completed 2026-08-14; encrypted workflows and all three timers active
• Approver and executor: owner-approved staged execution; Codex-assisted terminal execution

ScopeCopy Anchor Link

• Install rclone if absent.
• Prepare root-only /etc/contabo-backups/{backup.env,rclone.conf}.
• Configure contabo_drive/contabo_crypt and gorsistudio_drive/gorsistudio_crypt directly on the VPS.
• Validate encrypted WordPress and database canaries plus scratch restores.
• Validate weekly PostgreSQL globals/per-database dumps and an online physical snapshot through an isolated PostgreSQL 18 restore.
• Set DB_OFFSITE_ENABLED=true only after validation.
• Set POSTGRES_PHYSICAL_OFFSITE_ENABLED=true only after physical restore validation.
• Reconcile the installed WordPress/weekly units and add the monthly physical timer.

No Docker stack redeploy, database migration, or production service restart was included. Only verified rolling-retention targets and explicitly named scratch resources were removed.

Risk assessmentCopy Anchor Link

• Risk level: medium.
• Main risks: incorrect OAuth scope/token, lost crypt key, wrong tenant destination, empty-volume archive, incomplete database coverage, bandwidth/disk pressure, or retention before verification.
• Controls: root-only secrets, Swarm discovery, explicit destination registries, volume validation, logical/native manifests, cryptcheck, shared six-hour-wait lock, bandwidth caps, scratch-name guards, and delayed pruning.

PreconditionsCopy Anchor Link

• Host bootstrap approved by owner on 2026-08-14.
• Current timer units and local backup inventory captured.
• Both OAuth Desktop applications configured.
• Both destination accounts authorized and verified as distinct intended accounts.
• Separate crypt password/salt pairs entered directly and stored outside Git/chat.
• Repository tests and dry-run unit renders pass at the rollout revision.

Execution logCopy Anchor Link

• 2026-08-14: installed Ubuntu rclone package version 1.60.1+dfsg-3ubuntu0.24.04.6.
• 2026-08-14: created /etc/contabo-backups as root:root 0700 and created backup.env plus an empty rclone.conf as root:root 0600.
• 2026-08-14: populated rclone.conf with the non-secret contabo_drive, contabo_crypt, gorsistudio_drive, and gorsistudio_crypt structure. All client, token, crypt password, and crypt salt values were unset at this checkpoint and were entered directly afterward.
• 2026-08-14: both OAuth remotes were authorized, identities and quotas were verified, and all four crypt password fields were converted from plaintext entry to the obscured format required by rclone. No secret values were printed or recorded.
• 2026-08-14: both crypt remotes reached the expected pre-canary state: credentials and encryption settings are valid, while each Contabo root is not yet created.
• 2026-08-14 preflight: 96 GiB host disk space is available; PostgreSQL, MySQL, and MongoDB are 1/1; MSSQL is intentionally/currently scaled to 0/0 and requires a separate decision before four-engine validation.
• 2026-08-14 later rollout: the weekly database and WordPress units were reinstalled from canonical operations/systemd/ paths and remain enabled/active. The new monthly PostgreSQL physical unit is not installed. The WordPress timer will be disabled during final activation, while weekly local database coverage remains active.
• 2026-08-14: uploaded one 4 KiB canary through each crypt remote. Both passed cryptcheck, decrypted download SHA-256 comparison, and underlying .bin filename verification. Evidence: tmp/backups/evidence/rclone-canary-20260814T010558Z.txt.
• 2026-08-14: store.gorsistudio.com dry-run and encrypted backup passed. The downloaded archive matched its manifest, restored 4,411 files into an isolated Docker volume, and restored 12 tables into isolated MySQL. All scratch resources were removed. Evidence: tmp/backups/evidence/wp-restore-validation-20260814T011208Z.txt.
• The first isolated MySQL readiness attempt produced a false-positive readiness result; it did not touch the production database, its scratch volume was removed, and the corrected retry passed.
• 2026-08-14: the existing tier1-20260426T010139Z set uploaded to the central Drive and all eight artifacts passed cryptcheck. Its PostgreSQL artifact downloaded with a matching manifest checksum, restored into a uniquely named scratch database, passed a validation query, and was dropped. Evidence: tmp/backups/evidence/postgres-offsite-restore-20260814T011851Z.txt.
• Two isolated PostgreSQL-container attempts failed and cleaned up: the first used an incompatible PostgreSQL 18 data path and the second exited 137. The successful low-resource validation used the healthy existing PostgreSQL task without restarting it.
• No production service/container restart, environment activation, or timer change was performed. Only explicitly approved OAuth/crypt setup, uploads, and isolated or uniquely named scratch restores were performed.
• Package installation reported a pre-existing pending kernel update; no reboot was performed.
• 2026-08-14 repository hardening: backup scope was reduced to four WordPress sites; PostgreSQL logical coverage was expanded to globals plus every connectable non-template database; online monthly physical backup, guarded restore, first-Sunday timer artifacts, failure evidence, and sliding-retention tests were added. These are repository/DEV changes only at that checkpoint.
• 2026-08-14 final activation: captured rollback files under the root-only /etc/contabo-backups/rollback-20260814T211000Z, disabled only the WordPress timer, kept weekly local database scheduling active, applied the reviewed host settings, and validated canonical WordPress/weekly units with both offsite flags still false.
• The four-site production dry-run passed authenticated MySQL database and non-empty volume checks for dbskc, both Gorsi sites, and wcblahore.
• Complete logical set tier1-20260814T211722Z passed local checksums and encrypted delivery: nine PostgreSQL databases plus globals/metadata, four MySQL databases, and one MongoDB artifact. All 17 remote files matched cryptcheck; local retention is three and remote inventory is two against limits of three/twelve. The verified oldest local set tier1-20260426T010139Z was pruned as designed; its encrypted remote copy remains intact. Because that generated set had historically been tracked, its removal is also represented as repository deletions.
• The first physical attempt produced a valid large native tar but was stopped before upload by a pipefail/early-exit listing check. Its staging remains local and sanitized failure evidence was captured. The backup and restore validators now use a streaming full-list parser, and automatic local retention counts only explicitly successful physical snapshots.
• Physical set postgres-physical-20260814T212317Z passed local manifest/archive checks, uploaded with zero cryptcheck differences, downloaded through contabo_crypt, and matched both manifest checksums.
• The physical snapshot passed native pg_verifybackup, readiness, nine-database count, and validation query in an isolated no-network/no-port container using the recorded image digest and a new scratch volume. Production PostgreSQL remained 1/1; the named scratch resources and downloaded copy were then removed.
• The real four-site WordPress workflow passed encrypted upload/verification for every site. Remote inventories are within newest-four retention and local WordPress staging is empty.
• Set DB_OFFSITE_ENABLED=true, POSTGRES_PHYSICAL_OFFSITE_ENABLED=true, and DB_ENGINES=postgres,mysql,mongodb; installed/enabled the first-Sunday physical timer and re-enabled WordPress. All three timers are enabled/active, both remotes connect, all unit results are successful, and MSSQL remains 0/0.

Execution and validationCopy Anchor Link

Follow docs/operations/execution/phase5-offsite-backup-rollout-execution-bundle.mdx. Record only sanitized command outcomes and evidence paths here; never record OAuth, database, or crypt secrets.

• Canary upload/cryptcheck/download result: passed for both Drives
• WordPress scratch restore result: passed
• Database scratch restore result: passed with PostgreSQL; MSSQL remained 0/0
• Full manual run result: passed for four WordPress sites and all active database engines
• First physical backup/isolated restore result: passed; scratch cleanup complete
• Final three-timer status result: passed; all enabled/active
• Diagnostic result: passed; both destinations connected and evidence sanitized
• Evidence paths:
  • tmp/backups/evidence/rclone-canary-20260814T010558Z.txt
  • tmp/backups/evidence/wp-gdrive-backup-20260814T010922Z.txt
  • tmp/backups/evidence/wp-restore-validation-20260814T011208Z.txt
  • tmp/backups/evidence/tier1-backup-delivery-20260814T011246Z.txt
  • tmp/backups/evidence/postgres-offsite-restore-20260814T011851Z.txt
  • tmp/backups/evidence/tier1-backup-delivery-20260814T211722Z.txt
  • tmp/backups/evidence/postgres-physical-delivery-20260814T212308Z.txt
  • tmp/backups/evidence/postgres-physical-delivery-20260814T212317Z.txt
  • tmp/backups/evidence/postgres-physical-restore-20260814T212529Z.txt
  • tmp/backups/evidence/wp-gdrive-backup-20260814T212707Z.txt
  • tmp/backups/evidence/backup-activation-final-20260814T213322Z.txt

RollbackCopy Anchor Link

1. Disable the WordPress and physical timers if activated by this change.
2. Set both offsite flags false in the root-only host env.
3. Reinstall the last validated canonical weekly unit if required and confirm local weekly backups remain active.
4. Reload systemd and verify no production service was restarted.
5. Preserve all verified Google Drive data.

OutcomeCopy Anchor Link

• Result: successful
• Owner sign-off: approved and executed 2026-08-14