Degraded platform performance due to Google Cloud Incident

Incident Report for CipherHealth, Inc.

Resolved

Service performance has been restored, and all systems are operating normally. Our team will remain on high alert and continue to monitor the situation until Google Cloud confirms a full resolution on their end. No further customer impact is expected at this time.
Posted Jun 12, 2025 - 19:30 EDT

Monitoring

Google is reporting that most Google Cloud products have confirmed full service recovery. They are reporting that they expect the recovery to complete in less than an hour. We are seeing recovery to our services and will continue to monitor.
Posted Jun 12, 2025 - 18:02 EDT

Update

We are continuing to work on a fix for this issue.
Posted Jun 12, 2025 - 17:18 EDT

Update

Google Cloud has implemented mitigation for the issue in us-central1 and they are reporting signs of recovery.

They expect the recovery to complete in less than an hour.
Posted Jun 12, 2025 - 17:08 EDT

Update

Google Cloud has identified the root cause and applied appropriate mitigations, however our region in GCP is still impacted, us-central1. Google has not provided an ETA.
Posted Jun 12, 2025 - 16:30 EDT

Identified

The issue has been identified and fix is being implemented by Google Cloud.
Posted Jun 12, 2025 - 16:00 EDT

Investigating

We are currently investigating an issue causing degraded performance, including slow response times and intermittent errors, across multiple services within the Evolve platform.
Our investigation indicates this is related to a broader, ongoing incident with Google Cloud Platform (GCP). We are working with GCP support on a solution and will provide an update as soon as one is available.
Posted Jun 12, 2025 - 15:07 EDT
This incident affected: Evolve (Evolve Application, Insights Dashboards, CipherRounds App, CipherKiosk App, CipherOutreach, Single Sign On (SSO) and Login, Data Feeds (HL7), Reports, Manual Exports, SFTP Imports, Email, Scheduled Exports, Appointment Rescheduling, Medication Affordability Programs, Self-Service Rounding).