API experiencing issues

Incident Report for IEX Cloud

Postmortem

Our system is configured to auto scale during high load or ahead of periods of heavy traffic. The system responsible for auto scaling was unable to communicate with API processes for a period of time resulting in API servers being removed from network routing and a restart of API services. SRE manually rerouted traffic to backup servers. The network issue has been fixed and more defensive measures are being put in place.

Posted Feb 17, 2021 - 16:24 EST

Resolved

This incident has been resolved.
Posted Feb 17, 2021 - 15:54 EST

Monitoring

API traffic has been rerouted and service should be stabilizing.
Posted Feb 17, 2021 - 15:52 EST

Identified

Connection was lost to internal server resources. Additional servers have been brought online and service should be partially restored.
Posted Feb 17, 2021 - 15:48 EST

Investigating

API services are experiencing an issue at this time and are not accessible. Our team is investigating.
Posted Feb 17, 2021 - 15:36 EST
This incident affected: Core Data.