Unfortunately, we're experiencing a series of incidents today.
The first one had started 3 hours ago. The Standuply service lost connection between our Amazon instances and cloud database Mongo Atlas and wasn't able to reconnect all that time.
When we were figuring out and trying to fix the issue with the Mongo support team, Slack events API was automatically disabled for the app, because the service can't get API events from Slack. That's a usual procedure.
More than 1 hour ago Mongo resolved the connection error but we're not able to turn on back the API events for the app for some unknown reason.
We're in touch right now with the Slack team and they are looking for the problem to resolve it.
Unfortunately, we can only wait because can't control this module at all.
The whole our team is on and we'll find the solution and fix everything for sure, but it can take a few more hours of downtime.
Please, take our apologies for such inconvenience.
We're doing our best.
Thanks for your patience and understanding,
We've just got the update from Slack team that they fixed the problem with events API and we've been able to restart it for our app.
Standuply is stable right now and all components work properly.
Sorry again for the inconvenience, despite on the root of the incidents wasn't on our end, it's our fail for sure .
We'll do the retrospective tomorrow to understand can we be able to prevent the same cases in the future somehow, because today is 3am for our dev team. 😴😴😴
Thanks for your understanding. 🙏🙏🙏