Hi everyone,
I wanted to follow up with some more details about the incident. We experienced two regressions that affected apps on the immediate release cluster during the following timeframes:
- Wednesday, June 18th at 2:45PM EDT to Wednesday, June 18th at 6:25PM EDT
- Monday, June 23rd at 11:55AM EDT until Tuesday, June 24th at 8:10 AM EDT
The first regression affected API calls with parameters delimited by _*_ . The second affected API calls with parameters delimited by _*_ where the parameter value contained an underscore.
Since then, we have implemented additional testing to prevent similar issues from reoccurring, and have identified safer rollout strategies for these kinds of deployments. We have also reverted the change that caused this, so if you’re still experiencing issues, please file a bug report.
We apologize for the inconvenience this has caused.