Service-to-Service Communication Degradation
Summary
During the incident, we observed intermittent, seemingly random delays when refreshing the application at app.transifex.com. These delays affected the handling of customer requests and overall platform responsiveness.
Investigation & Findings
Following the initial reports, we investigated our load balancer logs and identified an unusually high volume of requests between our core service and the Transifex-delivery service.
By correlating the load balancer logs with our application logs, we determined that a substantial number of requests were being throttled and retried between those services. This led to performance degradation when handling incoming requests on our main platform.
Impact
The issue caused performance degradation across our platform at irregular time intervals.
Mitigation & Resolution
As part of scheduled maintenance, we temporarily paused translation activity to adjust certain configurations.
After the maintenance changes were applied, translation activity was re-enabled. However, the system did not immediately return to a fully stable state. To ensure reliable request handling and prevent further performance degradation, we made additional adjustments to the affected language configurations to reduce internal service load and fully stabilize the platform.
Once system stability was fully confirmed, all affected languages and their translations were restored, and the platform was gradually returned to full functionality.
Preventive Actions & Improvements
To reduce the risk of similar issues occurring in the future, we have introduced improvements to how requests are handled between our core platform services and the Transifex-delivery service, including configuring throttling and retry mechanisms.