Obs.: We use AI Gateway as a proxy to OpenAI, but this issue happened with people who use other proxies, such as Helicone. Therefore, I don't think it's a proxy issue.
Yesterday (2024-03-12), something really strange happened with our application that uses the OpenAI API.
At 20:11*, we received our first alarm indicating that OpenAI was returning an excessive number of errors. We received four alarms between 20:11 and 20:14, until we disabled the features utilizing the OpenAI API.
Upon investigating the nature of these errors, we found them to be predominantly Rate Limit Errors (429), which we found peculiar. As a Tier 5 account, we were nowhere near hitting our RPM or TPM limits.
No incident warning was posted on status.openai.com at that time.
Our examination of the API response logs revealed multiple instances of the following message:
Rate limit reached for gpt-3.5-turbo-1106 in organization org-REDACTED on tokens per min (TPM): Limit 40000, Used 39970, Requested 2066. Please try again in 3.054s. Visit https://platform.openai.com/account/rate-limits to learn more. You can increase your rate limit by adding a payment method to your account at https://platform.openai.com/account/billing.
However, the organization identified in these logs was not ours! In fact, the log messages cited two different organizations for that error, neither of which was ours.
By 20:24, the OpenAI status page updated to "We're currently investigating degraded performance impacting our APIs," but made no mention of this specific issue.
We have not changed any API keys in production recently, and all our API keys belong to the same organization. We were limited because those organizations (identified in the log message) are on the Free Tier, where the rate limit is significantly lower (3 RPM, 40000 TPM). This situation led me to worry: Did I inadvertently make API requests on behalf of another account?
This issue was resolved around 21:00.
Currently, I'm pondering several questions:
- Had those organizations not been on the free tier, could I have inadvertently made numerous requests on their behalf, consuming their tokens and incurring charges on their bill?
- If it's possible that I made requests on behalf of others, how can I ensure that no other account made requests on behalf of mine?
I've noticed only a few people discussing this bug that occurred yesterday, suggesting it affected a very small percentage of accounts:
- https://twitter.com/nbashaw/status/1767700262888914968
- https://twitter.com/Samueltates/status/1767695494757978206
*All times are in UTC -03:00.