Skip to content

Instantly share code, notes, and snippets.

@jordangarcia
Created May 15, 2024 19:34
Show Gist options
  • Save jordangarcia/11e9192b57be513ad351c469cbd70e5b to your computer and use it in GitHub Desktop.
Save jordangarcia/11e9192b57be513ad351c469cbd70e5b to your computer and use it in GitHub Desktop.

This is the overridden Stream.fromSSEResponse static method for the bedrock SDK

https://github.com/anthropics/anthropic-sdk-typescript/blob/main/packages/bedrock-sdk/src/streaming.ts#L36-L61

When an error event happens it throws APIError.generate with the first parameter (the status code) being undefined

https://github.com/anthropics/anthropic-sdk-typescript/blob/main/packages/bedrock-sdk/src/streaming.ts#L82-L88

Which ends up creating an error class here

https://github.com/anthropics/anthropic-sdk-typescript/blob/main/src/error.ts#L101-L110

This caused us to have to write a function like this to try to figure out the status code based on the error constructor and the message

/**
 * Errors that present as stream messages are inflated into APIErrors without
 * status codes.  AFAIK this happens only with @anthropic-ai/bedrock-sdk.
 */
const extractStatusFromError = (error: unknown): number | undefined => {
  if (typeof error !== 'object' || !(error instanceof Error)) {
    return
  }

  if (
    error instanceof AnthropicClient.APIError &&
    typeof error.status === 'number'
  ) {
    return error.status
  }

  // for bedrock errors once streaming has started, the best way to figure out
  // the status code is to look at the underlying errors name
  // See: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModelWithResponseStream.html#API_runtime_InvokeModelWithResponseStream_ResponseSyntax
  //
  // NOTE(jordan): the ts typing for this omits the cause property
  // @ts-ignore
  const causeName = error.cause?.name
  if (causeName === 'ValidationException') {
    return 400
  } else if (causeName === 'InternalServerException') {
    return 500
  } else if (causeName === 'ModelStreamErrorException') {
    return 424
  } else if (causeName === 'ThrottlingException') {
    return 429
  }

  if (RE_INTERNAL_SERVER_MESSASGE.test(error.message)) {
    return 500
  }
  if (RE_RATE_LIMIT_MESSAGE.test(error.message)) {
    return 429
  }

  return undefined
}

Lastly in the makeRequest method of the anthropics core.ts file we never see errors that happen when we make the request. This may be because we're not actually getting rate limited at the API level, but in general the more requests that could return a response that is not ok when an error happens the better. this is a lot easier to handle than pulling apart the stream for errors

https://github.com/anthropics/anthropic-sdk-typescript/blob/main/src/core.ts#L432-L451

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment