index

AWS JS SDK v3 migration hurdles

While migrating our in-house framework from AWS JS SDK v2 to v3, some aspects were trickier than expected.

Despite notable changes being documented, some things are not.

Avoiding MD5

SQS uses MD5 for message checksums. In v2, computeChecksums allowed disabling it, but in v3 this option only applies to S3 (although the docs mention SQS, the code is only for S3).

MD5 is an issue for FIPS - when enforced, your application is deliberatly crashed if MD5 or other non-compliant algorithms are used through the FIPS provider (Node's crypto with OpenSSL FIPS provider).

To disable MD5 in v3, a new md5 flag was introduced in April 2024, but is not yet documented (as of November 2024).

Forcing HTTP

Services offloading TLS to a proxy (otherwise known as TLS origination/termination) may need to call the AWS endpoint with HTTP as the protocol if they wish to go through said proxy, instead of explicitly proxying the request. This is a common arrangement on an istio service-mesh.

In v3, AWS docs indicate passing an httpAgent in the client options for http, however, it also requires explicitly setting the endpoint (as http://...) in all clients, which may not be possible when endpoints are dynamically determined, or may simply be undesirable.

With v2, http could be forced without explicit endpoints by overriding the send event listener to replace https:// with http://, but this is no longer available in v3.

v3's new middleware stack can be leveraged for this purpose:

const httpPlugin: Pluggable<any, any> = {
     applyToStack: (mwStack: MiddlewareStack<any, any>) => {
         mwStack.add((next: any, _: any) => async (args: BuildHandlerArguments<any>) => {
             if ((args?.request instanceof HttpRequest) && args.request.protocol === 'https:') {
                 args.request.protocol = 'http:';
             }
             return next(args);
         },
             {
                 step: 'build',
                 name: 'http',
             });
     }
 };

When initializing the AWS service, just call httpPlugin.applyToStack(client.middlwareStack). There is a catch, however, regarding STS.

STS gotchas

If you use STS for authentication, e.g. using fromTokenFile from @aws-sdk/credentials-providers, there are two things you need to be aware of:

You need to pass in your plugins, like so: fromTokenFile(config, plugins), including the httpPlugin above if you use it
The AWS_REGION environment variable is ignored and the region has to be explicitly set (in config, above)

The region env var used to be read in v2, and the fact that it's ignored in v3 for some credentials providers is arguably a bug.

DynamoDB gotchas

Dynamo started using account-id endpoints by default in v3 (goes to account-id.dbb.us-west-2.amazonaws.com instead of dynamodb.us-west-2.amazonaws.com). This can be a problem in the aforementioned mesh setup, specially with multiple accounts.

Although this is well documented (1 2 3), it only applies in some circumstances, which is not documented. For instance, when using an IAM User's key and secret, account-id endpoints are NOT used, but when assuming an IAM Role using STS and fromTokenFile like above, account-id endpoints are used (maybe because AWS_ROLE_ARN env var is set in this scenario).

So if you cannot use account-id endpoints, make sure to disable it using accountIdEndpointMode: "disabled", as documented.

Proxy error handling

AWS SDK expects response bodies to be JSON, however, errors may come from a proxy in between the SDK and AWS and might not be JSON. This results in the SDK client code throwing Unexpected token u in JSON at position 0 with Node 18, for example if the error response body is upstream timeout. This can be tricky to debug.

This is the complete stack:

SyntaxError: Unexpected token u in JSON at position 0
   Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.
     at JSON.parse (<anonymous>)
     at /app/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:112:19
     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
     at async parseJsonErrorBody (/app/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:125:17)
     at async de_CommandError (/app/node_modules/@aws-sdk/client-sqs/dist-cjs/index.js:1185:11)
     at async /app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
     at async wrappedHeaderMw (/app/node_modules/newrelic/lib/instrumentation/aws-sdk/v3/common.js:26:12)
     at async /app/node_modules/@smithy/core/dist-cjs/index.js:168:18
     at async /app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
     at async /app/node_modules/@aws-sdk/middleware-sdk-sqs/dist-cjs/index.js:134:16 {
   '$metadata': { attempts: 1, totalRetryDelay: 0 }
 }

There is not a single piece of consumer code, just the SDK. As suggested in the error message, the hidden field can be used to get the raw body:

// When processing error responses
const res = err["$response"];
if (res) {
  err.stack = `${err.stack}\n    body type: ${typeof res.body}\n    body: ${res.body}\n    statusCode: ${res.statusCode}
}

This gives a better understanding of what is going on:

...
    body type: string
    body: upstream request timeout
    statusCode: 504

The aforementioned error is thrown in Node 18. In newer versions of Node, JSON.parse has more friendly error messages:

> JSON.parse('upstream timeout')
Uncaught SyntaxError: Unexpected token 'u', "upstream timeout" is not valid JSON

This was added to v8 in 2022.

As for the AWS SDK, as can be seen in the aforementioned stack, the treatment for JSON.parse errors is to add the original value to the hidden fields, along with the hint to use them.

> try { JSON.parse('upstream timeout'); } catch (e) { console.log(e.name); };
SyntaxError

In v2 the equivalent function is here, see the EXTRACT_ERROR event listener. Although we never saw such errors with v2, when trying to reproduce it we see “Unkown Error” being thrown with the original body as complement, which is a bit easier to debug than the JSON.parse error message, which, then again, is only cryptic in Node 18.

Other considerations

Wireshark is useful to debug and verify the aforementioned scenarios. You can inspect if the traffic is egressing as HTTP by filtering for (_ws.col.protocol == 'HTTP' || _ws.col.protocol = 'HTTP/XML), except for the STS client / credentials provider - STS only listens on port 443, connection attempts to port 80 fail. You can inspect that the traffic is going to the correct region and not using the account-id endpoints by filtering for frame contains "amazon" or frame contains "sts" (replace with service of interest). Just make sure you are not in a VPN, such that you can observe said traffic.

Public domain. Originally published on 17 December 2024.

Home / Contact Info | rss