Sending an image Buffer on a multipart/form-data request on Node.js

I need to send an image Buffer on a multipart/form-data (RFC) HTTPS request from a Node.js client to a Java/Spring server over an mTLS connection.

There is a popular form-data npm package for creating multipart/form-data streams. You can use Node's https or axios to send the request. Although form-data accepts a Buffer param, it doesn't build the request as one might expect, which breaks things with cryptic errors from some servers (example questions on SO: 1 2 3 4).

It took me more than I'm happy to admit to debug this so hopefully this saves you a couple of hours.

tl;dr

You need to specify both filename and content-type in formData.append, as the package is unable to infer those from a Buffer (unlike from fs.ReadStream and other streams). Like this:

formData.append('image', buffer, {filename: 'image.jpg', contentType: 'image/jpg'});

In-depth explanation

Let's first examine a request that actually works, using fs.ReadStream:

const formData = new FormData(); // from 'form-data'
formData.append('image', fs.createReadStream(image_path)); // image
formData.append('metadata', JSON.stringify({ // additional data
   ...
}));

const res = await axios.post(url, formData, options)

{
    // only relevant for mtls
    httpsAgent: new https.Agent({
        cert: fs.readFileSync(process.env.clientCertPath!),
        key: fs.readFileSync(process.env.clientKeyPath!),
        ca: fs.readFileSync(process.env.clientCaPath!),
    }),
    // mitmproxy
    proxy: {
        host: 'localhost',
        port: 8080,
        protocol: 'https:'
    },
}

Using mtmproxy to inspect the request we see it is posted as such:

POST /my/server/path HTTP/1.1
Accept: application/json, text/plain, */*
content-type: multipart/form-data; boundary=--------------------------794262681322510475281872
User-Agent: axios/1.1.2
Accept-Encoding: gzip, deflate, br
host: my.server.com
Connection: close
Transfer-Encoding: chunked

2fe5b
----------------------------794262681322510475281872
Content-Disposition: form-data; name="image"; filename="my-image.jpg"
Content-Type: image/jpeg

...a bunch of binary data...

----------------------------794262681322510475281872
Content-Disposition: form-data; name="metadata"

{ ... }
----------------------------794262681322510475281872--

Now we know how a request accepted by our server looks like. Notice the lack of content-length. This might be required by some servers, if so, you need to use getLength but NOT getLenghtSync, confusingly.

Now that we know how a good request looks like, instead of streaming from a file let's try to send a Buffer. The only change to the code above is:

-    formData.append('image', fs.createReadStream(image_path));
+    formData.append('image', fs.readFileSync(image_path));

This is rejected by the server with 400 bad request. The difference from the old request is how the image is encoded in the multipart form-data:

2fe44
----------------------------181015033994339952076268
Content-Disposition: form-data; name="image"
Content-Type: application/octet-stream

In the new request it is missing a filename and the content-type is now application/octet-stream. This is because the form-data package had previously infered the filename and content type from fs.ReadStream, which contains such attributes, while Buffer does not.

So we need to supply that in the options parameter in formData.append, and now it works with a Buffer:

-    formData.append('image', fs.createReadStream(image_path));
+    formData.append('image', fs.readFileSync(image_path), {filename: 'image.jpg', contentType: 'image/jpg'});

This generates a request just like the original one with streams.

Axios

Using form-data

See above, we already used axios and form-data.

Without form-data (automatic serialization)

Axios supports automatic serialization, such that you don't even need to use form-data to build the form, just axios.post with content-type: multipart/form-data. Axios just does the same thing behind the scenes (uses form-data). However, axios also doesn't know how to infer filename and content type from a buffer. For such use cases, it gives you the ability to write a custom serializer (they call it a visitor). Below I show how you could write such a function. This could be useful if you are writing a framework and want to create a generic serializer, the example below is specific to my use case and not very useful (easier to just use form-data ourselves):

const visitor: SerializerVisitor = function (
  this: GenericFormData,
  value: any,
  key: string | number,
  path: null | Array<string | number>,
  helpers: FormDataVisitorHelpers
): boolean {
    let ans = false;
    switch (typeof value) {
        case 'object':
            if (value instanceof Buffer) {
                this.append(key as string, value, {
                    filename: 'image.jpg',
                    contentType: 'image/jpg'
                });
            } else {
                this.append(key as string, JSON.stringify(value));
            }
            break;
        default:
            ans = helpers.defaultVisitor.call(this, value, key, path, helpers);
            break;
    }
    return ans;
};

const res = await axios.post(url, {
    image: fs.readFileSync(image_path),
    metadata: {
      ...
    }
}, {
    httpsAgent: new https.Agent({
        cert: fs.readFileSync(process.env.clientCertPath!),
        key: fs.readFileSync(process.env.clientKeyPath!),
        ca: fs.readFileSync(process.env.clientCaPath!),
    }),
    proxy: {
        host: 'localhost',
        port: 8080,
        protocol: 'https:'
    },
    formSerializer: {
        visitor
    },
    headers: {
        'content-type': 'multipart/form-data'
    }
})

https

Finally, using form-data and https from Node, if we don't want to use axios. I won't proxy these through mtmproxy to keep the client simple, as we already know how a correct request looks like. See this if you need a proxy.

const formData = new FormData();
formData.append('image', fs.readFileSync(image_path), {filename: 'image.jpg', contentType: 'image/jpeg'});
formData.append('metadata', JSON.stringify({
  ...
}));

let chunks: Buffer[] = [];

const _url = new URL(url);

const req = https.request({
    host: _url.hostname,
    path: _url.pathname,
    protocol: _url.protocol,
    method: 'POST',
    headers: {
        ...formData.getHeaders(),
    },
    agent: new https.Agent({
        cert: fs.readFileSync(process.env.clientCertPath!),
        key: fs.readFileSync(process.env.clientKeyPath!),
        ca: fs.readFileSync(process.env.clientCaPath!),
    })
}, (res) => {
    res.on('close', () => {
        const data = Buffer.concat(chunks);
        console.log(data.toString());
    });
    res.on('data', (chunk) => {
        chunks.push(chunk);
    });
});

req.on('error', (e) => {
    console.error(e);
});

formData.pipe(req);

Public domain. Originally published on 17 November 2024.