DocumentationExamplesGuides

Enabling file streaming operations

Support for streaming is critical for applications that need to send or receive large amounts of data between client and server without first buffering the data into memory, potentially exhausting this system resource.

Streaming Download

If you create an endpoint with a top-level binary response body, you can treat that response as a streamable, and iterate over it without loading the entire response into memory. This is useful for large file downloads, long-running streaming responses and more.

In an OpenAPI documented this can be modeled as a binary stream. Here's an example of a get operation with content type as application/octet-stream.


/streamable:
get:
operationId: streamable
responses:
"200":
description: OK
content:
application/octet-stream:
schema:
title: bytes
type: string
format: binary

For response streaming in Typesccript SDKs expose a ReadableStream, a part of the Streams API web standard (opens in a new tab).


import fs from "node:fs";
import { Writable } from "node:stream";
import { SDK } from "@speakeasy/super-sdk";
async function run() {
const sdk = new SDK();
const result = await sdk.streamable("UR123");
const destination = Writable.toWeb(
fs.createWriteStream("./report.csv")
);
await result.data.pipeTo(destination);
}
run();

Streaming Uploads

Uploading a very large file is one use case where streaming can be useful. Certain SDK methods will be generated that accept files as part of a multi-part request. It is possible and typically recommended to upload files as a stream rather than reading the entire contents into memory. This avoids excessive memory consumption and potentially crashing with out-of-memory errors when working with very large files. In this example a request to upload a file is managed as a multipart/form-data request.


/file:
post:
summary: Upload file
operationId: upload
requestBody:
content:
multipart/form-data:
schema:
$ref: '#/components/schemas/UploadFileRequest'
required: true
responses:
'200':
description: ''
headers:
Action-Id:
required: true
schema:
type: string
content:
application/json:
schema:
$ref: '#/components/schemas/File'

As an example, in Node.js v20, streaming a large file to a server using an SDK is only a handful of lines. On the browser, users would typically select files using <input type="file"> and the SDK call is identical to the sample code below.


import { openAsBlob } from "node:fs";
import { SDK } from "@speakeasy/super-sdk";
async function run() {
const sdk = new SDK();
const fileHandle = await openAsBlob("./src/sample.txt");
const result = await sdk.upload({ file: fileHandle });
console.log(result);
}
run();

Depending on your JavaScript runtime, there are convenient utilities that return a handle to a file without reading the entire contents into memory: