s3-lite-client

This is a lightweight S3 (object storage) client for JavaScript runtimes (Deno, Node 22+, Bun, browsers, etc.). It is designed to offer all the key features you may need, without bloat.

Key features:

• Has no dependencies and minifies to about 21kB (< 8 kB gzipped).
  • For comparison, the official @aws-sdk/client-s3 has 56 dependencies and weighs 399 kB (94 kB gzipped) at the time of writing.
• Works with any S3-compatible storage service: AWS S3, MinIO, Cloudflare R2, Backblaze B2, DigitalOcean Spaces, Supabase storage, etc.
• Works with any JavaScript runtime that supports the fetch API, web streams API, and ES modules (ESM). This client does not depend on any NodeJS APIs nor Deno APIs.
• Implemented in TypeScript and fully typed.
• MIT licensed (originally based on the MinIO JavaScript Client).
• Used in production since 2021.

Supported functionality

• Authenticated or unauthenticated requests
• List objects: for await (const object of client.listObjects(options)) { ... }
  • Handles pagination transparently
  • Supports filtering using a prefix
  • Supports grouping using a delimiter (use client.listObjectsGrouped(...))
• Check if an object exists: client.exists("key")
• Get metadata about an object: client.statObject("key")
  • Can include custom headers in the request: client.statObject("key", { headers: { 'x-amz-checksum-mode': 'ENABLED' } })
• Download an object: client.getObject("key", options)
  • This just returns a standard HTTP Response object, so for large files, you can opt to consume the data as a stream (use the .body property).
• Download a partial object: client.getPartialObject("key", options)
  • Like getObject, this also supports streaming the response if you want to.
• Upload an object: client.putObject("key", streamOrData, options)
  • Can upload from a string, Uint8Array, or ReadableStream
  • Can split large uploads into multiple parts, and uploads a limited number of parts in parallel.
  • Uploads of 64MB or more are split into 64MB parts by default. Since S3 allows at most 10,000 parts, that supports objects up to 640GB; to upload something bigger, or to stream something bigger without passing size, pass a larger partSize (up to 5GB).
  • Can set custom headers, ACLs, and other metadata on the new object (example below).
• Copy an object: client.copyObject({ sourceKey: "source", options }, "dest", options)
  • Can copy between different buckets.
• Delete an object: client.deleteObject("key")
• Create pre-signed URLs: client.presignedGetObject("key", options) or client.getPresignedUrl(method, "key", options)
• Create pre-signed POST policy: client.presignedPostObject("key", options) for direct browser uploads
• Check if a bucket exists: client.bucketExists("bucketName")
• Create a new bucket: client.makeBucket("bucketName")
• Remove a bucket: client.removeBucket("bucketName")

Installation

• Deno: deno add @bradenmacdonald/s3-lite-client
• Deno (no install): import { S3Client } from "jsr:@bradenmacdonald/[email protected]";
• NPM: npm add @bradenmacdonald/s3-lite-client
• Yarn: yarn add jsr:@bradenmacdonald/s3-lite-client
• pnpm: pnpm add jsr:@bradenmacdonald/s3-lite-client
• Bun: bunx jsr add @bradenmacdonald/s3-lite-client
• Browser:

  <script type="module">
  import { S3Client } from "https://esm.sh/jsr/@bradenmacdonald/[email protected]";
  // Or:
  const { S3Client } = await import("https://esm.sh/jsr/@bradenmacdonald/[email protected]");
  </script>
  

Usage Examples (Quickstart)

List data files from a public data set on Amazon S3:

import { S3Client } from "@bradenmacdonald/s3-lite-client";

const s3client = new S3Client({
  endPoint: "https://s3.us-east-1.amazonaws.com",
  region: "us-east-1",
  bucket: "openalex",
});

// Log data about each object found under the 'data/concepts/' prefix:
for await (const obj of s3client.listObjects({ prefix: "data/concepts/" })) {
  console.log(obj);
}
// {
//   type: "Object",
//   key: "data/concepts/updated_date=2024-01-25/part_000.gz",
//   etag: "2c9b2843c8d2e9057656e1af1c2a92ad",
//   size: 44105,
//   lastModified: 2024-01-25T22:57:43.000Z
// },
// ...

// Or, to get all the keys (paths) as an array:
const keys = await Array.fromAsync(s3client.listObjects(), (entry) => entry.key);
// keys = [
//  "data/authors/manifest",
//  "data/authors/updated_date=2023-06-08/part_000.gz",
//  ...
// ]

(Note: As shown above, the list functions normally hide pagination details from you and convert the result to a single list as you retrieve each entry. If you need to request specific pages, see this example.)

Uploading and downloading a file using a local S3-compatible server (e.g. RustFS):

import { S3Client } from "@bradenmacdonald/s3-lite-client";

// Connecting to a local RustFS server:
const s3client = new S3Client({
  endPoint: "http://localhost:9000",
  region: "dev-region",
  bucket: "dev-bucket",
  accessKey: "AKIA_DEV",
  secretKey: "secretkey",
});

// Upload a file:
await s3client.putObject("test.txt", "This is the contents of the file.");

// Now download it
const result = await s3client.getObject("test.txt");
// and stream the results to a local file:
const localOutFile = await Deno.open("test-out.txt", { write: true, createNew: true });
await result.body!.pipeTo(localOutFile.writable);
// or instead of streaming, you can consume the whole file into memory by awaiting
// result.text(), result.blob(), result.arrayBuffer(), or result.json()

Creating a bucket on the S3 service of a local supabase development server:

const client = new S3Client({
  endPoint: "http://127.0.0.1:54321/storage/v1/s3",
  region: "local",
  accessKey: "paste from output of supabase start",
  secretKey: "paste from output of supabase start",
});
await client.makeBucket("my-bucket");

Set ACLs, Content-Type, custom metadata, etc. during upload:

await s3client.putObject("key", streamOrData, {
  metadata: {
    "x-amz-acl": "public-read",
    "x-amz-meta-custom": "value",
  },
});

Create a presigned POST policy for direct uploads from a browser:

// Create a presigned POST policy
const { url, fields } = await s3client.presignedPostObject("my-file.txt", {
  expirySeconds: 3600, // URL expires in 1 hour
  fields: {
    "Content-Type": "text/plain",
  },
});

// In the browser, use the policy for direct uploads:
const formData = new FormData();
// Add all required fields from the presigned POST
Object.entries(fields).forEach(([key, value]) => {
  formData.append(key, value);
});
// Add the file content
formData.append("file", fileInput.files[0]);

// Upload the object using the presigned POST
const response = await fetch(url, {
  method: "POST",
  body: formData,
});

if (response.ok) {
  console.log("File uploaded successfully!");
}

For more examples, check out the tests in integration.ts

Developer notes

To run the tests, please use:

deno lint && deno test

To format the code, use:

deno fmt

To run the integration tests, first start RustFS with this command:

docker run --rm --name rustfs -e RUSTFS_ACCESS_KEY=AKIA_DEV -e RUSTFS_SECRET_KEY=secretkey -e RUSTFS_REGION=dev-region -e RUSTFS_CONSOLE_ENABLE=true -p 9000:9000 -p 9001:9001 --entrypoint /bin/sh rustfs/rustfs:1.0.4 -c 'mkdir -p /data/dev-bucket && exec /entrypoint.sh rustfs'

Then while RustFS is running, run

deno test --allow-net integration.ts

(If you encounter issues and need to debug what RustFS is seeing, you can browse the bucket using the web console at http://localhost:9001/rustfs/console/index.html (log in with the access key and secret key above) and view the server's warning/error logs with this command:)

docker exec rustfs tail -f /logs/rustfs.log