Skip to main content

File streaming

For large files (hundreds of MB to tens of GB), use a RangeSource instead of loading the entire file into a Uint8Array. A RangeSource is a JS object with synchronous readAt — backed by fs.readSync in Node.js — so the data streams from disk without being loaded into the SDK's memory all at once.

sign-large.ts
import { copyFileSync, openSync, readSync, writeSync, ftruncateSync, fstatSync, closeSync } from "node:fs";
import { AssetSdk, embed } from "@limboai/verifox-sdk-issuer";

function rangeSource(path: string) {
  const fd = openSync(path, "r");
  return {
    get length() { return fstatSync(fd).size; },
    readAt(offset: number, length: number) {
      const buf = Buffer.alloc(length);
      const n = readSync(fd, buf, 0, length, offset);
      return new Uint8Array(buf.buffer, buf.byteOffset, n);
    },
    close: () => closeSync(fd),
  };
}

// The writable form of a `RangeSource` — `embed` patches the manifest into it in
// place (growing via `setLen` as needed), so a master too large to hold in memory
// is never buffered whole either. `embed` writes back through the sink instead of
// returning bytes, so `sidecar` is spliced into a *copy* of the original here.
function rangeSink(path: string) {
  const fd = openSync(path, "r+");
  return {
    get length() { return fstatSync(fd).size; },
    readAt(offset: number, length: number) {
      const buf = Buffer.alloc(length);
      const n = readSync(fd, buf, 0, length, offset);
      return new Uint8Array(buf.buffer, buf.byteOffset, n);
    },
    writeAt(offset: number, data: Uint8Array) { writeSync(fd, data, 0, data.length, offset); },
    setLen(length: number) { ftruncateSync(fd, length); },
    close: () => closeSync(fd),
  };
}

const source = rangeSource("large.mp4");
const { sidecar } = await sdk.sign(
  source,
  "Large Video",
  { origin: { sourceType: "digitalCapture" } },
  { storage: { mode: "embedded" }, format: "video/mp4" }, // ← format required for RangeSource
);
source.close();

copyFileSync("large.mp4", "large.signed.mp4"); // kernel-level copy, no JS buffering
const sink = rangeSink("large.signed.mp4");
embed(sink, sidecar, "video/mp4"); // format required for RangeSink; patches the copy in place, no return value
sink.close();

When data is a Uint8Array the format is inferred from content sniffing. With a RangeSource it must be passed explicitly via options.format — there are no bytes on hand to sniff.

Large video masters​

A multi-GB video master streams end to end: sign and verify read it through a RangeSource, and embed writes it back through a RangeSink (reads plus writeAt/setLen), so the master never enters the JS heap.

See also​