Bulk S3 archive

Every dataset is published to S3 as daily Parquet files, one file per symbol per UTC day. For anything longer than a few days of book data, reading the archive directly is faster than paging through the REST API, and it does not count against your rate limit.

The archive is included in every plan, for the datasets and venues the plan covers. Files are identical in content and types to format=parquet responses from the REST API.

Bucket layout

The bucket is in ap-northeast-1 (Tokyo). Partition directories use Hive-style key=value names, so DuckDB, Polars, Spark, Athena and pyarrow can read the partition values as columns without extra configuration.

s3://fdc-archive/trades/venue=hyperliquid/date=2026-09-15/BTC-PERP.parquet
s3://fdc-archive/book_l2/venue=hyperliquid/date=2026-09-15/ETH-PERP.parquet
s3://fdc-archive/book_l4/venue=hyperliquid/date=2026-09-15/SOL-PERP.parquet
s3://fdc-archive/candles_1m/venue=hyperliquid/date=2026-09-15/BTC-PERP.parquet
s3://fdc-archive/extended/venue=hyperliquid/date=2026-09-15/BTC-PERP.parquet
s3://fdc-archive/_manifests/date=2026-09-15/manifest.json

Dataset prefixes

The raw tier (venue messages exactly as received, before normalization) is available on request under raw/venue={venue}/date={YYYY-MM-DD}/ as zstd-compressed JSON lines, one file per channel per capture region per hour. CSV and newline-delimited JSON copies of any dataset can also be delivered in the same layout, gzip-compressed (.csv.gz, .jsonl.gz).

File naming and partitions

• File names are the normalized symbol from Symbology, for example BTC-USD-INV-26MAR27.parquet. Symbols only use [A-Z0-9-], so no escaping is needed. The native symbol is in the venue_symbol column.
• The date partition is the UTC day of ts_capture: a row belongs to the day we received it. A message stamped 23:59:59.999 by the venue and received after midnight is in the next day's file. Rows with source = "venue_archive" have no capture time (ts_capture is null), so they are partitioned by the UTC day of ts_exchange.
• A file exists only for days with at least one row. Delisted instruments stop producing files; the instruments file for the day records the delisting.
• Rows inside a file are sorted by ts_capture, then seq, then position within the venue message.
• Book files (book_l2, book_l3, book_l4) open with a full snapshot at 00:00:00 UTC (snapshot_reason = "file_start") and contain a full-depth snapshot at every UTC hour ("checkpoint"), so each file replays on its own. See Order book depth.

Parquet settings

Files use zstd compression at level 3 and row groups of up to 1,000,000 rows, with min/max statistics on ts_capture so readers that push down time filters skip most row groups on intraday queries. Enum and string columns are dictionary-encoded. Decimals are decimal128(38,18) and timestamps are int64 nanoseconds.

Each file carries key-value metadata in its footer:

Median day, Hyperliquid BTC-PERP, September 2026, from our internal capture.

Getting access

Access is granted per customer to IAM principals in your own AWS account. There are no shared credentials and no access keys to rotate on our side.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FdcArchiveList",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::fdc-archive"
    },
    {
      "Sid": "FdcArchiveRead",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::fdc-archive/*"
    }
  ]
}

Objects are encrypted with S3-managed keys (SSE-S3), so no KMS grant is needed. Data transfer is included in your plan and the bucket is not requester-pays. Reads from ap-northeast-1 are fastest; reads from other regions work the same way. Plans include 5 TB of archive transfer per month on Researcher and 25 TB on Desk; Enterprise limits are set per contract.

Publication schedule

• Files for UTC day D are finalized and published before 02:00 UTC on day D+1. Over the last 90 days of internal capture, the median completion time was 01:12 UTC.
• Each file is written once it has passed validation: sequence continuity checks, cross-region merge, and gap recording. See Data quality.
• The day's manifest is written last. Treat the manifest as the signal that a day is complete; files that appear before it may still be joined by others.
• Rows from a venue's licensed historical archive, where the agreement with that venue includes one, are published in bulk when that venue launches, with source = "venue_archive" in every row.

If publication runs late, the delay is posted on the status page.

Manifests and checksums

One manifest per day lists every file published for that day across all datasets and venues. Your bucket policy grants read access to all manifests, so you can see what exists even for prefixes outside your plan.

{
  "date": "2026-09-15",
  "schema_version": "0.9",
  "generated_at": "2026-09-16T01:09:42.118Z",
  "files": [
    {
      "path": "trades/venue=hyperliquid/date=2026-09-15/BTC-PERP.parquet",
      "dataset": "trades",
      "venue": "hyperliquid",
      "symbol": "BTC-PERP",
      "rows": 1904417,
      "bytes": 42977310,
      "file_sha256": "9f2c4e81b7a05d63e1f8c2a94b7d0e35f6a1c8b29d4e70f3a5b6c1d8e2f9a047",
      "row_sha256": "4be07c19a2f3d6e58b0c7a41e9d2f5b38c6a0e17d4b9f2c85a3e61d07f4cd91a",
      "seq_range": "1094058240111203-1094119875520481",
      "ts_capture_min": 1789430400004211870,
      "ts_capture_max": 1789516799999102335,
      "gap_count": 0,
      "revision": 1
    }
  ]
}

file_sha256 is the digest of the file bytes; verify downloads against it. row_sha256 equals fdc.sha256 in the footer and stays the same if you re-encode the file. S3 ETags are not content hashes for multipart uploads and should not be used for verification.

Revisions

Published data is never edited in place. When a file is corrected (a normalization fix, messages recovered after the daily merge, or a venue busting trades), the canonical key is replaced by the new revision, with fdc.revision incremented in the footer and in the manifest. Earlier revisions stay readable as S3 object versions of the same key for 24 months; list them with aws s3api list-object-versions and read one with --version-id, or pass revision= to the API or SDK. Every revision gets a DATA REVISION entry in the changelog and an email to account contacts on the day it is published.

Reading the archive

AWS CLI

# list available days
aws s3 ls s3://fdc-archive/trades/venue=hyperliquid/

# one file
aws s3 cp s3://fdc-archive/trades/venue=hyperliquid/date=2026-09-15/BTC-PERP.parquet .

# a month of one symbol, keeping the partition layout
aws s3 sync s3://fdc-archive/book_l2/venue=hyperliquid/ ./fdc/book_l2/venue=hyperliquid/ \
  --exclude "*" --include "date=2026-09-*/BTC-PERP.parquet"

DuckDB

INSTALL httpfs; LOAD httpfs;
CREATE SECRET fdc (TYPE s3, PROVIDER credential_chain, REGION 'ap-northeast-1');

-- hourly buy and sell volume for September, read straight from S3
SELECT date_trunc('hour', make_timestamp(ts_exchange // 1000)) AS hour,
       side,
       sum(size) AS volume
FROM read_parquet(
  's3://fdc-archive/trades/venue=hyperliquid/date=2026-09-*/BTC-PERP.parquet',
  hive_partitioning = true
)
GROUP BY ALL
ORDER BY hour, side;

Polars

import polars as pl

lf = pl.scan_parquet(
    "s3://fdc-archive/book_l3/venue=hyperliquid/date=2026-09-1*/ETH-PERP.parquet",
    hive_partitioning=True,
    storage_options={"aws_region": "ap-northeast-1"},
)

cancels_per_day = (
    lf.filter(pl.col("event") == "cancel")
      .group_by("date")
      .agg(pl.len().alias("cancels"))
      .sort("date")
      .collect()
)

Verify against the manifest

import hashlib, json, pathlib
import boto3

s3 = boto3.client("s3", region_name="ap-northeast-1")
day = "2026-09-15"
manifest = json.loads(
    s3.get_object(Bucket="fdc-archive", Key=f"_manifests/date={day}/manifest.json")["Body"].read()
)

for f in manifest["files"]:
    local = pathlib.Path("fdc") / f["path"]
    if not local.exists():
        continue
    h = hashlib.sha256()
    with local.open("rb") as fh:
        for chunk in iter(lambda: fh.read(1 << 20), b""):
            h.update(chunk)
    assert h.hexdigest() == f["file_sha256"], local

Without an AWS account, fdc.download() in the SDK fetches the same files through the API with your key, in the same layout, and verifies each one. See Python SDK.