1990 Web Archive
v2.4.1 • DOC-041

WARC Compression & Byte-Range Fetching Specifications

Technical guidelines for efficiently retrieving historical web content using standardized compression and HTTP byte-range requests.

ⓘ Important: This specification applies to all WARC/WACZ files archived from 1990–1999. Ensure your client supports gzip and brotli decoding before making requests.

Overview

The 1990 Web Archive stores historical web content in WARC (Web ARChive) 1.1 format, compressed using modern algorithms to optimize storage and transfer speeds. DOC-041 defines the exact protocol for requesting partial content and handling compressed archives programmatically. This endpoint is designed for researchers, developers, and digital preservationists who need granular access to specific HTTP records without downloading entire multi-gigabyte archives.

Prerequisites

API Endpoint

GET /api/v2/archives/{warc_id}/content/{record_index}

Path Parameters

Parameter Type Description
warc_id string Unique identifier for the WARC container (UUID v4)
record_index integer Zero-based index of the HTTP response record within the archive

Byte-Range Request Support

The archive server fully supports RFC 7233 byte-range requests. This is critical for large WARC files (>2GB) to avoid downloading entire archives when only specific records are needed. The server maintains a block-aligned index mapping logical record offsets to compressed byte boundaries.

Request Example

GET /api/v2/archives/7f9d2c1a-88b4-4e11-9c3a-001b2c3d4e5f/content/42
Host: api.1990webarchive.org
Authorization: Bearer <YOUR_API_KEY>
Range: bytes=1048576-2097151
Accept-Encoding: gzip, br

Response Headers

Header Description
Content-Range Specifies the byte range returned (e.g., bytes 1048576-2097151/3221225472)
Content-Length Size of the returned byte range in bytes
Accept-Ranges Always returns bytes
WARC-Record-Offset Custom header mapping the byte range to the WARC record boundary for faster client-side parsing
⚠️ Warning: Byte-range requests are only supported on uncompressed (identity) or block-aligned compressed archives. Random access on stream-compressed files without block indexes will return 416 Range Not Satisfiable.

Compression Handling

Archives are stored with dual compression layers to balance storage efficiency and retrieval speed:

  1. Block Compression: WARC records are compressed individually using brotli (preferred, window=22) or gzip (legacy, level=6).
  2. Container Indexing: A companion .idx file maps record offsets to compressed block boundaries for O(1) lookup during range requests.

When requesting content, the server transparently decompresses only the requested range and returns it with the appropriate Content-Encoding header. Clients should not attempt to decompress partial streams without validating block boundaries.

Error Responses

Status Code Meaning Resolution
206 Partial Content Byte range successfully returned Process payload and seek to record boundary
400 Bad Request Invalid range format or unsupported encoding Verify Range header syntax
403 Forbidden Insufficient API permissions Check scope includes archive:read
404 Not Found WARC ID or record index does not exist Verify identifiers via list endpoint
416 Range Not Satisfiable Requested range exceeds archive size Use HEAD to fetch Content-Length first

Rate Limits & Best Practices

Version History

Version Date Changes
v2.4.1 Stable 2024-03-15 Added brotli block alignment support; updated error codes and retry logic
v2.3.0 2023-11-22 Introduced WARC-Record-Offset header for faster client-side parsing
v2.1.0 Legacy 2023-06-08 Initial release of DOC-041 specification

Documentation generated for 1990 Web Archive • Last updated: March 2024

Report an issue • Edit this page • API Status