Storage & Format Architecture

Storage Architecture

The archive utilizes a multi-tiered, geographically distributed storage system designed for durability, retrieval efficiency, and cost optimization. All data is encrypted at rest using AES-256-GCM and in transit via TLS 1.3.

Hot Tier (NVMe SSD): Active crawling jobs, recent captures, and frequently accessed metadata indexes. Low-latency retrieval for real-time API queries.
Warm Tier (Enterprise HDD): Structured archives from the last 10 years. Optimized for sequential reads and batch processing pipelines.
Cold Tier (LTO-9 + Object Storage): Long-term preservation of raw captures, legacy format bundles, and backup replicas. 3-2-1 backup strategy enforced across 4 continental zones.

                    # Example storage routing rule
                    route /archive/{id} {
                      if age < 365 days: serve from hot-tier
                      if age >= 365 && < 5y: replicate to warm-tier
                      if age >= 5y: migrate to cold-tier + tape vault
                    }
                

Format Standards

We adhere to open, self-describing, and internationally recognized preservation standards to ensure long-term accessibility without vendor lock-in.

WARC-2017 / WARC-2022: Primary web archive format for HTTP/S captures, including headers, payloads, and concurrency logs.
WACZ: Zipped container bundling WARC, CDX, index, and metadata into a single portable file. Default export format for developers.
Raw HTML + Asset Bundles: Unmodified DOM snapshots paired with extracted CSS/JS/images for emulation environments.
JSON-LD + Schema.org: Machine-readable metadata for pages, domains, capture timestamps, and provenance chains.

Checksums & Integrity

Every archived object is fingerprinted and verified upon ingestion and periodic integrity audits. Bitrot detection runs nightly across all tiers.

                    integrity {
                      payload: "sha256:8f43...2a19"
                      headers: "blake3:e7b2...9c41"
                      manifest: "crc32c:a3f8..."
                    }
                

Format Migration Pipeline

As rendering engines and storage media evolve, obsolete formats are systematically assessed, transcoded, or emulated to preserve functional fidelity.

01

Assessment

Scan format health, dependency rot, and emulation viability using automated heuristics.

02

Transcoding

Convert legacy structures (framesets, ActiveX, Flash) to static HTML5/CSS or sandboxed WASM emulation.

03

Verification

Digital diff rendering, checksum comparison, and metadata cross-validation.

04

Technical Specifications

Format	Extension	Compression	Retention	Status
WARC	.warc.gz	gzip (level 6)	Permanent	ACTIVE
WACZ	.wacz	zip64 + brotli	Permanent	EXPORT
ARC	.arc	none / bzip2	Legacy	ARCHIVED
MHTML	.mht	base64 inline	Reference	LEGACY
JSON-LD	.json	none	Permanent	METADATA

API & Developer Access

Query storage metadata, request format-specific downloads, and monitor integrity reports via our RESTful API. Rate limits apply; authenticate with API keys.

Endpoints

GET /api/v2/archive/{capture_id}/manifest

GET /api/v2/archive/{capture_id}/download?format=wacz

POST /api/v2/integrity/verify

GET /api/v2/storage/tier-status

                    # CLI tool example
                    1990-archive fetch --id "1995-geo-8821" --format wacz --output ./captures/
                    # → [✓] Downloaded 14.2 MB | SHA256 verified | 3 pages bundled
                

Full SDKs are available for Python, Node.js, and Go. Documentation and rate limit policies are available in the developer portal.