Protocol Overview | 1990 Web Archive

Capture Protocol 1990-CP v4.2

The Capture Protocol defines how historical URLs are discovered, requested, buffered, and stored. It operates as a deterministic, ethically-scoped HTTP/1.1 crawler optimized for legacy infrastructure.

Depth Control

Configurable recursion limits (1–7 levels) with domain boundary enforcement

Rate Limiting

Adaptive throttling (0.5s–5s delays) to prevent service disruption

Agent Rotation

Ethical UA spoofing matching era-specific browsers (Netscape 3.0–6.0, IE 4–6)

Snapshot Timing

Post-DOMContentLoaded capture with optional 2s render delay for legacy JS

// 1990-CP Configuration Example
{
  "crawl_id": "ARC-1997-BATCH-042",
  "seed_urls": ["http://www.geocities.com/~user"],
  "max_depth": 5,
  "delay_ms": 1500,
  "render_delay_ms": 2000,
  "user_agent": "Mozilla/4.0 (compatible; 1990Archive-Bot/4.2)",
  "respect_robots": true
}

Storage & Format WARC-90

All captured content is packaged using the WARC-90 format, an extension of the W3C WARC standard optimized for large-scale historical preservation, chunked indexing, and cryptographic integrity verification.

Raw Stream

HTTP/1.1 Response

→

Chunking

64MB Segments

→

Compression

Zstd Level 3

→

Signing

SHA-256 + ED25519

→

Index

CDXJ Metadata

Component	Format	Specification
Archive Container	`.warc.gz` / `.warc.zst`	WARC-90 v2.1 (Chunked)
Index File	`.cdxj`	Webrecorder CDXJ Standard
Integrity	SHA-256 Block Hash	Per-chunk verification manifest
Signature	ED25519 Public Key	Non-repudiation archival seal

Metadata Schema MDS-1990

Every archived resource is annotated with a structured metadata profile combining Dublin Core, Schema.org, and custom temporal/technical fields for precise historical retrieval.

// Metadata Payload Structure
{
  "@context": "https://schema.org",
  "type": "WebPage",
  "dcterms:temporal": "1997-08-14T14:32:00Z",
  "dcterms:source": "http://www.geocities.com/SunsetStrip/1990",
  "techStack": {
    "browser_target": "Netscape Navigator 3.0",
    "layout_engine": "HTML Tables / Frameset",
    "multimedia": ["GIF89a", "MIDI", "AVI"],
    "plugins": ["Flash 4", "QuickTime 3"]
  },
  "archiveId": "1990-CP-09A7F2"
}

Retrieval & Rendering RRP-90

The Retrieval Protocol handles temporal resolution, Memento compliance, and legacy rendering emulation. It ensures archived pages are served exactly as they appeared in their original era.

Memento Headers

Memento-Datetime, Link; rel="original", Link; rel="first memento"

Temporal Resolution

Nearest-neighbor or exact-match retrieval via Date header

Legacy Sandbox

Sandboxed IE6/Netscape6 rendering engine with font fallback chains

Asset Rewriting

Automatic path remapping for broken links, missing GIFs, and offline MIDI files

// HTTP Response Headers (Memento Compliant)
HTTP/1.1 200 OK
Memento-Datetime: Sun, 14 Aug 1999 18:22:11 GMT
Link: <http://www.oldsite.com/>; rel="original"
Link: <https://archive.1990web.org/warc/1990-CP-09A7F2>; rel="memento"
X-Archive-Emulation: "Netscape-6.0, Table-Layout, GIF89a"
X-Integrity-Hash: "sha256:a1b2c3..."

API Specification REST v3

Programmatic access to the archive via our JSON:API-compliant endpoints. Supports batch crawling requests, temporal queries, and raw WARC streaming.

Endpoint	Method	Description	Status
`/v3/mementos`	`GET`	Retrieve nearest archived version	Stable
`/v3/crawl`	`POST`	Submit custom capture job	Stable
`/v3/streams/{id}`	`GET`	Download WARC-90 chunks	Beta
`/v3/index/cdxj`	`GET`	Query CDXJ index metadata	Stable

// Authentication & Rate Limits
"Authentication": Bearer Token via API Key
"Rate Limit": 120 requests/min (Standard) | 600 req/min (Research Tier)
"Pagination": Cursor-based with max 1000 records/page
"Response Format": application/vnd.api+json

Standards Compliance

The 1990 Web Archive maintains strict adherence to international digital preservation standards, ensuring long-term accessibility and academic validity.

Memento Protocol

Full HTTP archival specification compliance (RFC 7089)

WARC Standard

IETF WARC 1.1 base with 1990-CP chunking extensions

IIIF Manifests

Image and document sequences exposed via IIIF v3 endpoints

OpenArchive

FAIR data principles, open metadata, and transparent integrity logging

// Compliance Verification Endpoint
GET /v3/compliance/verify
{
  "memento_headers": true,
  "warc_integrity": true,
  "cdxj_schema": true,
  "last_audit": "2024-11-02T09:14:00Z",
  "certification": "OpenArchive Certified v3.1"
}