Capture Standards | 1990 Web Archive

Overview

The 1990 Web Archive adheres to a rigorous, reproducible set of standards designed to capture, package, and preserve web resources exactly as they existed during the foundational era of the internet (1990–1999). Our methodology prioritizes bit-for-bit fidelity, metadata richness, and long-term accessibility.

⚠️ Important Note

All captures are immutable once ingested into the cold storage tier. Amendments require a new capture session with explicit versioning and delta tracking.

Core Principles

Temporal Fidelity

Captures are timestamped to the second and preserve original HTTP headers, server responses, and client-side rendering contexts.

Asset Completeness

Every linked resource (images, scripts, stylesheets, MIDI/audio, frames) is recursively resolved and archived alongside the primary document.

03 03

Render Emulation

Captured content is validated against period-accurate browsers (Netscape 3.0–4.8, IE 3.0–5.0) to ensure visual and functional parity.

Open Standards

All archives utilize WARC 1.0/1.1, MIMET, and OpenArchive formats. No proprietary lock-in. Full API access for researchers.

Supported Formats & Protocols

Category	Supported Standards	Fidelity Target
`Markup`	HTML 2.0, 3.2, 4.01 Transitional/Strict, XHTML 1.0	100% Structural parity
`Styling`	CSS 1, CSS 2 (partial), Inline styles, Presentational HTML tags	98% Visual parity
`Media`	GIF89a, PNG-8/24, JPEG, BMP, MIDI (.mid), AIFF, WAV	Lossless Bit-exact
`Layout`	HTML Framesets, Table-based layouts, <marquee>, <blink>	Emulated Runtime rendering
`Protocols`	HTTP/1.0, HTTP/1.1, FTP, Gopher, WAIS	Mapped Gateway translation

Capture Pipeline

Every URL submitted to the archive undergoes a deterministic 6-stage pipeline:

STAGE 01

URL Normalization

Canonicalization, fragment stripping, and redirect resolution. Duplicate checks against the global index.

STAGE 02

HEAD Probe

Non-destructive request to verify availability, content-type, size, and cache headers. Rate-limiting applied.

STAGE 03

Resource Graph

DOM parsing to extract all relative/absolute assets. Build dependency tree for concurrent fetching.

STAGE 04

Parallel Fetch

HTTP/1.0 GET requests with period-accurate User-Agent strings. Retry logic with exponential backoff.

STAGE 05

WARC Packaging

Generate WARC file with digest (SHA-1), WARC-Profile headers, and concurrent record streaming.

STAGE 06

Index & Store

METADATA injection, Solr indexing, cold storage replication (3x geographic redundancy).

Quality Assurance & Validation

Before a capture is marked COMPLETE, it must pass automated and manual validation checks:

Checksum Verification: Every resource is hashed. Mismatches trigger a re-fetch or flag for manual review.
Link Integrity: All internal and external references are resolved. Broken links are logged but do not halt archival.
Render Audit: Automated screenshot comparison against reference browser engines. Delta > 5% triggers alert.
Metadata Completeness: Title, author, date, copyright notices, and robots directives are extracted and tagged.

// Validation Report Example
{
  "capture_id": "1996-11-04_geocities_cyberia_0892",
  "status": "VALIDATED",
  "assets_resolved": 42,
  "assets_missing": 0,
  "render_delta": "2.1%",
  "checksum_pass": true,
  "warC_size": "4.8MB",
  "timestamp": "1996-11-04T14:22:08Z"
}

Compliance & Ethics

Preservation of historical web content requires careful navigation of legal and ethical boundaries. Our standards mandate:

robots.txt Respect: Disallow directives are honored unless an explicit --force-archive flag is used with researcher justification.
Privacy Redaction: Automated heuristics detect and mask PII (emails, phone numbers, addresses) in public archives. Raw captures retain original data for authorized academic access.
Copyright & Fair Use: All captures are preserved under archival exemption frameworks. Takedown requests are processed within 72 hours with cryptographic unlinking.
Attribution: Every archived page includes a machine-readable provenance block linking back to the original domain and capture timestamp.

WARC Metadata Schema

Standardized header injection for all WARC records:

WARC-Type: response
WARC-Target-URI: http://www.example.com/index.html
WARC-Date: 1995-08-12T09:14:32Z
WARC-Block-Digest: sha1:ABCD1234...
WARC-Concurrent-To: 
WARC-Profile: http://netpreserving.net/warc/warc_1.1/profiles/arcbrowser_1.0
Capture-Engine: 1990-WebArchive-Crawler/v2.4
Browser-Context: Netscape-Communicator/4.05

This schema ensures interoperability with standard preservation toolchains (Arcanist, WARC-IT, OpenWayback) while maintaining era-specific rendering context.

Need Custom Capture Rules?

Enterprise and academic partners can define custom crawl scopes, user-agent rotations, and authentication headers via our API or dedicated SFTP drop.

Configure Capture Rules →