/crawl-strategies

Technical methodologies, extraction pipelines, and preservation policies used to capture, render, and archive the fragile infrastructure of the early World Wide Web.

Crawl Philosophy

Modern web crawlers are optimized for dynamic JavaScript frameworks, CDNs, and responsive layouts. The early web (1990–1999) operates on entirely different architectural principles. Our crawling strategies prioritize structural fidelity, resource completeness, and temporal accuracy.

Every crawl session is treated as a digital archaeology operation. We don't just save HTML; we reconstruct the exact environment a browser would have encountered in 1994, 1997, or 1999.

Core Methodologies

📡 Frame-Aware Traversal PRIORITY

Recursive mapping of <frameset> structures, preserving individual frame URLs, target attributes, and layout geometry to prevent modern renderers from collapsing multi-frame documents.

🔗 Relative Path Resolution CRITICAL

Early sites heavily relied on relative linking. Our parser resolves all ../ and `./ references against the originating server's directory structure before archival.

⏱️ Temporal Snapping FEATURE

Crawls are timestamped and versioned. When multiple revisions exist, we snapshot the exact file states per day, allowing researchers to track site evolution over time.

🛡️ Respectful Throttling ETHICS

Strict adherence to robots.txt directives, with a 2-second delay between requests to legacy servers. We prioritize preservation over speed.

Archival Pipeline

Seed Ingestion & Queueing

Root URLs are validated, checked against blocklists, and assigned to specialized crawlers based on detected tech stack (CGI, SSI, PHP3, etc.).

Resource Discovery & Download

HTML is fetched, parsed for links, images, CSS, MIDI, and embedded objects. All assets are downloaded with original HTTP headers and MIME types preserved.

Dependency Graph Construction

A directed acyclic graph (DAG) is built mapping every resource to its parent document. Circular references and missing assets are flagged for reconstruction.

Archival & Cryptographic Verification

Files are compressed, stored in our immutable repository, and SHA-256 hashed. A verification manifest is generated to ensure bit-perfect integrity.

Legacy Technology Handling

Technology	Challenge	Our Solution
CGI-BIN Scripts	Dynamic content that fails to execute on modern servers	Emulated execution environments + static output capture via headless legacy browsers
Server-Side Includes (SSI)	Headers/footers injected at request time, missing in raw HTML	Proxy-level SSI parsing and static injection before archival
MIDI/AU Audio	Proprietary codecs, browser plugin dependencies	Automated conversion to Ogg/Vorbis + original file preservation
ActiveX / NPAPI	Dead plugins, security blocks in modern engines	Sandboxed emulation layer + fallback static render for documentation
Early JavaScript	Unescaped quotes, missing semicolons, `document.layers`	Transpilation to ES5 + polyfill injection for accurate rendering

Compliance & Ethics

We operate under a strict preservation-first, respect-second framework. Our crawling infrastructure is governed by the following policies:

Robots.txt Compliance: We honor Crawl-delay and Disallow directives. Sites with Disallow: / are only archived if public domain or under explicit preservation license.
Takedown Protocol: Legitimate copyright holders or individuals can request removal via our DMCA & Privacy Portal. Requests are processed within 72 hours.
Academic Access: All crawl data is available to verified researchers under Creative Commons Attribution-NonCommercial 4.0.
Server Safety: We never crawl IP ranges without explicit seeding. Legacy servers are rate-limited to prevent DDoS-like behavior.

Sample Crawl Log

[1990-WA] > Initializing crawler node: geo-nyc-04

[1990-WA] > Target: http://www.oldsite.geocities.com/WebRing/9821/

[WARN] Detected frameset layout. Switching to multi-frame traversal mode.

[OK] Downloaded 14 HTML, 22 IMG, 3 MIDI, 1 CSS

[OK] Archive committed. SHA-256: a7f3c9...e41b | Size: 1.2MB

                # crawl_config.yaml - Legacy Web Profile

                protocol: http

                user_agent: "1990WebArchive-Bot/1.4 (+https://1990webarchive.org/bots)"

                delay: 2.0

                max_depth: 5

                render_engine: "netscape-navigator-3.0"

                preserve: ["frames", "ssi", "cgi-output", "headers"]

                ethics:

                    respect_robots: true

                    block_private_ranges: true

                    takedown_endpoint: "api/1990webarchive.org/takedown"