🌐

1990-web-archive public

Comprehensive web archive preserving the early internet (1990–1999). Crawl, render, and explore the dawn of the World Wide Web.

main 1,247 commits v3.2.1 2,847 stars 312 forks 18 open issues

1990 Web Archive

1990 Web Archive is an open-source web preservation platform dedicated to capturing and rendering the early internet era (1990–1999). It provides tools for crawling, archiving, and viewing historical web content with pixel-perfect accuracy to the technologies of that time.

ℹ️
Currently in active development. We're building the most comprehensive archive of early web content. Check out our roadmap for upcoming features.

πŸ“‘ Table of Contents


βš™οΈ Installation

Prerequisites

  • Node.js β‰₯ 18.0
  • npm β‰₯ 9.0 or yarn β‰₯ 1.22
  • Docker β‰₯ 24.0 (optional, for containerized deployment)
  • At least 50GB free disk space for full archive

From Source

Bash
# Clone the repository
git clone https://github.com/1990webarchive/1990-web-archive.git
cd 1990-web-archive

# Install dependencies
npm install

# Configure environment variables
cp .env.example .env
# Edit .env with your configuration

# Run database migrations
npm run migrate

# Start the development server
npm run dev

Using Docker

Docker
# Pull and run with Docker Compose
docker compose up -d

# Verify the service is running
curl http://localhost:3000/api/health
⚠️
Storage note: The full archive contains over 4.2 million pages. Ensure you have sufficient disk space. The initial download is approximately 180GB compressed.

πŸš€ Quick Start

TypeScript
import { ArchiveClient } from '@1990web/archive-sdk';

const archive = new ArchiveClient({
  apiKey: process.env.ARCHIVE_API_KEY,
  baseURL: 'https://api.1990web.archive/v2',
});

// Search for pages from 1995
const pages = await archive.search({
  era: '1995',
  maxResults: 50,
  sortBy: 'relevance',
});

// Render a preserved page
const rendered = await archive.render(pages[0].id);
console.log(rendered.html);
console.log(rendered.timestamp); // '1995-03-14T08:30:00Z'

✨ Features

Feature Status Details
Deep Web Crawling βœ… Live Crawls archived URLs with protocol-specific handlers
Pixel-Perfect Rendering βœ… Live Authentic Netscape-era rendering engine
GeoCities Recovery βœ… Live 180k+ personal homepages restored
Advanced Search βœ… Live Era filters, technology tags, visual style search
Cryptographic Provenance πŸ”„ Beta Blockchain-verified archive timestamps
Live Preview Mode πŸ”„ Beta Interactive browser simulation
API v2 REST/GraphQL πŸ”„ Beta Full programmatic access
AI-Powered Recovery 🚧 WIP ML-based damaged page reconstruction
Collaborative Annotations πŸ“‹ Planned Community-driven page annotations

πŸ—οΈ Architecture

The system is composed of several core services:

Service Language Description
archiver TypeScript Distributed crawler with era-specific browser emulation
renderer Rust + Wasm Pixel-perfect rendering of legacy HTML/CSS/JS
search-engine Go Inverted index with era-aware ranking
storage Rust Distributed blob storage with deduplication
api-gateway Rust REST + GraphQL API with rate limiting
provenance Go Cryptographic signing and verification
Architecture Overview
# β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
# β”‚                  Client / SDK                      β”‚
# β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
#                        β–Ό
# β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
# β”‚              API Gateway (Rust)                    β”‚
# β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
# β”‚  β”‚   REST   β”‚  β”‚  GraphQL  β”‚  β”‚  WebSocket     β”‚ β”‚
# β”‚  β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
# β””β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
#         β–Ό              β–Ό                β–Ό
#   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
#   β”‚ Search  β”‚   β”‚ Render   β”‚    β”‚Provenance β”‚
#   β”‚ Engine  β”‚   β”‚ Engine   β”‚    β”‚ Service   β”‚
#   β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
#        β–Ό              β–Ό                β–Ό
#   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
#   β”‚           Distributed Storage               β”‚
#   β”‚       (Deduplicated Blobs + Indexes)        β”‚
#   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
#                                                    β”‚
#         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
#         β–Ό
#   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
#   β”‚           Archive Nodes (Crawlers)           β”‚
#   β”‚  (Emulating Netscape, IE, Mosaic, etc.)     β”‚
#   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
# ──────────────────────────────────────────────────

πŸ“‘ API Reference

Base URL

REST API
BASE_URL = https://api.1990web.archive/v2

Endpoints

Method Endpoint Description
GET /health Health check and service status
POST /search Search archived pages with era filters
GET /pages/:id Retrieve page metadata and content
GET /pages/:id/render Get rendered HTML with original styling
GET /pages/:id/provenance Cryptographic proof of archival date
GET /collections/:id Get curated collection of pages
POST /collections Create a new curated collection
GET /stats/era/:year Statistics for a specific year
GET /technologies List technologies used across archive

Search Request Example

JSON - Request Body
{
  "query": "web ring guestbook",
  "filters": {
    "era": "1990-1999",
    "technology": ["frameset", "gif"],
    "siteType": "personal"
  },
  "sortBy": "relevance",
  "page": 1,
  "perPage": 20
}

Response Example

JSON - Response
{
  "ok": true,
  "total": 42187,
  "page": 1,
  "perPage": 20,
  "results": [
    {
      "id": "gct:geocities:8832:index.html",
      "url": "http://geocities.com/Area51/Cavern/8832/",
      "snapshotDate": "1997-06-12T00:00:00Z",
      "title": "β˜… Cool Web Page β˜…
        - Under Construction",
      "technologies": ["frameset", "animated-gif",
                        "marquee", "guestbook"],
      "provenance": "sha256:a8f5f167f44f4964..."
    }
  ]
}

πŸ“– Usage Examples

CLI Usage

Terminal
# Search for pages from the 1990s
1990archive search "web ring" --era 1990s --limit 10

# View details of a specific archived page
1990archive show "gct:geocities:8832:index.html"

# Render a page in a simulated browser
1990archive render "gct:geocities:8832:index.html" --browser "netscape3"

# Download all pages from a specific GeoCities area
1990archive download "gct:Area51/*" --output "./archive/area51"

# Generate a report on technology adoption
1990archive stats --era 1990s --output "report.pdf"

Programmatic Archive Collection

TypeScript
import { CollectionBuilder } from '@1990web/archive-sdk';

// Build a collection of GeoCities pages from 1996
const builder = new CollectionBuilder({
  name: 'Geocities Golden Era',
  description: 'Best of 1996 GeoCities personal pages',
});

builder
  .filter({ era: '1996', siteType: 'personal' })
  .filter({ technologies: ['animated-gif', 'marquee'] })
  .sortBy('popularity')
  .limit(100);

// Export as HTML (authentic 90s style!) or JSON
await builder.export('geo-golden-era', 'html');
console.log(`Exported ${builder.count} pages`); 
// β†’ Exported 100 pages

πŸ“‚ Project Structure

File Tree
1990-web-archive/
β”œβ”€β”€ # Core Services
β”‚   β”œβ”€β”€ archiver/          # Distributed crawler
β”‚   β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ protocols/     # HTTP, FTP, Gopher handlers
β”‚   β”‚   └── renderers/    # Browser emulators
β”‚   β”‚
β”‚   β”œβ”€β”€ renderer/          # Pixel-perfect rendering engine
β”‚   β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ wasm/          # WebAssembly modules
β”‚   β”‚   └── fonts/         # Period-appropriate fonts
β”‚   β”‚
β”‚   β”œβ”€β”€ search-engine/     # Go-based search index
β”‚   β”œβ”€β”€ storage/           # Distributed blob storage
β”‚   β”œβ”€β”€ api-gateway/       # REST + GraphQL gateway
β”‚   └── provenance/        # Cryptographic signing
β”‚
β”œβ”€β”€ packages/              # SDK & Libraries
β”‚   β”œβ”€β”€ archive-sdk/       # TypeScript SDK
β”‚   β”œβ”€β”€ cli/               # Command-line interface
β”‚   └── react-components/  # React archive viewer
β”‚
β”œβ”€β”€ docs/                  # Documentation
β”‚   β”œβ”€β”€ getting-started.md
β”‚   β”œβ”€β”€ api-reference/
β”‚   └── contributing.md
β”‚
β”œβ”€β”€ test-data/             # Test fixtures & samples
β”œβ”€β”€ .github/               # CI/CD workflows
β”œβ”€β”€ docker-compose.yml
β”œβ”€β”€ package.json
β”œβ”€β”€ tsconfig.json
β”œβ”€β”€ .env.example
└── README.md

🀝 Contributing

We welcome contributions from everyone! Whether you're a web historian, a browser engine developer, or just someone who loves the aesthetic of the early web β€” there's a place for you here.

Getting Started

  1. Fork the repository
  2. Clone your fork locally
  3. Create a branch for your feature: git checkout -b feature/amazing-rendering
  4. Commit your changes: git commit -m 'feat: add animated GIF support'
  5. Push to the branch: git push origin feature/amazing-rendering
  6. Open a Pull Request

Areas We Need Help With

  • 🎨 Browser Emulation β€” Help add support for Mosaic, Internet Explorer 3, Opera 3.x
  • πŸ“ Documentation β€” Improve docs, add tutorials, write blog posts
  • πŸ§ͺ Testing β€” Expand test coverage for rendering edge cases
  • 🌍 Protocol Support β€” Add handlers for FTP browsing, Gopher, WAIS
  • πŸ—„οΈ Storage Optimization β€” Improve deduplication and compression algorithms
  • πŸ“Š Analytics Dashboard β€” Build visualizations for web history trends
πŸŽ‰
First-time contributor? Look for issues labeled good first issue β€” these are specifically curated for new contributors.

πŸ“‹ Code of Conduct

This project adheres to the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to conduct@1990web.archive.


πŸ‘₯ Contributors

This project exists thanks to all the people who contribute.

          +47 contributors


πŸ“ˆ Contribution Activity

Last 12 months

πŸ“œ License

This project is licensed under the MIT License. See the LICENSE file for details.

The early web was built by everyone β€” let's keep it that way. This archive belongs to the community.


Made with πŸ’š by the 1990 Web Archive team. Preserving history, one page at a time.
⬆ Back to top

"}