● STABLE v2.4 LEGACY PROTOCOL Gopher/WAIS

Veronica Indexing Service

High-fidelity reconstruction of the 1992 Veronica search protocol for indexing Gopher and WAIS text menus. Provides modern API access, batch crawling, and metadata extraction for pre-web protocol archives.

View API Reference → Download CLI Tool

Overview

Veronica was the first search engine for Gopher servers, created in 1992 by Steve Dunbar and Michael Grose at NC State University. Unlike modern web search, Veronica didn't crawl HTML; it queried Gopher servers for their directory menus and indexed plain-text resource listings.

The 1990 Web Archive Veronica Indexing Service replicates this historical behavior using modern infrastructure. It allows researchers, archivists, and developers to:

Query reconstructed Gopher/WAIS menu trees across 4.2M archived endpoints
Extract structured metadata from unstructured text menus
Run batch indexing jobs against custom protocol endpoints
Export indexed data to JSON, CSV, or original Gopher+ formats

ℹ️

Veronica indexing is specifically designed for pre-1995 protocol ecosystems. For standard HTTP/HTML archiving, see our Legacy Web Crawler documentation.

Protocol Architecture

Our implementation maintains strict compatibility with the original Veronica protocol while adding modern transport layers and query optimization.

Component	Description	Status
Menu Parser	Regex & heuristic engine for Gopher/WAIS text structures	Production
Session Handler	Maintains stateful 70-port connections with timeout fallbacks	Production
Metadata Extractor	Identifies selectors, hostnames, descriptions, and access flags	Production
Query Optimizer	Limits recursive depth to prevent menu loop traps (default: 4 levels)	Production
Transport Adapter	Wraps legacy TCP streams in REST/gRPC for modern clients	Production

API Reference

All endpoints require authentication via API key. Base URL: https://api.1990webarchive.com/v2/veronica

POST /index

Initiates a new indexing job against a Gopher/WAIS endpoint.

$ curl -X POST https://api.1990webarchive.com/v2/veronica/index \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target": "gopher.example.edu:70",
    "max_depth": 4,
    "timeout_ms": 5000,
    "format": "json",
    "include_metadata": true
  }'

GET /query

Search the indexed Veronica database using keyword or selector matching.

$ curl -G https://api.1990webarchive.com/v2/veronica/query \
  --data-urlencode "q=linux kernel source" \
  --data-urlencode "era=1993-1995" \
  --data-urlencode "protocol=gopher"

GET /jobs/{id}

Retrieve status and results of an asynchronous indexing job.

Configuration

Indexing behavior can be customized via YAML configuration or query parameters.

# veronica-config.yaml
indexer:
    default_timeout: 4000
    max_concurrent: 8
    retry_failed: true
    loop_detection: strict

parser:
    gopher_menus: true
    wais_documents: true
    strip_ansi: false

output:
    format: jsonl
    compress: gzip
    include_raw_menus: true

CLI & Developer Tools

For local testing and offline research, we provide the veronica-cli package supporting Node.js, Python, and Rust.

$ npm install -g @1990archive/veronica-cli
$ veronica scan --target "gopher.tc.umn.edu:70" --depth 3 --export ./archive/
[✓] Connected to legacy endpoint
[✓] Parsing menu tree (42 nodes)
[✓] Exported to ./archive/gopher.tc.umn.edu.json.gz

Full SDK documentation and language bindings are available in the Developer Hub.

FAQ & Troubleshooting

Why does indexing take longer than HTTP crawling?

Veronica relies on synchronous, line-by-line TCP stream parsing over port 70. Legacy servers often have strict rate limits or idle timeouts. Our engine handles this gracefully but requires longer dwell times per node.

Can I index modern websites with this?

No. Veronica is strictly for Gopher, WAIS, and pre-HTTP/1.1 protocol endpoints. Use the Legacy Web Crawler for HTML content.

How do I handle circular menu references?

Enable loop_detection: strict in your config. The parser maintains a visited-selector hash and will truncate branches that exceed the depth limit or repeat selectors.