v2.4.1 • Updated Dec 2024

Migration Guide

Transfer your legacy crawls, archive collections, and preservation sets to the 1990 Web Archive platform with zero data loss.

Overview

This guide walks you through migrating from third-party archive tools, local WARC/ARC collections, or deprecated self-hosted solutions to our cloud-native preservation infrastructure. The process is designed to be incremental, verifiable, and fully reversible until you confirm the final index sync.

ℹ️
Migration can be run in dry-run mode first. No data is written to the production archive until you explicitly confirm the final commit step.

Prerequisites

  • Active 1990 Web Archive account with ARCHIVIST or higher role
  • Python 3.8+ or Node.js 16+ runtime
  • Source data in WARC, ARC, Memento, or JSON-LD format
  • Minimum 500MB free disk space for the migration tool cache
  • Valid API key with migrate:write permissions
⚠️
Files larger than 2GB must be split into chunks before migration. Our parser handles multi-part archives automatically.

Migration Steps

Prepare Your Data

Organize your source files into a single directory. Ensure filenames use UTF-8 encoding and avoid special characters. The tool expects a flat or shallow tree structure.

terminalbash
# Validate directory structure $ ls -la ./source-archive/ total 845M -rw-r--r-- 1 user staff 420M Jan 12 14:02 crawl-1998-geo.warc -rw-r--r-- 1 user staff 180M Jan 12 14:02 crawl-1999-commercial.warc -rw-r--r-- 1 user staff 45K Jan 12 14:02 metadata.json

Authenticate & Connect

Set your API credentials in the environment. The CLI uses OAuth2 token rotation automatically.

terminalbash
$ export WA_API_KEY="your_api_key_here" $ wa-cli config set --region us-east-1 ✓ Configuration saved to ~/.wa-cli/config.yml $ wa-cli auth verify ✓ Authenticated as admin@yourorg.net (Role: ARCHIVIST)

Run Migration CLI

Execute the migration command. Use --dry-run to preview operations before committing.

terminalbash
$ wa-cli migrate start \ --source ./source-archive/ \ --target "collections/legacy-import-2024" \ --format warc \ --parallel 4 [1/4] Validating source files... ✓ [2/4] Uploading chunks to secure gateway... 84.2% [3/4] Reconstructing memento graph... ✓ [4/4] Generating deduplication signatures... ✓ Migration complete. 12,847 pages queued for indexing.

Verify & Index

Run the verification suite to ensure structural integrity. The platform will automatically index assets once verification passes.

terminalbash
$ wa-cli migrate verify --target "collections/legacy-import-2024" Scanning 12,847 records... ✓ 12,847 intact ✓ 0 corrupted ✓ 0 missing redirects Index sync initiated. Estimated completion: ~14 min.
✅
Your collection is now live and searchable. Access it via the Dashboard or query the Memento API using your collection ID.

Troubleshooting

Error CodeMessageResolution
AUTH_403Insufficient privilegesEnsure your API key has migrate:write scope. Contact admin to upgrade role.
WARC_011Invalid magic bytesFile may be corrupted or mislabeled. Use warc-validator before migration.
NET_049Gateway timeoutLarge chunks (>1GB) may require splitting. Run with --chunk-size 500MB.
IDX_007Duplicate URI detectedEnable --dedup merge to append to existing memento timelines instead of failing.

For verbose logging, append --log-level debug to any CLI command. Logs are written to ~/.wa-cli/logs/migrate-YYYYMMDD.log.

Stuck or need enterprise migration?

Our preservation engineers can handle large-scale transfers, custom format parsing, and on-premise air-gapped migrations.

Contact Engineering Read API Docs
"}