--- name: archive-archaeology description: Recover a person's or a company's own online history from public archives (the Wayback Machine, GitHub, company registers, blogs, press) and turn it into a structured archive of entries, each with cited sources, a confidence level and a publishing gate. Use when someone wants to rebuild a portfolio, CV, catalogue or record of work from what the internet still holds, without inventing anything. --- # Archive archaeology A procedure for mining a subject's online history into a structured archive. The subject is the person or organisation whose record you are compiling; the owner is whoever is directing you and can answer questions about that record. Often they are the same person. The method was developed compiling one person's site from twenty-five years of dead domains, and it is written here with that person taken out. The whole method rests on one idea: extraction, never inference. You are a scribe, not a biographer. If a source does not say it, the archive does not say it. ## Before you start Agree these with the owner in writing, in the archive folder, before the first fetch: 1. **Scope.** Whose contribution is being recorded. Everything else is context. 2. **Confidential list, by omission.** What must never be recorded at all: live fundraising, private financials, anything under NDA, children's names, home addresses. Do not write these into a "do not publish" file; a private fact written anywhere in the repository is in its history forever. Omission is the only safe mechanism. 3. **Where the raw material lives.** Outside the code repository. Raw scrape is unedited, unverified and full of other people's words. Track only the procedure and the finished entries; leave the raw corpus out of version control. 4. **Candidate domains and accounts.** Every domain, handle and repository the owner recalls. Treat each as a lead, not a fact. Ownership is checked in step 2, never assumed from a name. ## Principles - **Extraction only.** Text is recovered verbatim. Nothing is inferred, completed, tidied or guessed: not a date, a client, a role, a figure or an outcome. Absent in the source means absent in the entry. - **Provenance on every record.** Every entry cites at least one source, and every source carries the original URL and, where one exists, the dated snapshot that proves it. A claim a reader can check in one click is a citation; anything else is a boast. - **Recovered is not verified.** A figure recovered from a page is evidence that the page said it, not that it is true. Confidence levels (below) record the state of the evidence, not anyone's opinion of it. - **Recollection is marked, never laundered.** A date the owner remembers but cannot document is stored with its basis and rendered hedged ("c. 1997"). It never hardens into a documented fact. Expect the owner's memory to be wrong sometimes; documented sources win. - **A capture date is not a publication date.** The single most likely extraction error. A snapshot proves a page existed by that date, not when the thing it describes was made. Write this rule into every extraction prompt. - **Scope the dig to the subject's own contribution.** A partner's archive is context, not the target. Ask what the entry is before deciding what to fetch; the depth of research should track the size of the contribution, not the fame of the collaborator. ## Procedure ### 1. Inventory the domains For each candidate domain, ask the Wayback Machine's CDX index for every address it has ever captured: ``` http://web.archive.org/cdx/search/cdx?url=&matchType=domain&output=json &collapse=urlkey&fl=original,timestamp&filter=statuscode:200&filter=mimetype:text/html ``` Keep the full list. It is as valuable as the pages: it tells you what existed and when it was last seen, and later it becomes the redirect map (step 12). If a domain changed hands, restrict the years to the period the subject held it (drop `collapse=urlkey` so you see every capture's timestamp, then filter). ### 2. Verify ownership against content Name-matching is a lead, not a verdict. Open a handful of snapshots and check the site is the subject's: page structure, an about page, a client list, a byline. Record every verdict, included or excluded, with the reason, in one table in the archive folder. Expect reversals in both directions: a domain excluded on recollection may be reinstated when its structure matches, and a domain that matches a CV line may turn out to be client hosting or somebody else's project with the same name. ### 3. Fetch original bytes, and make every step re-runnable Fetch each page in its original form, not through the archive's viewer: ``` https://web.archive.org/web/id_/ ``` The `id_` flag returns the page as captured, without the archive's toolbar. The archive drops connections constantly; expect a third to a half of fetches to fail on any one pass. Write every miner to be idempotent: a second run fills gaps and never refetches what it already has. Never treat a failed fetch as a missing page. Keep one record per fetched page (URL, timestamp, snapshot URL, title, extracted text), one file per domain. ### 4. Filter platform sites to their shell Where a site hosted other people's material (a job board, a community platform, a forum, an events listing, a multi-author blog), mine only the shell: the home page, about, how-it-works, pricing, press, team, and any page that describes the product. Skip item pages, threads, comments and profiles. Record the concept, not the contents. One job board in the original dig came to more than three thousand addresses and eighteen pages of actual product. ### 5. Repositories Use the code host's API, not the web pages. Take metadata first: name, description, homepage, created and last-pushed dates, fork flag, primary language, stars. Read a README only afterwards, and only when it describes this project rather than the framework it started from. Decline to describe a repository whose README is starter boilerplate; an empty description is more honest than a plausible one. Republished is not authored. A repository that is not a fork can still be someone else's work repackaged (a font package, an API client, "npm version of..."). Read the description for words like "clone", "port", "created by". Check the commit authors: a repository the subject never committed to is not the subject's work, whatever its name. ### 6. Company registers A statutory register (Companies House in the UK, or its equivalent) is the right and only source for legal identifiers about a company: registered number, incorporation date, dissolution date, company type. Never reconstruct these from memory; leave a visible marker and ask the owner. The register is not a source for who to credit. Officers listed in a filing have not published their involvement. Credit only names the person or the organisation published about themselves. Never carry a register's name format ("SURNAME, Forename Middle") into an entry. Never record a registered office that is a residential address; "a registered office in the city" carries the meaning without publishing someone's home. Where a register and a public account disagree, record that there is a discrepancy without reproducing the register's contents. ### 7. The subject's own writing Blogs, essays and articles the subject wrote. Recover the body verbatim as Markdown with frontmatter carrying the original URL, the snapshot and the published date as stated on the page. Do not edit the text. Where a piece's date is not stated, look for an index page whose dated neighbours bracket it and record the date as circa, with the bracketing evidence in the notes. On a multi-author platform, authorship is a gate with quoted evidence. Attribute a piece to the subject only when a byline says so, and quote the byline in the source record. Keep two separate outcomes for everything else: "byline names someone else" and "no byline found". The first is a fact; the second is unknown, and quite possibly the subject's own work awaiting another look. Never report the second as the first. ### 8. Press, exhibitions and events Record press as literature: outlet, title if recovered (never reconstructed from a URL slug), date, URL, snapshot, and a short verbatim quote. Record showings as exhibited: event, venue, date, kind (exhibited, presented, screened, performed, commissioned), and the page that states it. Both belong to the thing the entry is, never to a larger work it fed into: building the website for a touring show does not inherit the show's tour, and a review of the show is not press about the website. ### 9. Self-curated and private sources The public archive goes dark for recent years. Close the gap with sources the subject controls: a LinkedIn or CV export (self-curated and dated; the highest value per minute of any source), the first and last commit dates of local repositories, membership lists and old calendars as memory prompts. Cite these as their own kind of source ("cv-export", "local-repo", "recollected") so a reader can see which facts rest on testimony rather than on a public page. ### 10. Extract in three layers 1. **Deterministic first, no language model.** API metadata, frontmatter, dates in URL paths. Cheap, exact, no room for invention. 2. **Model extraction for prose, one run per source group,** with the output validated against the entry schema (below) and the prompt carrying the hard rules: extraction only; every field traceable to quoted source text; a capture date is not a publication date; decline rather than guess. A good run refuses to describe things the source does not describe. 3. **Deterministic merge.** Join entries about the same thing by slug. Where two sources disagree, do not let the model or yourself pick a winner: demote the entry to the lowest confidence level, keep both values, and surface the conflict to the owner as an evidence table. ### 11. Assign confidence and apply the gate See the two sections below. Do this in the data, so that the gate is a filter over one set of files rather than a second, hand-maintained list. ### 12. Build the redirect map Invert each entry's cited sources: every historical URL of the subject's own sites maps to the entry it now supports. Serve those as permanent redirects so years of inbound links resolve again instead of failing. Build it during recovery; it is nearly impossible to build credibly afterwards. Reserve the new site's live routes first, or a historical page cited as a source will redirect a current page to an entry. ### 13. Keep a review queue for the owner Every conflict, every unresolved date, every fact only the owner can know goes to a queue with the evidence beside it. The owner rules; you record the ruling and its date in the entry's notes. Never resolve a conflict yourself. ## The entry One thing made is one file, named by its slug. Sections of a site or a portfolio are views over this one set, chosen by rules over the fields; they are never separate data models. The test of a good schema is that every correction costs one field, not one refactor. A sketch of the shape (JSON Schema, abridged): ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "Entry", "type": "object", "required": ["slug", "kind", "title", "oneLiner", "sources", "confidence"], "additionalProperties": false, "properties": { "slug": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, "kind": { "enum": ["company", "org", "role", "project", "hack", "release", "writing", "talk", "award", "grant", "education", "note"] }, "title": { "type": "string", "minLength": 1 }, "oneLiner": { "type": "string", "description": "One line, verbatim from a source wherever one exists." }, "dates": { "type": "object", "additionalProperties": false, "properties": { "from": { "type": "string", "pattern": "^\\d{4}(-\\d{2})?(-\\d{2})?$" }, "to": { "type": "string", "pattern": "^\\d{4}(-\\d{2})?(-\\d{2})?$", "description": "With no `from`: the thing existed BY this date (a capture bound)." }, "ongoing": { "type": "boolean" }, "basis": { "enum": ["documented", "recollection", "circa"] } } }, "stage": { "enum": ["sketch", "prototype", "shipped", "scaled", "handed-over", "dead"] }, "status": { "enum": ["live", "dead", "archived-only", "unknown"] }, "role": { "type": "string", "description": "Only if a source states it." }, "stack": { "type": "array", "items": { "type": "string" } }, "outcome": { "type": "string", "description": "Scale, exit, award, shutdown, or a visible TODO marker. Never a guess." }, "collaborators": { "type": "array", "items": { "type": "string" }, "description": "Only names the source names AND the person published about themselves. Never from a register. Never children." }, "parent": { "type": "string", "description": "Slug of the company or body this was made at or with, where a source states the link." }, "ref": { "type": "string", "description": "Catalogue or issue number within the parent's scheme." }, "links": { "type": "object", "properties": { "live": { "type": "string" }, "repo": { "type": "string" }, "snapshot": { "type": "string", "description": "Preferred over a dead live link; never ship a 404." }, "screengrab": { "type": "string", "description": "Self-hosted copy of the evidence." } } }, "sources": { "type": "array", "minItems": 1, "items": { "type": "object", "required": ["url"], "properties": { "url": { "type": "string" }, "snapshot": { "type": "string", "description": "https://web.archive.org/web//" }, "kind": { "type": "string", "description": "e.g. github-api, cv-export, local-repo, recollected" } } } }, "exhibited": { "type": "array", "items": { "type": "object", "required": ["event"], "properties": { "event": { "type": "string" }, "venue": { "type": "string" }, "date": { "type": "string" }, "to": { "type": "string" }, "kind": { "enum": ["exhibited", "presented", "screened", "performed", "commissioned"] }, "url": { "type": "string" }, "snapshot": { "type": "string" } } } }, "literature": { "type": "array", "items": { "type": "object", "required": ["outlet"], "properties": { "outlet": { "type": "string" }, "title": { "type": "string" }, "date": { "type": "string" }, "url": { "type": "string" }, "snapshot": { "type": "string" }, "screengrab": { "type": "string" }, "quote": { "type": "string", "description": "Short and verbatim." } } } }, "relations": { "type": "array", "description": "Lineage, stored on the EARLIER thing pointing forward.", "items": { "type": "object", "properties": { "kind": { "enum": ["became", "seeded", "renamed-to", "absorbed-into", "superseded-by", "inspired"] }, "slug": { "type": "string" }, "name": { "type": "string" }, "url": { "type": "string" }, "snapshot": { "type": "string" }, "note": { "type": "string" } } } }, "private": { "type": "boolean", "description": "Kept in the data, excluded from every route and index at load time." }, "confidence": { "enum": ["recovered", "partial", "needs-owner"] }, "notes": { "type": "string", "description": "Extraction caveats: conflicts seen, ambiguities left open, rulings and their dates." }, "body": { "type": "string", "description": "Longer recovered prose, verbatim, Markdown-safe." } } } ``` Notes on the shape: - **Kind is single-valued.** A hack that grew into a company is two entries and a `relations` edge, never one entry with two kinds. Each has its own dates, evidence and confidence. - **Dates carry their own hedge.** `{ to }` alone means "existed by"; `basis: "circa"` means bounded by neighbouring evidence but stated by nobody; `basis: "recollection"` means the owner remembers it. Render every hedged form visibly ("by 2012", "c. 1997"). A rough date must never pass as a documented one. - **The museum-catalogue trio.** `sources` is provenance, `exhibited` is where the thing was shown, `literature` is what was written about it. All three belong to the thing this entry is. - **An identifier is not the thing it identifies.** One row is one record. A record can carry two catalogue numbers; the same record can sit under a spare number; a different series can share a prefix; an episode number can masquerade as a catalogue number. Before adding a number, check the thing is not already present under another, and cross-check its date against its neighbours. - **One institution can have several entities in the same database.** Before concluding a roster is complete, search the database for other entities with the same name, and confirm which one you are in. - **Absence in a source is not evidence of absence.** Record "no evidence found" and "proven never to have existed" as different states. The first invites another dig; the second closes one. ## Confidence levels - **recovered.** Every field was taken verbatim from a cited source. - **partial.** Recovered, with normalisation on the way in: a date reshaped to ISO form, a slug's casing, a title trimmed of navigation chrome. Nothing added. - **needs-owner.** Something is uncertain, two sources disagree, or only the owner can know it. Both values are kept; the question sits in the review queue. Confidence is not a to-do flag: an entry the owner has ruled on and that is now fully sourced goes back up. ## The publishing gate Confidence gates display, not existence. Decide per display surface: - **Surfaces that speak for the subject now** (a front page, a work history, a current-roles list) show only `recovered` and `partial` entries. - **The complete record** (an archive page, a year-by-year list) shows everything, with every unverified fact rendered as a visible marker such as `«TODO: confirm the client's name»` rather than a plausible guess. A page cannot ship with invented details if the invention has nowhere to hide. - **Surfaces where the artefact is its own evidence** (a list of published writing, a list of awards) may show every level, because the piece or the prize is checkable directly. - **Curation is a separate act.** A "selected" flag the owner sets is not a recovered fact and is not gated by confidence. - **`private` is absolute.** Kept in the data for the next dig to find already ruled on; excluded from every route, index and machine-readable listing at load time. Run the gate in code over the one set of files. Validate every entry against the schema in CI; an entry that fails does not exist as far as the site is concerned. Add the cross-file checks the schema cannot express: unique slugs, no dangling `parent` or `relations` targets, filenames matching slugs, and a wordlist gate for names that must never appear (children, anyone the owner names), with the wordlist kept outside the repository. ## Rules learned the hard way Each of these cost real time in the original dig. Most are about overclaiming, the failure the method exists to prevent. 1. **Ask what the entry is before deciding what to fetch.** A component of someone else's production is a small entry. The first pass on one website built for a theatre company's touring show returned the company's tour dates, reviews, staff page and job adverts, and produced an entry that read as a major theatre credit. Trimmed to the website and two press items. 2. **A near-miss is not a smaller version of the claim.** The first review relabelled those tour rows rather than deleting them, and a reader still saw nine venues under the subject's name. Half-measures on an overclaim are still overclaims. Remove it. 3. **A statutory register is a source for the company, never a roster of people.** See step 6. One dig filed every officer of a company as a collaborator, full legal names included, and the notes rendered on the page. Two of them had never published their involvement anywhere. 4. **Ownership is checked against content, never assumed from a name.** See step 2. 5. **Republished is not authored.** See step 5. 6. **Platforms are recorded as concepts, not contents.** See step 4. 7. **Nothing confidential, by omission.** See "Before you start". 8. **Recollection is a lead, not a verdict.** Two of the domain verdicts in the original dig were settled by reading the archive rather than by memory, in both directions. 9. **The provenance rules apply to your own summaries.** Late in the original dig, a working note claimed a platform had hosted two well-known names. It had not; the names came from a rough search that matched capitalised words after "by", and they were people mentioned in body text, not authors. The extraction runs, held to written rules, made no such error; the narration around them did. Treat an exploratory command's output as a lead. Re-derive a number before repeating it. And when a summary is corrected, write the correction where the next session will read it; silent fixes teach nobody. 10. **Self-host the evidence.** The archive is provenance, not infrastructure. Mirror screengrabs of dead pages into the site's own static files, and prefer a snapshot link over a dead live link. Never ship a 404. ## People - Name nobody who has not published their own involvement. Public organisations may be named freely. - Never record children's names, anywhere, in any field. - Never carry a register's name format into an entry. - Never record a home address, even as a registered office. - When in doubt, describe the role ("a collaborator", "the company's founders") rather than the person. ## Checklist before anything publishes - Every entry validates against the schema and the cross-file checks. - Every entry has at least one source with a URL; every source that can have a snapshot has one. - Every hedged date renders hedged. - Every `needs-owner` entry is hidden from the surfaces that speak for the subject now, and marked visibly everywhere else. - No name appears that the person did not publish about themselves. - Nothing from the confidential list appears anywhere, including notes and history. - The redirect map covers every historical URL cited as a source, and the live routes are reserved. - Every number quoted on a page is computed from the data at build time, not typed in.