Every lead we ever scraped or emailed — for every client, past and present — merged into ONE Google Sheet, one row per person, with full who-used-it-when provenance. · 2026-07-09
Every lead we ever scraped or emailed — for every client, past and present — merged into ONE Google Sheet, one row per person, with full who-used-it-when provenance.
Master summary — the gist in 30 seconds
TL;DRWe build the master leads database as one Google Sheet (not Notion), in 4 stages — find every lead sheet, extract + normalize, dedup into one-row-per-person, then a gated batched import. Two human gates: you confirm the archived-client list, and nothing is imported until you sign off on the staging CSV.
Inputs: every internal MAIN + client-facing lead sheet across 6 hub clients + archived projects (~30–50k raw rows), plus Instantly send records. Output: one 'Master Leads' Google Sheet, deduped to unique persons, each row saying which clients used them and roughly when (month-level, with the guess-source recorded).
Why this mattersRight now the same person can be scraped and emailed twice by two different clients and nobody would know. One deduped master DB makes lead reuse, cross-client suppression, and 'have we touched this company before?' a 5-second filter instead of an archaeology dig.
flowchart LR
A[Discovery\nall lead sheets] --> B[Extract + normalize\nprovenance kept]
B --> C[Dedup + merge\n1 row per person]
C --> D[staging.csv]
D -->|OWNER sign-off| E[Batched import\nMaster Google Sheet]
I[Instantly API] --> C
Discovery & inventory (A1–A3)
TL;DRA script builds a machine-readable list of every lead sheet — hub WORKSHEETS.md files, Drive on both accounts, old project folders, and sheet links found in sent mail.
In: the six clients' documented sheets + a sweep for forgotten ones. Out: inventory.json — every sheet tagged internal / client-facing / candidate, included or excluded with a reason.
Why it mattersThe handoff's list is a floor, not a ceiling — archived projects hold leads too. You personally confirm which archived candidates count (human gate #1), and inbound website-form sheets are excluded by rule so end-client enquiries never pollute the outbound DB.
flowchart LR
W[WORKSHEETS.md x6] --> I[inventory.json]
D[Drive sweep\nboth accounts] --> I
M[Sent mail\nsheet links + dates] --> I
I -->|candidates| O{Owner confirms}
Extract & normalize (B1–B3)
TL;DRPull every included tab into one raw file that remembers exactly where each row came from, then translate the chaos of column names into one canonical schema.
In: heterogeneous tabs (Név/Name/full_name, E-mail/email…). Out: normalized rows — clean lowercase emails, normalized domains, email-less leads kept and flagged, and a month-level 'used' date per row with its basis recorded (dated tab name → campaign launch → share-email date → sheet modified).
Why it mattersProvenance and the date-basis field are what make the DB auditable — every approximate date can be traced back to why we believe it.
TL;DRSame person for two clients becomes ONE row: clients unioned, one date per client, best-verified email wins. Instantly's send records — not our notes — decide who was actually emailed.
In: normalized rows. Key: verified email; if none, domain+name. Out: master.jsonl, one row per unique person, Contacted = Emailed / Scraped-only straight from the live Instantly API.
Why it mattersThis is the whole point of the DB — the one-row-per-person grain. And trusting only Instantly for contacted-status is the lesson of the K-X stale-notes incident.
flowchart LR
N[normalized rows] --> K{key:\nemail else\ndomain+name} --> M[1 row/person\nclients unioned]
API[Instantly API] --> M
Staging & gated import (D1–D4)
TL;DREverything lands in a staging CSV you review first. The importer physically refuses to run without your sign-off token; then it writes the master sheet in 5,000-row batches with crash-safe resume.
In: master.jsonl. Out: staging.csv + summary stats → your sign-off → the Master Leads Google Sheet, verified by reading rows back and comparing counts + a 20-row spot check.
Why it mattersTwo safety nets: no bulk import can ever happen behind your back (exit-4 guard), and batching + resume means 50k rows import in minutes without quota stalls or double-writes.
TL;DRGoogle Sheet, not Notion (your override — Notion's API is too slow at 50k rows and Sheets is your daily tool). Dedup = verified email first, domain+name fallback; email-less leads kept flagged; month-level dates only.
Recorded in plans/master-leads-sheet/docs/adr/0001 + 0002 so future sessions don't relitigate.
Why it mattersThe one accepted risk: domain+name fallback can occasionally merge two same-named people at one company — flagged rather than silently split, so it's reviewable in staging.
How it gets verified (master-leads-qa)
TL;DRA project QA skill with 14 check-IDs (A1…T2) runs each stage against synthetic fixtures — every check must first FAIL red before the stage exists, then go green.
Run: python3 .claude/skills/master-leads-qa/scripts/assert.py --check (or --all). Done = every Tier-1 check prints PASS and both Tier-2 confirms (spot-check + no-secrets/no-excluded-leaks) hold.
Why it mattersThe builder can never tick a box on reasoning alone — the same discipline that caught the reply-ops and name-guard bugs.