Without getting too deep into details, we found the shape that held up was this. Keep acquisition separate from analysis, keep one stored source of truth, and make every later chart, quote packet, and category read trace back to that source.
The campaign made a good test because I knew it well enough to distrust easy answers. If the tool exaggerated something, missed a window, made a category too clean, or flattened the public language, I had enough prior context to know where to recheck the rows.
The Design Was Simple On Purpose
I built around one main idea: store the public posts once in a durable place, then do as much follow-on analysis as possible without pulling from the platform again.
I kept the live authority in a VM-backed DuckDB lane instead of letting it live only on my laptop.
The VM-backed store gives the project one authoritative database. It lets us run stored-data reads repeatedly without turning each analytic question into another paid API call. It also keeps the live pull separate from category review and later interpretation. The operational value is that a bug in a chart, quote packet, or category read sends us back to the tool and stored rows, not into a one-off hand edit.
The live worker lane handles collection. The local machine handles review and publication prep. Categorization sits between them: first as structured fields we can query, then as text surfaces we can read directly. That split mattered more than I expected. It meant a category problem could be reviewed as a category problem instead of being buried inside the collection job or hidden in the final prose.
I did not want live pulls to become the workflow
The local snapshot pattern came from that discipline.
Instead of hitting X every time I wanted a new chart, quote packet, or language read, I could pull a fresh worker snapshot locally. Then I could run the post-analysis tools against stored DuckDB data. The budget stays under control, and the analytic work stays reproducible. Just as important, the corpus stops changing underneath the question while I am trying to read it.
In practice, the flow is:
- Count and scope the topic first.
- Run the governed live job when a pull is justified.
- Store the results in the worker-backed DuckDB authority.
- Snapshot the database locally when the next job is analysis rather than acquisition.
- Generate dashboards, quote packets, corpus outputs, and modeling surfaces from stored data.
We found this method worked best: do not let every interesting question become a reason to spend money again. Pull when the question deserves it, then keep learning from the stored record.
Small Tools Kept The Work Legible
I kept local scripts and CLI surfaces for each layer of the job.
Governed collection has its own layer. Post-analysis has its own layer. Quote packets get their own lane. Corpus growth and modeling stay separate from publication.
One monolithic tool usually hides too much. The smaller-tool path made each artifact easier to inspect:
- Build the topic dashboard.
- Build the quote packet.
- Build the client-scoped corpus package.
- Build the sparse modeling surface.
- Build the public-facing charts from the stored sidecars.
When something looks wrong, the fix is narrower: review the tool that produced that chart, quote packet, or category surface; correct it; rerun that layer against the stored data. You do not have to review every moving part just to fix one bug.
The Text Stack Stayed Boring On Purpose
For text analysis, I leaned on tools that are strong enough to be useful and common enough that another researcher could reproduce something similar.
The stack I leaned on included:
- spaCy for structured text enrichment.
- Gensim for phrase-candidate growth.
- NLTK-backed support utilities.
- VADER and TextBlob as baseline sentiment audit layers.
- Sparse matrix exports for downstream GLM-style modeling rather than jumping immediately to heavier stacks.
The stack had to be boring enough that I could audit it.
Heavier methods can come later. I did not want an embedding or modeling layer to cover for weak categorization. Before adding complexity, we pushed the simple surfaces hard: counts, grouped windows, phrase surfaces, quote review, and sparse baselines. If those did not explain the pattern, then heavier methods would have a reason to enter.
I was not asking AI to judge my prose. The AI step came after the posts had already been pulled into the store and categorized. I used it as one review layer on the text and categories, not as proof of what people believed and not as a substitute for reading the rows. When an AI read said a category looked coherent, the next step was to check the underlying examples and see whether the label was doing real work.
The limit I kept was simple. AI can help summarize a bucket, compare language across buckets, and surface rows worth reading. It cannot decide that X proved persuasion, and it cannot rescue a bad category design. The stored data and the human read still have to carry the claim.
The Discipline Was Restraint
The discipline behind this project was restraint more than code. Every new pull has two costs: the obvious API cost, and the quieter cost of changing the evidence base while the interpretation is still moving.
The hardest habit was knowing when not to pull again.
Treat the neutral bucket as a review surface, not a trash can. Mixed, unclear, or structured disagreement can hide there.
Keep broad conversation and amplification from collapsing into the same analytic job.
Let X tell you about narrative timing, public language, and attention concentration without asking it to do the same job as a weekly population tracker.
A familiar topic gives the tool a built-in quality check. The wrong windows stand out. Implausible language stands out. You know when the tool is teaching you something new and when it is merely flattering itself.
By the end, the lane had more boring artifacts than I expected: manifests, sidecars, prompt notes, category files, and health checks. Those were not ceremony. They were how I kept the final read from floating away from the rows.
What I Would Reuse
If we were building this lane again for another topic, I would reuse the same checkpoints.
Start with a question that justifies the data pull.
Store what you pulled in one authority that you can query later.
Build local post-analysis tools so you can learn more without reopening the tap every time.
Use AI read only after the text has been pulled into the store and connected back to examples through categories.
Keep the text stack understandable enough that you can tell another person what each layer is doing.
At publication time, generate the charts from the stored sidecars. Keep the prompts and provenance close enough that the publication process does not become a second, unreviewed analysis step.
The useful part is the checkability: when a number, category, or sentence looks wrong, the work gives you a way back to the rows.