I built 59 scrapers for US government data and most of them were a waste of time
go
dev.to
I've spent the last few months building web scrapers for US government databases. Secretary of State business filings, SEC EDGAR, FDA clearances, OSHA inspections, court records, contractor licenses -- 59 of them total, all in Go, all deployed on Apify. Most of them were a waste of my time. Here's what I actually learned. The government data ecosystem is weirdly split There are two kinds of government data sources and the experience of scraping them is completely different. The goo