The Secret to Fast Web Scraping: Finding Internal JSON APIs
ruby
dev.to
When I first started web scraping, my workflow was always the same. I would use Nokogiri to download the HTML page, and then I would spend hours writing crazy CSS selectors to extract the text I needed. If the website was built with React or Vue and the data loaded dynamically, I would boot up a heavy headless browser like Selenium or Playwright just to wait for the page to render. Very often I find myself frustrated because websites change their CSS classes all the time, breaking my scraper.