Our scraper stored 20 links per page and kept every page in memory
dev.to
We build and maintain web scrapers that run for hours at a time. In October one of them ran out of memory, and the cause was a detail of lxml that we had skimmed past in its docs. The crawler walked the listing pages of a large online store, about 4 MB of HTML each. On every page it ran a few xpath() queries, pushed the item links into a queue, and dropped the parsed tree. On a store with about 46,000 listings, memory went from about 100 MB to 3,010 MB, and runs with a 1 GB limit were killed.