1 point by scraperdude 7 months ago flag hide 15 comments
scraper-builder 7 months ago next
Asking for recommendations on the best libraries for building multi-threaded web scrapers. I want to build something that can handle multiple request and parse data efficiently. What do you recommend?
efficient-scraper 7 months ago next
I would recommend Scrapy, a powerful python library for web scraping. It has built-in support for handling concurrent requests, and you can write your own multi-threaded spiders as well.
scraper-builder 7 months ago next
Thanks, I've heard of Scrapy, but I were wondering are there any other libraries I should consider?
go-scraper 7 months ago next
If you're willing to learnGo, I would check out Colly, which is a fast and efficient web scraping library for Go that also supports multi-threading.
scraper-builder 7 months ago next
@go-scraper That's an interesting idea, I haven't worked with Go before, but it seems to be a up and coming language. How long did it take you to get comfortable with Colly and Go?
go-scraper 7 months ago next
It took me a few days to get familiar with Go and Colly, but after that, I found it quite easy and intuitive to use. I would recommend giving it a try.
parallel-crawler 7 months ago prev next
Another option is Scrapy-Crawlera, which is a Scrapy plugin that enables more robust and efficient crawling by integrating with the Crawlera service.
scraper-builder 7 months ago next
@parallel-crawler Thanks for the suggestion. What are the advantages of Scrapy-Crawlera over the built-in Scrapy?
parallel-crawler 7 months ago next
Crawlera takes care of a number of common problems when scraping, such as automatically resolving captchas, handling javascript rendering, and dealing with IP blocking. It can be a big time-saver.
been-there 7 months ago prev next
I wrote my own multi-threaded web scraper in node.js using the async library and the request package. It works pretty well and I rarely have any problems.
scraper-builder 7 months ago next
@been-there I'll keep an eye on node.js, do you have any advice for someone starting out with multi-threading and web scraping?
been-there 7 months ago next
Definitely ease into it. Start with single threading and simple scraping, then gradually introduce more advanced topics like multithreading as you gain more experience and understand the nuances. Good luck!
regular-expression-scraper 7 months ago prev next
For smaller projects, I like to use regular expressions to both make requests to a website and parse the results. It may not be as fast as dedicated libraries, but it's very flexible.
scrapy-veteran 7 months ago prev next
Scrapy has the advantage of a large community, which means you can easily find support and resources. It's also been actively maintained for many years.
scraper-builder 7 months ago next
@scrapy-veteran That's great to hear, I think I'll start with Scrapy then.