45 points by webscraper 1 year ago flag hide 10 comments
user1 1 year ago next
I recommend Scrapy with a distributed architecture using something like Celery. It's a powerful package for scraping and has great support for various types of extractions.
user2 1 year ago next
Scrapy is indeed great. Another alternative is BeautifulSoup, which you can use in combination with other tools like gevent for better concurrency.
user4 1 year ago next
gevent is a great choice for handling concurrent tasks in Python. I've used it in production successfully.
user8 1 year ago prev next
@user7 That's a fantastic idea. I haven't thought of using Selenium Grid with AWS Lambda. Will definitely give it a try!
user3 1 year ago prev next
I've had good experiences with Selenium for scraping. It's ideal for websites that rely heavily on JavaScript, although it can be slow.
user5 1 year ago next
Selenium is indeed a lifesaver when it comes to JavaScript-heavy sites. I would definitely keep it in your toolkit for more challenging scraping tasks.
user7 1 year ago next
Selenium Grid combined with AWS Lambda could be a powerful solution for JavaScript-ish websites.
user9 1 year ago next
I agree. I'm using this technique for running dynamic content scraping tasks for a project and it's quite efficient.
user6 1 year ago prev next
Awesome thread! While not a tool itself, I suggest looking into serverless architectures like AWS Lambda for running distributed web scraping tasks with minimal overhead.
user10 1 year ago next
I've also had great success using a Python Flask API to run tasks through a queue server (like RabbitMQ) that sits on top of multiple AWS Lambda instances.