1 point by web_scraping_fan 7 months ago flag hide 8 comments
scrapersam 7 months ago next
My favorite tool for web scraping is BeautifulSoup with Python. It's simple and easy to use for parsing HTML. I've also used Selenium to interact with websites that utilize JavaScript.
pythonpro 7 months ago next
I agree, BeautifulSoup and Python make a great combo. But don't forget about Scrapy, it's also a powerful Python framework for web scraping with built-in support for handling AJAX requests.
javajive 7 months ago prev next
For Java-based scraping, I suggest using Jsoup or HtmlUnit, which allow for programmatic control over web content and support for JavaScript.
scrapersam 7 months ago next
JavaJive, have you tried Selenide with Java? I've heard it's a great solution for dealing with web scraping tasks that heavily involve JavaScript.
javajive 7 months ago next
ScraperSam, I haven't tried Selenide yet, but it's on my to-do list. I've heard great things about it as well! Thanks for the suggestion.
rustranger 7 months ago prev next
If you're working in Rust, I recommend working with the Scraper crate. It's an efficient and user-friendly library for web scraping.
parsingpete 7 months ago next
RustRanger, I've used the Scraper crate, and it's fantastic. What are your thoughts on Reqwest for handling API requests?
rustranger 7 months ago next
parsingPete, I've had very good experiences with Reqwest. It handles get requests with ease, and the users on the forums are always eager to help.