114 points by webscraper007 1 year ago flag hide 12 comments
johnsmith 1 year ago next
I'd recommend starting with BeautifulSoup and Python requests for web scraping. They're both simple and easy to learn.
newbiecoder 1 year ago next
Thanks for the suggestion! I've heard good things about BeautifulSoup. I'm looking for more advanced techniques though, like using Selenium or Scrapy for handling dynamic web pages.
janelee 1 year ago prev next
Scrapy is a great tool for web scraping, especially if you're comfortable with Python. It has built-in support for handling AJAX requests and can handle large scale projects.
savethedata 1 year ago next
Absolutely! Scrapy is perfect for large projects, but if you're just starting out, I'd recommend going with BeautifulSoup or even just using requests and regex. Don't overcomplicate things until you have to.
rookie_scraper 1 year ago prev next
What's the best way to handle JavaScript driven websites when using BeautifulSoup? I've heard Selenium can help with that, but I'm not sure if it's the best solution.
pythonista 1 year ago next
Selenium is a good option for handling JavaScript, but it's slower and can be more difficult to set up than other tools. You might also want to look into using tools like requests-html or Splinter, which are based on Selenium but are easier to use.
web_scraper 1 year ago prev next
When scraping websites, what's the best way to avoid being blocked? Is there a specific approach that works well or is it just a matter of being respectful and following robots.txt?
automated 1 year ago next
It's a bit of both. Following robots.txt and being respectful are important, but there are also some technical things you can do to avoid being blocked, like using a user agent, randomizing your requests, and using proxies. I've written a lot about this on my blog.
data_lover 1 year ago prev next
If you're just starting out, I'd recommend checking out the Scrapy tutorial on Real Python. It's a great resource for learning modern web scraping techniques.
newbie 1 year ago next
I second that recommendation. Real Python is a really helpful site for learning how to scrape websites. They have a lot of great resources for beginners.
rookie 1 year ago next
I'm new to web scraping and I'm feeling a bit overwhelmed. There are so many tools and techniques out there. Where's the best place to start?
japser 1 year ago next
Start with the basics. Learn how to make HTTP requests and parse HTML using tools like Python requests and BeautifulSoup. From there, you can start learning more advanced techniques like using Scrapy or Selenium. And don't forget to learn about web technologies like JavaScript and AJAX!