67 points by webcrawlguy 7 months ago flag hide 14 comments
john_doe 7 months ago next
I'm looking to build a web crawler that can handle a large number of requests and don't know where to start. Any advice?
crawler_expert 7 months ago next
First, consider using a language with asynchronous capabilities like Python or JavaScript. They'll help you make a larger number of concurrent requests. Also, look into using a distributed crawl setup with tools like Scrapy or Selenium.
crawler_expert 7 months ago next
@web_dev, yes, managing the rate of requests is crucial, especially for larger crawls. Using a queue will help distribute the requests over a longer period of time and avoid unnecessary blocking and latency.
crawler_expert 7 months ago next
@john_doe, I personally recommend checking out the docs for AWS SQS and Google Cloud Pub/Sub. They provide a lot of good info on how to set up and use their services. And, there are tons of tutorials on YouTube and Medium to help you get started.
john_doe 7 months ago next
@crawler_expert, I'll check them out. Thanks! @web_dev, I'll check them out too. These resources look great. Much appreciated.
web_dev 7 months ago prev next
You could also use a queue management system to keep track of your crawling and avoid overloading websites. Consider using a managed queue service like AWS Simple Queue Service (SQS) or Google Cloud Pub/Sub.
john_doe 7 months ago next
@web_dev, how would I go about setting up a managed queue? Do you have any tutorials or resources to recommend?
web_dev 7 months ago next
@john_doe, here's a tutorial on how to use SQS with Python: <https://docs.aws.amazon.com/sdk-for-python/v1/developer-guide/sqs-examples.html> And for Google Cloud Pub/Sub, take a look here: <https://cloud.google.com/python/docs/tutorials/pub-sub>
another_user 7 months ago prev next
If you're looking to build a really large scale web crawler, you might want to consider using Apache Hadoop for distributed processing. It's highly scalable and can support tens of thousands of nodes.
john_doe 7 months ago next
@another_user, thanks for the advice. I'll look into Hadoop. It seems pretty powerful.
crawler_fan 7 months ago prev next
I used Apache Nutch for a project last year and it worked well. it's easy to configure and includes a great set of features. recommend it.
john_doe 7 months ago next
@crawler_fan, awesome, glad to hear it worked for you. I'm looking forward to trying it out myself.
apache_crawl 7 months ago prev next
For high volume crawling, consider using Apache Nutch, built on top of Hadoop. It's specifically designed for web crawling and includes features for site scheduling, duplicate removal, and content depth control.
john_doe 7 months ago next
@apache_crawl, this looks interesting. I'll definitely take a look at Apache Nutch. Thanks for the recommendation.