356 points by scraping_genius 6 months ago flag hide 10 comments
theawesomeprogrammer 6 months ago next
This is really cool! I've been looking for a real-time web scraper with machine learning capabilities. Definitely going to give this a try. Thanks for sharing!
mlengineer 6 months ago next
The machine learning aspect is very interesting. Could you give some examples of how it can be used for web scraping?
theawesomeprogrammer 6 months ago next
Sure, for example, the web scraper could be trained to understand and extract specific data patterns, such as product information or reviews. It could also be used to detect changes in a website and notify the user in real-time. The possibilities are endless!
mlengineer 6 months ago next
That's fascinating! I would love to learn more about the specific machine learning algorithms you're using under the hood.
theawesomeprogrammer 6 months ago next
For the machine learning component, I'm using both TensorFlow and Keras. Keras is a high-level Neural Networks API that's built on top of TensorFlow. It's very user-friendly and allows you to quickly build and train complex networks.
opensourcejoan 6 months ago prev next
I suggest using TensorFlow or PyTorch for the machine learning component. They're both popular and well-documented libraries that should be easy to integrate with your project.
hackermaster 6 months ago prev next
I've been working on a similar project but I haven't been able to integrate machine learning capabilities yet. This is very impressive, will definitely check it out!
opensourcejoan 6 months ago next
I'd love to collaborate! I've been working on a machine learning library that could be a good fit for your project.
hackermaster 6 months ago next
That's a great idea, let's do it! I'm always looking to work with other developers on exciting projects.
mlengineer 6 months ago next
I agree, TensorFlow and Keras are powerful libraries that should be able to handle the job. I'm looking forward to collaborating with you on this project!