234 points by data_ninja 7 months ago flag hide 16 comments
user1 7 months ago next
This is an interesting project! Using machine learning to scrape real-time data is an innovative approach.
user2 7 months ago next
I agree! I'm curious about the specific machine learning algorithms being used here. Can you provide more details?
user3 7 months ago prev next
This project reminds me of web scraping bots that extract data to build trading algorithms. Do you think this tool could be used for that purpose as well?
user1 7 months ago next
Yes, you're right. With real-time machine learning capabilities, this tool could potentially be used for high-frequency trading algorithms. However, I think it's important to note that any use of bots for financial transactions needs to be closely monitored to ensure ethics and compliance.
user4 7 months ago prev next
What framework are you using for the ML component? I'm guessing something like TensorFlow, but I'd be interested to know if you're using anything else.
user1 7 months ago next
Great guess! Yes, we are using TensorFlow for the ML component, along with some custom scripts to handle the real-time aspect.
user5 7 months ago prev next
What kind of data are you scraping? Is it structured or unstructured data?
user1 7 months ago next
Both! This tool can handle structured data from tables and APIs, as well as unstructured data from webpages and PDFs.
user6 7 months ago prev next
Is this project open source? I'd love to take a look at the code and possibly contribute.
user1 7 months ago next
Yes, the project is open source. You can find the code and documentation at [github.com/user1/realtime-webscraper-ml](http://github.com/user1/realtime-webscraper-ml)