234 points by data_ninja 1 year ago flag hide 16 comments
user1 1 year ago next
This is an interesting project! Using machine learning to scrape real-time data is an innovative approach.
user2 1 year ago next
I agree! I'm curious about the specific machine learning algorithms being used here. Can you provide more details?
user3 1 year ago prev next
This project reminds me of web scraping bots that extract data to build trading algorithms. Do you think this tool could be used for that purpose as well?
user1 1 year ago next
Yes, you're right. With real-time machine learning capabilities, this tool could potentially be used for high-frequency trading algorithms. However, I think it's important to note that any use of bots for financial transactions needs to be closely monitored to ensure ethics and compliance.
user4 1 year ago prev next
What framework are you using for the ML component? I'm guessing something like TensorFlow, but I'd be interested to know if you're using anything else.
user1 1 year ago next
Great guess! Yes, we are using TensorFlow for the ML component, along with some custom scripts to handle the real-time aspect.
user5 1 year ago prev next
What kind of data are you scraping? Is it structured or unstructured data?
user1 1 year ago next
Both! This tool can handle structured data from tables and APIs, as well as unstructured data from webpages and PDFs.
user6 1 year ago prev next
Is this project open source? I'd love to take a look at the code and possibly contribute.
user1 1 year ago next
Yes, the project is open source. You can find the code and documentation at [github.com/user1/realtime-webscraper-ml](http://github.com/user1/realtime-webscraper-ml)