89 points by rustaceanprogrammer 1 year ago flag hide 21 comments
s0ftw4r3 1 year ago next
Have you thought about integrating your crawler into existing frameworks like Stream data? The combination could offer a unique value proposition.
creator 1 year ago next
I'm excited by the idea of integrating our crawler with Stream data! I'll see if I can find time to develop a proof-of-concept and documented it for the community.
rand0muser 1 year ago prev next
Great work! Love to see Rust being used in the browser.
creator 1 year ago next
Thanks rand0muser! Let me know if you have any questions on how the real-time crawling works.
n0ss14c 1 year ago prev next
@rand0muser, There are a couple typos in your post. It should be that the crawling is happening 'live' not 'live'.
creator 1 year ago next
Thank you for pointing it out! I've fixed the typo. I'm impressed that you noticed it.
w1z4rd 1 year ago prev next
Interesting approach, streaming the results with WebAssembly. What were some of the challenges?
creator 1 year ago next
The main challenge was dealing with memory management. Having to manually manage memory in Rust via the alloc crate was intimidating at first, but after some practice, it started to make sense.
h4ck3rm4n 1 year ago prev next
Can you post the open source link? I'd like to see the internals of this project.
creator 1 year ago next
Sure! Here's the GitHub link: https://github.com/yourgithubusername/realtime-crawler
ther0ckst4r 1 year ago prev next
A real-time crawler is exciting and can have many use-cases. We're considering building a product that utilizes fetching real-time stock data. Would this be similar to that?
creator 1 year ago next
@ther0ckst4r, that's perfect. Our project can be adapted for real-time data fetching. I'll follow up and you can let me know if you have any questions.
m4st3r 1 year ago prev next
How much of a performance boost did you get going with Rust and WebAssembly instead of a standard web-based language?
creator 1 year ago next
For this specific project, I noticed about a 20% performance improvement. However, if you're looking for low-level and concurrent programming, Rust with WebAssembly would provide more benefits.
m4st3r 1 year ago next
Thanks for your elaborate response. I'm going to take a closer look at how the stack works within the project.
k33n 1 year ago prev next
From the URL, I'm assuming the main focus is on web crawling. Would this library also be suitable for real-time content pipelines like Tweeter?
creator 1 year ago next
@k33n, Yes! Given enough system resources, you could adapt the project for other real-time data feeds, like an event-driven architecture for a Twitter bot.
ph4nt0m 1 year ago prev next
My main concern with WebAssembly-based apps is the initial load time overhead. Do you have metrics to share about user experience?
creator 1 year ago next
That's valid feedback. Our users did notice a delay the first time loading, but we're using service workers to cache the build to improve subsequent loads.
h4x0r 1 year ago prev next
Did you look into using AssemblyScript? It's similar to TypeScript and compiles to WebAssembly, too.
creator 1 year ago next
#h4x0r I'll take a look at AssemblyScript. Thanks for the recommendation! From my brief review, the syntax appears to be similar to Rust's.