89 points by rustaceanprogrammer 7 months ago flag hide 21 comments
s0ftw4r3 7 months ago next
Have you thought about integrating your crawler into existing frameworks like Stream data? The combination could offer a unique value proposition.
creator 7 months ago next
I'm excited by the idea of integrating our crawler with Stream data! I'll see if I can find time to develop a proof-of-concept and documented it for the community.
rand0muser 7 months ago prev next
Great work! Love to see Rust being used in the browser.
creator 7 months ago next
Thanks rand0muser! Let me know if you have any questions on how the real-time crawling works.
n0ss14c 7 months ago prev next
@rand0muser, There are a couple typos in your post. It should be that the crawling is happening 'live' not 'live'.
creator 7 months ago next
Thank you for pointing it out! I've fixed the typo. I'm impressed that you noticed it.
w1z4rd 7 months ago prev next
Interesting approach, streaming the results with WebAssembly. What were some of the challenges?
creator 7 months ago next
The main challenge was dealing with memory management. Having to manually manage memory in Rust via the alloc crate was intimidating at first, but after some practice, it started to make sense.
h4ck3rm4n 7 months ago prev next
Can you post the open source link? I'd like to see the internals of this project.
creator 7 months ago next
Sure! Here's the GitHub link: https://github.com/yourgithubusername/realtime-crawler
ther0ckst4r 7 months ago prev next
A real-time crawler is exciting and can have many use-cases. We're considering building a product that utilizes fetching real-time stock data. Would this be similar to that?
creator 7 months ago next
@ther0ckst4r, that's perfect. Our project can be adapted for real-time data fetching. I'll follow up and you can let me know if you have any questions.
m4st3r 7 months ago prev next
How much of a performance boost did you get going with Rust and WebAssembly instead of a standard web-based language?
creator 7 months ago next
For this specific project, I noticed about a 20% performance improvement. However, if you're looking for low-level and concurrent programming, Rust with WebAssembly would provide more benefits.
m4st3r 7 months ago next
Thanks for your elaborate response. I'm going to take a closer look at how the stack works within the project.
k33n 7 months ago prev next
From the URL, I'm assuming the main focus is on web crawling. Would this library also be suitable for real-time content pipelines like Tweeter?
creator 7 months ago next
@k33n, Yes! Given enough system resources, you could adapt the project for other real-time data feeds, like an event-driven architecture for a Twitter bot.
ph4nt0m 7 months ago prev next
My main concern with WebAssembly-based apps is the initial load time overhead. Do you have metrics to share about user experience?
creator 7 months ago next
That's valid feedback. Our users did notice a delay the first time loading, but we're using service workers to cache the build to improve subsequent loads.
h4x0r 7 months ago prev next
Did you look into using AssemblyScript? It's similar to TypeScript and compiles to WebAssembly, too.
creator 7 months ago next
#h4x0r I'll take a look at AssemblyScript. Thanks for the recommendation! From my brief review, the syntax appears to be similar to Rust's.