150 points by edgenetworkdudes 6 months ago flag hide 27 comments
svartalf 6 months ago next
Great article! I've been curious about building a CDN with Rust, but didn't know where to start. This gives me some solid ideas.
rwtodd 6 months ago next
I've been working on a similar project recently. The challenge isn't just creating the CDN nodes, but also managing the distribution of data and scaling. Trying to coordinate all of that without a central authority can be tricky.
athas 6 months ago next
We've experimented with self-organizing maps to handle the coordination of edge nodes. Results have been promising so far.
athas 6 months ago next
It is a challenge, but the beauty of decentralization is that you distribute the workload. You don't have a single point of coordination that becomes a bottleneck.
seanmonstar 6 months ago prev next
Just be aware of the added latency you might encounter from a completely decentralized system. Depending on the use case, the benefits may not outweigh the costs.
svartalf 6 months ago next
True, that's a valid concern. However, as long as you architect it properly, you should be able to maintain a responsive and efficient CDN.
hugsvic 6 months ago prev next
Have you considered using WebRTC and taking advantage of established P2P connections?
rwtodd 6 months ago next
WebRTC might be a great choice for real-time content delivery, but for something more general, I think we still need HTTP-based communication channels.
sir_banana 6 months ago prev next
Why not use an existing decentralized storage system like IPFS or Filecoin?
nebur 6 months ago next
IPFS and Filecoin are too generalized. They don't necessarily provide optimal solutions for CDN, particularly for building edge networks.
onetwoseven 6 months ago prev next
IPFS is designed for immutable data, so it's not the best option for a CDN, which typically deals with dynamic data.
sir_banana 6 months ago next
Good point, I hadn't thought of it that way. Thanks!
femtomachines 6 months ago prev next
Have you done any comparisons between Rust and other languages like Go or Elixir for this type of project?
nebur 6 months ago next
We have done some limited comparisons, and Rust outperforms Go and Elixir in terms of memory usage and throughput.
femtomachines 6 months ago next
Rust's performance is impressive, but how much is the learning curve impacting your development speed?
nebur 6 months ago next
You'll definitely be slower upfront when learning Rust, but the overall gains in the long term can offset this.
borkman 6 months ago prev next
It'd be interesting to see how this scales globally compared to AWS CloudFront or similar services.
rwtodd 6 months ago next
Absolutely. However, our initial focus is to validate the concept and ensure we're meeting the performance needs for our targeted use cases.
hyperknow 6 months ago prev next
Enterprise CDNs still have the advantage of dedicated hardware and networks, which kicks many DIY solutions to the curb. Thoughts?
svartalf 6 months ago next
True, and it's a trade-off you have to consider. Flexibility, cost savings, and potentially better responsiveness to edge changes can help offset that, though.
zxcat 6 months ago prev next
Security is a huge concern for any distributed infrastructure. How are you addressing that in your project?
hugsvic 6 months ago next
Our approach has been to leverage Rust's built-in language features for memory safety, combine it with a flexible permission model, and then continuously audit our endpoints using tools like `rust-audit` and third-party services.
yipyapper 6 months ago prev next
How does this work for combining content from multiple sources? Is there a routing layer on top or does the user have to specify multiple nodes on their side?
nebur 6 months ago next
We're working on a routing layer that should handle this type of situation. Our current prototype relies on a client-side approach, but we hope to improve that in the future.
jvb 6 months ago prev next
This all sounds interesting. I'd love to learn more about monitoring and debugging a decentralized system like this. Any resources you'd recommend?
rwtodd 6 months ago next
While we don't have a canonical answer for that specific issue, a combination of log scraping, instrumentation, and proactive load testing should be a good start. I'm sure the community would be happy to help you further if you join our gitter channel.
carpetfizz 6 months ago prev next
Looks like an easy +10 for the HN show Hacker News.