123 points by rustacean_ron 6 months ago flag hide 10 comments
john_carmack 6 months ago next
Fantastic work! I've been following the development of your project since its inception and it' just great to see such a decentralized SMP built in Rust. I have a few questions about how you approached censorship resistance on your network.
gabriel_luci 6 months ago next
Hey @john_carmack! We borrowed some concepts from IPFS, implemented a trustless content addressing system which eliminates the single point of failure. Also, our load balancing is based on DHT k-bucket paradigm, and content is replicated across nodes through Kademlia RPC requests. This pretty much eliminates the possibility of content censorship.
evan_you 6 months ago prev next
@gabriel_luci I'm thrilled to see DHT and Kademlia implementations in Rust. Can you elaborate more on the challenges faced in converting these algorithms to Rust?
gabriel_luci 6 months ago next
Sure @evan_you! Most of the problems stemmed from Rust's ownership and borrowing model. We had to use lifetimes, references, and smart pointers extensively to maintain the DHT and Kademlia structures during the routing table updates process. Making the RPC requests and transport secure over TLS was another hurdle we had to overcome. Ultimately, Rust's strict type checking helped ensure the integrity of our system, causing fewer headaches during runtime.
brendan_eich 6 months ago prev next
Nice work on the performance and scalability aspects of the system. Curious about the strategies you adopted for achieving consistency and high availability under churn?
jane_q 6 months ago next
@brendan_eich The core idea is to maintain an optimistic concurrency control (OCC) model for transactions. We took inspiration from Calvin-based protocols, which involve having a strong eventual consistency model using a Paxos-like consensus algorithm in the background. Our system pipelines the gossip protocol messages, ensuring nodes stay in sync with the latest state, consequently improving availability under churn.
vitalik_buterin 6 months ago prev next
@jane_q Interesting choice of using OCC for the transaction model! In the context of decentralized social media, is there an approach to incentivize more nodes to join and handle user transactions as validators, other than proof-of-work or proof-of-stake?
codehappiness 6 months ago next
@vitalik_buterin Yes, actually! We applied a proof-of-replica strategy, where nodes are incentivized based on how many replicas they manage to host accurately and consistently within the cluster. This way, the more copies they maintain of user content, the more they are rewarded, ensuring the system has better integrity and resistance to data loss.
fireinthevalley 6 months ago prev next
I noticed you are using Rust's native WebSocket library for real-time communications between nodes, how is the latency? What other communication optimizations did you employ?
frank_he 6 months ago next
@fireinTheValley We noticed a latency of about 50-100ms in most use cases. To improve on this, we used asymmetric communication channels leveraging a push-based model for high-priority data and a pull-based model otherwise. We also batched lower-priority messages based on priority queues and piggybacked them on high-priority data packets when necessary.