3 points by rustfan 7 months ago flag hide 14 comments
swizec 7 months ago next
Has anyone built a distributed task queue in Rust? I'm working on a project and considering Rust, but I'm worried about its ecosystem and library support for building something like this.
anonymouscoward 7 months ago next
I don't know if anyone has specifically built a distributed task queue in Rust, but I do know that Rust's async/await feature is quite powerful and can be used to build such a system. Also, the Tokio library provides excellent support for asynchronous I/O.
bildsten 7 months ago next
Another alternative for building a distributed task queue in Rust is the `crossbeam` library. It provides low-level primitives for concurrency and is ideal for building highly concurrent systems.
chrismorgan 7 months ago prev next
Yes, I have built a distributed task queue in Rust called `mk-queues`. It is open-source and available on GitHub. It uses the Tokio library and is highly reliable and scalable.
swizec 7 months ago next
Thanks for sharing, @chrismorgan! I'll definitely take a look. What was your experience like building something like this in Rust compared to other languages like Python or Java?
chrismorgan 7 months ago next
Rust's strong type system and ownership model made it easier to catch bugs and ensure thread safety than in Java. Also, Rust's macro system allowed for more concise code than in Python or Java. However, the learning curve was quite steep, and it took a while to get used to Rust's unique idioms and syntax.
anonymouscoward 7 months ago next
Is `mk-queues` production-ready, @chrismorgan? How do you handle failover and load balancing?
chrismorgan 7 months ago next
Yes, `mk-queues` is production-ready and has been used in several large-scale services. It uses the Raft consensus algorithm for leader election and replication, which ensures high availability and consistency. Additionally, it can be easily integrated with any load balancer such as NGINX or HAProxy.
chrismorgan 7 months ago next
Another approach is to use a distributed message queue like Redis' Pub/Sub feature. It provides a lightweight solution for distributing tasks and is quite popular in the Rust community.
swizec 7 months ago next
Thanks for the suggestions, everyone! I have a lot of research to do before making a decision.
rustacean 7 months ago prev next
Another consideration is Erlang's OTP framework and its distributed message passing capabilities. It has excellent support for building fault-tolerant distributed systems, and it's worth looking into if you're considering Rust.
tim_gray 7 months ago prev next
Have you considered using a message broker like Apache Kafka or RabbitMQ instead of building a custom distributed task queue? It can save you a lot of headaches and provide better fault tolerance.
swizec 7 months ago next
Thanks for the suggestion, @tim_gray! I'll definitely consider using a message broker instead. It's great to know that Rust has good support for integrating with external systems like Kafka or RabbitMQ.
anonymouscoward 7 months ago next
It's definitely worth looking into message brokers, but you may want to consider building a distributed task queue if you have specific performance or consistency requirements that a message broker can't meet.