325 points by dataengineer 1 year ago flag hide 18 comments
john_doe 1 year ago next
Great article! I've been looking for something on real-time data processing for a while now.
jane_doe 1 year ago next
Wow, your system sounds amazing. I'm particularly interested in the algorithms you used for data distribution. Any more details you can share?
engineer_node 1 year ago next
We used a combination of consistent hashing and dynamic load balancing for data distribution. I can share more details on that if you're interested.
algo_geek 1 year ago next
Consistent hashing always seems like a good choice for horizontal scaling of data. Do you have any implementation specifics or references to resources to learn more?
code_warrior 1 year ago prev next
Yeah, that's one thing I've found with consistent hashing - sometimes it's hard to figure out the best way to implement it. Would be great to hear more about your approach.
gp_fan 1 year ago prev next
I've used dynamic load balancing before, but not together with consistent hashing. Looks really interesting. Mind sharing some performance numbers?
node_0 1 year ago next
Thanks for sharing the numbers. Inspiring to see how well it works with such a complex setup.
user-67 1 year ago prev next
Can you share how you measured the performance? Would be great to replicate these results for myself.
hacker_1 1 year ago prev next
Nice work! Concurrency is always a pain to deal with when building real-time systems. How did you manage it in your system?
programmer_gal 1 year ago next
We used the standard C10K pattern to manage concurrency. It's an old but still valid technique I believe.
engineer_4 1 year ago next
We modified the ketama implementation to better work with our varying load sizes. Worked out great for us.
developer_1 1 year ago prev next
Interesting. When we built our real-time system, we used a different approach for concurrency - event loops with asynchronous I/O. But I'd love to hear more about your technique!
learn_code 1 year ago next
Would you be willing to share a high level flow of how your system works?
curious_user 1 year ago prev next
Do you have any benchmarks comparing your system to others?