450 points by scaling_genius 1 year ago flag hide 24 comments
johnsmith 1 year ago next
This is an interesting read, we've been looking to scale our ML infra recently too. Very informative!
jane 1 year ago next
I agree, the section about load balancing and using multiple models in parallel was particularly useful.
charlie_ml 1 year ago prev next
Scaling infrastructure is a critical step for any ML team, great work.
jen_dataengineer 1 year ago next
I second that, scalability is the key to a successful ML project.
std_machinelearning 1 year ago next
Well-said, a successful ML project requires a strong infrastructure foundation.
ann 1 year ago prev next
@johnsmith I'm curious to know more about how you managed to balance your infrastructure, could you elaborate?
sara 1 year ago prev next
Impressive! I'd love to hear more about the hardware and network requirements to sustaining such a large number of requests.
johnsmith 1 year ago next
Sure! We used Kubernetes to manage our containers and a custom load balancer to distribute requests evenly. This allowed us to horizontally scale and meet demand during traffic spikes.
ann 1 year ago next
Interesting, I'll look into Kubernetes as a possible solution for our scaling issues as well. Thanks for the suggestion!
julia 1 year ago prev next
Great post, I'm looking to implement similar scaling techniques in my own projects. Thanks for sharing!
jennifer 1 year ago prev next
Fascinating read, I'm working on a similar ML infrastructure and will definitely take a closer look at your implementation details.
johnsmith 1 year ago next
@jennifer We've open-sourced a significant portion of our infrastructure code on Github along with documentation, hope this helps!
avinash 1 year ago prev next
Impressive work, we've been struggling to handle over 10k requests/day. I'd love to hear more about your data pipelines and how you handle ETL.
deepak_etl 1 year ago next
@avinash we use Apache Kafka for streaming data and Apache Beam for ETL. It allows us to process high volumes of data in real-time.
brock 1 year ago next
Interesting, I'm looking to learn more about Kafka and Beam. How easy was the implementation and config for them?
deepak_etl 1 year ago next
It took us a few days to fully set up and test, but once it was running, it was very stable and reliable. Just be prepared to spend some time upfront getting it configured properly.
wilson_mlops 1 year ago next
I've heard great things about Apache Kafka and Apache Beam for ETL. How do you approach monitoring and logging for such a complex system?
deepak_etl 1 year ago next
We use a combination of monitoring tools, including Prometheus for monitoring metrics and Grafana for visualizing those metrics. For logging, we use the ELK stack: Elasticsearch, Logstash, and Kibana. Do you have any suggestions for additional monitoring options?
wilson_mlops 1 year ago next
That's a great setup. We use similar tools, and also find them to be helpful for ensuring our infrastructure's health. It's always good to have a robust monitoring and logging system in place.
harry 1 year ago next
Indeed, scalability and resilience are essential aspects of modern machine learning infrastructure. Keep up the great work, johnsmith!
frank 1 year ago prev next
I agree, Kafka and Beam can be complex initially, but once they're up and running, they're quite powerful. Thanks for the write-up!
mason 1 year ago prev next
Kafka and Beam are great tools, but they can be challenging to set up initially. I suggest checking out their documentation and tutorials for guidance.
john 1 year ago next
@mason Beam can be a bit of a puzzle at times, but I think the documentation does a good job explaining the various features and options available. Definitely worth the time investment.
benjamin 1 year ago prev next
I've found that Kafka and Beam integrate quite well, allowing for simple and efficient data transformation. Well-played!