46 points by machine_learning_apprentice 1 year ago flag hide 15 comments
mlmaster 1 year ago next
Some thoughts on how to handle large-scale ML data pipelines? I'm finding it challenging to manage and process all my data.
bigdatabob 1 year ago next
Try Apache Beam or Spark for distributed processing of large datasets. They can help you handle and process big data efficiently.
parallelpete 1 year ago next
I agree with bigdatabob. I've successfully used Apache Beam for large-scale ML pipelines. The programming model is powerful and flexible.
mapreducemarvin 1 year ago next
Apache Beam is a convenient solution. But remember that you could also consider using traditional MapReduce if you prefer a simpler approach.
mesong 1 year ago next
This is especially true for large-scale pipelines. Decoupling the components allows you to scale and adjust resources as needed independently.
bionicbrain 1 year ago next
Kubeflow is a great solution. But be prepared for some hassles during installation, especially if you're setting it up on-premises. As a result, some prefer running it on cloud environments.
streamstransformer 1 year ago next
Pachyderm sounds powerful and is performing well for me too in terms of handling version control and pipeline management for ML projects.
anoopcnx 1 year ago prev next
It's important to decouple data pre-processing and actual model training. You could use a workflow manager for pre-processing and TensorFlow or PyTorch for model training.
pointerpaul 1 year ago next
Using Kubeflow with a hadoop cluster, you can build a convenient, modularized pipeline for ML with easy scalability and orchestration.
tensorflower 1 year ago prev next
I recommend taking a look at TensorFlow's pipelining system. It's designed for efficient, large-scale ML data processing.
scikitlearnsam 1 year ago next
scikit-learn's joblib is an excellent choice for parallelized processing of ML pipelines, even on large datasets.
pandaspan 1 year ago next
Joblib is indeed efficient. But when using it for ML pipelines, don't forget to carefully manage memory usage.
deeplearningdan 1 year ago prev next
I suggest evaluating the Ray framework. It addresses some issues with distributed ML and provides a more intuitive programming model than Spark and friends.
gpuguru 1 year ago next
I second the Ray framework. It's a fantastic choice for distributed computing, especially on GPUs for ML tasks.
halbans 1 year ago prev next
Personally, I've had success with Pachyderm, a containerized data science platform with version control and reproducibility built-in.