43 points by bob_builder 7 months ago flag hide 15 comments
johnsmith 7 months ago next
Hey there, I'm looking for advice on building a multi-cloud data pipeline orchestration system. I'm thinking of using Apache Beam to build a unified programming model and leverage its pipeline execution capabilities.
cloudguru 7 months ago next
Apache Beam looks like a great choice! I'd recommend considering using a managed service like Google Cloud Dataflow or AWS Data Pipeline for pipeline execution. This will help abstract some of the underlying complexities of running your pipelines in a multi-cloud environment.
johnsmith 7 months ago next
@cloudguru I'll definitely consider that. Thanks for the recommendation!
containersaurus 7 months ago prev next
I've had good experience with Kubernetes and using something like KNative to manage your data pipeline's deployment and orchestration in a multi-cloud environment. It's worth checking out!
johnsmith 7 months ago next
@containersaurus Thanks for the tip, I think I'll also give KNative a look!
multicloud 7 months ago prev next
Have you looked into using Apache NiFi or StreamSets for multi-cloud data integration? Both are very powerful tools and I think could be helpful in your use case.
johnsmith 7 months ago next
@multicloud I've heard of Apache NiFi, but haven't looked into it too much. I'll definitely check it out. Thanks for the suggestion!
opsgenie 7 months ago prev next
Building a multi-cloud data pipeline orchestration system is definitely a big task, but it's definitely worth the effort! We recently open sourced our own multi-cloud data pipeline orchestration system, so feel free to check it out and see if it's helpful.
johnsmith 7 months ago next
@opsgenie Thanks for sharing! I'll definitely check it out and see if it's helpful for my use case.
opensourcefan 7 months ago prev next
I highly recommend looking into Apache Airflow for orchestrating your data pipelines. It's highly customizable, widely used, and has a great community behind it!
johnsmith 7 months ago next
@opensourcefan Thanks for the recommendation, I've heard good things about Apache Airflow, I'll add it to my list to check out!
cloudtech 7 months ago prev next
Have you considered using serverless architectures like AWS Lambda or Google Cloud Functions for your data pipelines? They're designed to be highly scalable and can reduce operational burden.
johnsmith 7 months ago next
@cloudtech That's an interesting thought. I'll definitely consider that as an option as well.
bigdataexpert 7 months ago prev next
I've worked on similar multi-cloud data pipeline orchestration systems before and I'd be happy to help! A few things you should consider: 1) Security and encryption, 2) Data lineage tracking, 3) Observability and monitoring, 4) Easy deployment and management
johnsmith 7 months ago next
@bigdataexpert Thank you so much! Those are all great points, I'll definitely keep them in mind as I'm building these systems.