789 points by datanerd 1 year ago flag hide 23 comments
johnny5alive 1 year ago next
Hey HN, I'm looking for recommendations on building a data pipeline and would love to hear about your experiences and some useful resources to check out.
datajedi 1 year ago next
Check out Apache Kafka and its ecosystem. Really useful for real-time data streaming and processing as well as message queues.
kafkaguy 1 year ago next
Kafka works really well for a wide range of data use cases, including stream processing, event-driven architectures, and of course data pipelines.
python_gal 1 year ago prev next
I've found the Apache Airflow project to be a great open-source tool for managing and building data pipelines. It allows you to programmatically create, schedule, and monitor workflows.
scriptkiddy 1 year ago next
I'm hearing more and more about Airflow. How does it compare to Luigi, another pipeline management project from Spotify?
workflowwarrior 1 year ago next
Both Airflow and Luigi offer similar functionality for data pipeline management, but Airflow is typically more flexible and scalable, thanks to its dynamic task DAGs.
databaseduke 1 year ago prev next
Have you looked into using a database for data synchronization instead of a full-blown pipeline? It really depends on your use case and throughput requirements.
python_gal 1 year ago next
Yes, it certainly does depend on the specific use case. For low-latency ingest and more complex transformations, something like Apache Beam may be better suited. However, for smaller data sets and simpler processing needs, a DB or ETL tool may be more appropriate.
bigdatadub 1 year ago prev next
Lately, I've been working with Apache Flink for my real-time data streaming and processing needs. Great integrations with Kafka, and you can use SQL for the data processing.
streamsmaster 1 year ago next
I've been meaning to take a closer look at Flink. Thanks for the recommendation! I've heard the learning curve can be a bit steep, though.
yan_streamer 1 year ago next
Flink's learning curve might be a little steeper, but it's worth it for how powerful it is. The community is active and continually improving the platform as well.
etlexpert 1 year ago prev next
I've had a great experience with AWS Glue. You can create ETL jobs and data pipelines quickly and easily, and it integrates nicely with the other AWS data and ML services.
cloudchief 1 year ago next
Yes, Glue has some great features and is continuously improved by AWS. However, the costs can be quite high if you have larger data sets and complex processing needs.
mlmonster 1 year ago prev next
Don't forget to monitor and validate the quality and integrity of your data during the pipeline. Tools like Apache Griffin and Great Expectations can help with that.
solitudeseeker 1 year ago next
Data observability tools are crucial in the realm of data engineering. Great Expectations indeed provides a wonderful and flexible platform to manage this.
datasage 1 year ago prev next
For those just starting out, you can't go wrong with the classics. Consider exploring open-source ETL tools like Pentaho and Talend, which can help with data integration and visualization.
extractguru 1 year ago next
That's a good point, dataSage. Open-source ETL tools can still be quite relevant and powerful, even when there's a lot of focus on more cutting-edge and complicated solutions. And they offer an easier entry into the world of data engineering.
ingestbuddy 1 year ago prev next
For real-time, high-throughput data ingestion, Apache NiFi is a powerful, scalable, and user-friendly open-source tool. It offers a wide array of processors for data routing and transformation.
streamstar 1 year ago next
NiFi's UI makes it a little bit easier for DevOps and engineers to pick it up and start defining data pipelines. I truly appreciate its web-based interface and ease of use.
visualizationking 1 year ago prev next
I would also recommend looking into cloud-native data pipeline services, such as Google Cloud Dataflow or Azure Data Factory. They're managed, and you can easily scale your pipelines horizontally as needed.
dan_the_dataman 1 year ago next
GCP Dataflow supports Apache Beam SDK, which allows you to build batch and streaming data pipelines and run them on various execution engines, such as GCP Dataflow, Spark, and Flink.
dataduchess 1 year ago prev next
When building a data pipeline, one crucial step that's often overlooked is data lineage. Make sure to keep track of how your data is processed, transformed, and where it flows.
lineagelady 1 year ago next
Having a well-defined data lineage is vital to facilitate debugging, data validation, and compliance purposes. Atlas is a great data governance tool from Hortonworks that helps manage data lineage.