1 point by data_engineering 2 years ago flag hide 15 comments
dataengineeringyc 2 years ago next
Excited to announce that Data Engineering (YC W20) is hiring Data Engineers! Come join us and help us build the future of data infrastructure. Apply here: (link to job application)
john_doe 2 years ago next
Congrats on the hiring! I'm a data engineer looking to work on exciting projects. I'll apply now.
dataengineeringyc 2 years ago next
@john_doe Thank you! We're looking forward to receiving your application. We have both remote and on-site positions available.
data_engineer_fan 2 years ago prev next
Are you looking for remote positions or only on-site? Nice to see a YC company growing!
dataengineeringyc 2 years ago next
@data_engineer_fan We're open to both remote and on-site candidates. Glad you're excited about our growth!
jane_doe 2 years ago prev next
I've been following your progress for a while. Can you tell me more about the scale of the data you're working with and the types of challenges your team faces on a daily basis?
dataengineeringyc 2 years ago next
@jane_doe Absolutely! We work with massive datasets, with petabytes of data being processed daily. Our team faces challenges related to real-time data processing, data governance, data quality, and developing scalable solutions.
data_enthusiast 2 years ago prev next
What data engineering tools and technologies are you using in your stack?
dataengineeringyc 2 years ago next
@data_enthusiast We primarily use tools like Apache Spark, Apache Kafka, Amazon Kinesis, Airflow, PostgreSQL, and Snowflake. We also have some internal tooling built with Go, Python, and React.
devops_guru 2 years ago prev next
What's your stance on data observability and monitoring for data pipelines? Do you have any favorite tools for this?
dataengineeringyc 2 years ago next
@devops_guru Data observability is crucial for us. We use tools like Datafold and Monte Carlo for data pipeline monitoring and to catch anomalies and data quality issues early.
ml_engineer 2 years ago prev next
Are there any opportunities for ML Engineers or Data Scientists in your company? I'd love to work on data-related projects with a focus on machine learning.
dataengineeringyc 2 years ago next
@ml_engineer Yes, indeed! We have ML Engineer and Data Scientist positions available as well. They work closely with our Data Engineering team and often collaborate on interesting projects. Check out the open roles here: (link to job application)
big_data_fan 2 years ago prev next
What's your approach to testing in data engineering projects?
dataengineeringyc 2 years ago next
@big_data_fan We believe in comprehensive testing, including unit tests, integration tests, and end-to-end tests. For data quality, we have an extensive suite of validation tests. We also practice continuous integration, which helps us catch and fix issues quickly.