Job Overview:
We are seeking a Platform Engineer with a strong focus on supporting and optimizing data platforms. This individual will play a key role in designing, building, and maintaining CI/CD pipelines, automated testing frameworks, and various tools to support data systems. The ideal candidate will have hands-on experience with ETL pipelines, data engineering processes, and platforms like Snowflake and FiveTran, and a keen understanding of how data flows through these systems. A self-starter with strong communication skills who thrives in a mid-sized environment will be an excellent fit for this role.
Key Responsibilities:
CI/CD Pipeline Creation and Maintenance:
Design, implement, and maintain continuous integration and continuous deployment (CI/CD) pipelines for data-related projects.
Automate data deployment processes to ensure smooth and efficient updates across environments.
Integrate automated testing frameworks into the CI/CD pipelines to ensure high-quality code delivery.
Development of Tools to Support Data Systems:
Build and maintain custom tools that support data infrastructure, including those that assist with data processing, monitoring, and reporting.
Develop and optimize automated workflows and frameworks to improve the reliability and scalability of data platforms.
ETL Pipeline Understanding and Optimization:
Develop, implement, and maintain ETL pipelines, ensuring smooth data flow through the various stages.
Troubleshoot and optimize ETL processes, identifying potential bottlenecks, inefficiencies, and errors.
Understand the interdependencies between data systems, ensuring that ETL pipelines are well integrated with other tools and platforms.
Experience with Snowflake and FiveTran:
Design and optimize Snowflake data warehouses to ensure efficient data storage, querying, and scaling.
Work with FiveTran for data integration, ensuring that data flows seamlessly between systems and is available for analytics and reporting.
Develop and maintain integration processes that utilize Snowflake and FiveTran.
Collaboration and Communication:
Collaborate with cross-functional teams, including data engineers, analysts, and DevOps, to ensure alignment and successful implementation of data solutions.
Provide support and guidance to internal teams on best practices for working with data platforms and tools.
Communicate technical concepts and progress to both technical and non-technical stakeholders effectively.
Problem Solving and Automation:
Continuously monitor and improve the reliability of the data infrastructure, identifying areas for automation and improvement.
Proactively address potential issues with data flow, pipeline performance, and tool functionality.
Qualifications:
Proven experience as a Platform Engineer or Data Engineer with a strong focus on CI/CD pipelines, automated testing, and data tools.
In-depth experience with ETL pipeline development and optimization.
Strong experience with Snowflake and FiveTran, including architecture design, data loading, and optimization best practices.
Familiarity with common data engineering tools and frameworks.
Excellent problem-solving skills with the ability to troubleshoot and optimize complex data systems.
Self-starter with the ability to work independently and take ownership of projects.
Strong communication skills, both written and verbal, with the ability to collaborate effectively in a mid-sized team environment.
Experience in a fast-paced environment with a focus on delivering results.
Familiarity with cloud-based data platforms (AWS, Azure, GCP) is a plus.
