Description:
The Senior Data Engineer will join a team developing a world class data platform to support our customers, products, researchers, and back-office teams.
You’ll join a small squad of Data Engineers and QA specialists empowered to define ways of working that work best for all. You’ll be adding features, developing data pipelines, automating the ETL of complex data and designing and configuring platform solutions that support a wide variety of applications. You’ll have the freedom to come up with innovative solutions to unique challenges and help us use data as a force for good.
What We Can Offer You
- You will design and implement the data storage model, defining the rules for data governance, data quality, and the data lifecycle.
- You will work closely with the Information Architect, Product Owners, and Data Platform stakeholders, ensuring high level designs.
- Be responsible for ingesting data into our data lake and provide frameworks and services for operating on that data.
- Design and support data pipelines and recommend improvements and modifications.
- Build the process required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
- Develop and implement processes to ensure data accuracy and integrity.
- Collaborate closely with our cloud infrastructure team on shared items of work and communicate well.
- You will proactively ensure code reviews are prioritised to help unblock the team's work.
- You will identify and help others to resolve blockers, aiding delivery.
The Skills You Will Bring
- Designed and built scalable, data-centric systems with over 6 years of engineering experience, including 3 years of specialized focus on processing and transforming large-scale data for enterprise-level systems.
- You have hands-on experience with AWS services like Data Pipelines, Glue, Redshift, S3 etc., or equivalent cloud technologies.
- You have exceptional coding and design skills, particularly in Spark, Python, and Java.
- You have knowledge of dbt, HBase, Apache Spark, Spark streaming, Kafka, Scala, and similar technology stacks.
- You have experience in business intelligence data modelling and familiarity with data warehouse concepts such as star schema and data de-normalizing patterns.
- You have experience mentoring engineers, helping them develop technical skills and improve team performance.
Achievements
- Successfully processed, transformed, and transported large-scale data, optimizing workflows and ensuring efficient data handling across complex systems.
- Applied architectural design principles to create scalable, efficient solutions, enhancing the overall performance and reliability of data platforms.
- Demonstrated an analytical and strategic mindset, driving successful project delivery by aligning execution with business goals, resulting in on-time delivery of key data platform features.
- Fostered collaborative relationships within cross-functional teams, driving strategic execution and maintaining effective communication to ensure timely delivery of data solutions.