Senior Data Engineer

Description:

My client is looking for a Senior Data Engineer to join their growing team. They need someone with experience in Big Data frameworks, cloud technologies, and data processing systems. They’re looking for someone to build, optimize and maintain ETL pipelines while leveraging modern cloud and distributed computing solutions. A big added would be someone from a finance or banking domain. They offer the opportunity to grow your career while developing scalable data solutions and high-quality analytics.

You’ll be part of a dynamic team tackling financial data challenges with cutting-edge technologies where you’ll lead initiatives and collaborate with your team to innovate and learn. You’d be joining a leading international IT Consultancy business with world class opportunities in their hub in Letterkenny, with a positive and nurturing work culture who is growing rapidly in Ireland.

The Role

You’ll build, optimize, and maintain ETL pipelines using Hadoop ecosystem tools (HDFS, Hive, Spark). Developer and design scalable data pipelines in Spark with programming languages such as Java, Scala, or Python.

Perform data modeling, quality checks, and system performance tuning. Collaborate with teams to ensure efficient and reliable data processing across the platform. Leverage Big Data frameworks like Hive, Impala, Oozie, Airflow, and HDFS for data processing and management.

Work on modernization projects, including potential cloud or Databricks integration. Utilize cloud services such as AWS S3, Athena, EMR, Redshift, Glue, and Lambda for cloud-based data solutions. Contribute to the development and maintenance of Data & AI platforms such as Databricks.

Skills/Experience Needed

Solid hands-on experience in Big Data technologies, including Hadoop, Spark, Hive, Kafka, and Unix scripting. Solid hands on experience in Java, Scala, or Python, with expertise in Spark concepts such as RDDs, DataFrames, and Spark Streaming.

Experience with ETL platforms like PySpark, DataStage, and Airflow, with strong skills in data integration and normalization. Proficiency in cloud services such as AWS S3, Athena, EMR, Redshift, Glue, and Lambda.

Good knowledge of data modeling, including OLAP, OLTP, physical/logical modeling, and performance tuning. Good understanding of data processing frameworks and tools, including Hive, Impala, and Databricks. Familiarity with CI/CD pipelines and tools such as Git, Jenkins, and BitBucket. Strong problem-solving skills and an analytical mindset.

Good to have: Experience with Confluent Kafka and Redhat JBPM. Expertise in integrating search solutions with middleware and distributed messaging systems. Certification in Big Data or related fields.

Based in Letterkenny, Co. Donegal. (Hybrid – 2-3 days a week on site). Permanent Role. Salary according to experience. Benefits include bonus, pension, health care, life insurance, laptop, phone, access to extensive training resources, company discounts, bike to work scheme, onsite parking and other.