Senior Data Engineer - Prophecy & PySpark

Job Description: The role of Senior Data Engineer - Prophecy & PySpark involves working with a large consultancy on a 12-month contract, requiring a mix of on-site and remote work. The position demands strong hands-on experience with Prophecy and PySpark, along with expertise in data engineering tools and concepts. The ideal candidate will also possess excellent communication and stakeholder management skills. This role is classified as inside IR35.

Key Responsibilities:

• Develop and maintain data engineering solutions using Prophecy and PySpark.
• Design and implement ETL/ELT pipelines and distributed data processing.
• Work with data lake architecture and modern Data Lakehouse concepts.
• Collaborate with stakeholders to ensure effective communication and project management.

Skills Required:

• Strong hands-on experience with Prophecy and PySpark.
• Proficiency in data engineering tools such as Spark, Databricks, Scala/PySpark, or SQL.
• Understanding of ETL/ELT pipelines and distributed data processing.
• Exposure to ETL tools like Informatica, Datastage, or Talend.
• Experience with Unity catalog, Delta lake, and Data Lakehouse concepts.
• Strong communication and stakeholder management skills.

Salary (Rate): undetermined

City: Leeds

Country: UK

Working Arrangements: hybrid

IR35 Status: inside IR35

Seniority Level: Senior

Industry: IT

Detailed Description From Employer:

Senior Data Engineer - Prophecy & PySpark

My client, a large consultancy, is in need of a Senior Data Engineer - Prophecy & PySpark for a 12 month contract opportunity inside IR35 based in Leeds, offering 2 days per week remote but requiring 3 days per week on-site.

The ideal candidate will have strong hands on experience with Prophecy (Using pyspark approach), experience in data engineering with tools such as Spark, Databricks, Scala/Pyspark or SQL, Strong understanding of ETL/ELT pipelines, distributed data processing and data lake architecture, Having exposure to ETL tools such as Informatica, Datastage or talend is added advantage, Experience with Unity catalog, Delta lake and modern Data Lakehouse concepts and Strong communication and stakeholder management skills.