Data Engineer - Databricks

Job Role We are looking for a Databricks Data Engineer to design, build and operate scalable data pipelines and data platform capabilities within a modern lakehouse architecture. You will work in a collaborative agile engineering team delivering production-grade data solutions using Databricks, Spark, Python and SQL in a cloud environment. The role focuses on developing robust data pipelines, ensuring high data quality and enabling analytics and data product delivery for enterprise clients. Key Responsibilities · Design and implement ELT/ETL pipelines using PySpark and Databricks · Build scalable batch and streaming data pipelines using Spark and Kafka · Develop optimised SQL and Python pipelines for data transformation and integration · Integrate external data sources using REST APIs and data ingestion frameworks · Optimise Spark jobs and cluster performance for reliability and cost efficiency · Implement data quality checks, validation and monitoring · Apply CI/CD practices and version control for pipeline deployment · Work closely with solution architects, DevOps engineers and business analysts to deliver data products · Produce technical documentation, architecture diagrams and operational runbooks Key Requirements · Eligible and willing to pass relevant background checks · 3+ years experience building production data pipelines · Strong Python and Microsoft SQL Server development skills · Hands-on experience with Databricks, Spark and PySpark · Experience with Delta Lake, Unity Catalog and Databricks Workflows · Experience with data pipeline design, testing and deployment · Familiarity with test-driven development, CI/CD practices and Git-based development · Strong collaboration and communication skills · Working knowledge of Hadoop · REST APIs and Kafka experience Nice to Have · Cloud experience with AWS, Azure or GCP · Certifications in Databricks, AWS or data engineering · Infrastructure as Code (Terraform) · Orchestration: Airflow or AWS Glue · Streaming technologies including Structured Streaming · Observability tooling for data pipelines · Experience working with regulated industries such as banking or financial services