Data Engineer - Databricks & Big Data Pipelines (SC Cleared is required for this role)
Location: UK only remote is possible
Working Mode: remote
Rate: £450-550 per day inside IR35
Description:
We're looking for a Senior Data Engineer - Databricks & Big Data Pipelines to join our team, working in our UK remote working division. This position requires active UK Security Check (SC) clearance and offers the opportunity to build advanced, large-scale data engineering solutions for an innovative analytics platform. You will design, develop and optimize distributed data pipelines on Databricks and cloud ecosystems while ensuring performance, scalability, and compliance. The role combines technical depth with client-facing delivery as part of an agile and collaborative environment.
Responsibilities
- Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python
- Lead the full pipeline lifecycle: analysis, architecture, development, testing, deployment and monitoring
- Implement ingestion, normalization and standardization workflows for high-volume and diverse data sources
- Ensure data platform architecture supports performance, security, and compliance requirements
- Collaborate cross-functionally with engineering, product teams and client stakeholders during project delivery
- Support ML readiness for ingestion and transformation pipelines where required
- Promote data platform best practices across security, governance and cloud adoption strategies
Mandatory Requirements
- Active SC Clearance
- Expertise in Databricks, including Delta Lake and Unity Catalog
- Strong background in Python, PySpark and advanced SQL
- Proven experience in AWS cloud platforms for data engineering
- Track record of building high-performance, large-scale data solutions in production environments
- Ability to lead pipeline lifecycle (design, deploy, monitor) autonomously
- Strong communication skills for stakeholder collaboration and client presentations
SkillSet Overview
Must Have:
- Databricks (Delta Lake, Unity Catalog, workflows)
- Python Data Engineering Technologies (including PySpark)
- SQL
- AWS
Nice to Have:
- AI and ML Ops
- PostgreSQL
- OpenSearch
- Microsoft Azure
- Claude Code (AI productivity tooling)
- Docker / Kubernetes
- REST APIs
- Cloud-native ML pipeline deployment
- Familiarity with managing data platform releases, monitoring and cost governance
- Domain understanding in supply chain analytics, compliance or public sector data
- Experience integrating AI tools into engineering workflows
About Korn Ferry
Korn Ferry unleashes potential in people, teams, and organizations. We work with our clients to design optimal organization structures, roles, and responsibilities. We help them hire the right people and advise them on how to reward and motivate their workforce while developing professionals as they navigate and advance their careers. To learn more, please visit Korn Ferry at www.Kornferry.com
