Senior Data Engineer & AI Expert

Data / AI Engineer (Databricks) - Outside of IR35Contract

We're taking an AI decision-support product from proof-of-concept to full production on Databricks and need a hands-on engineer to build, run and own it. It analyses data across ~1,000 sites, uses an LLM (Anthropic Claude) to generate per-site recommendations, and includes a conversational agent that lets managers question their own data. Delivery role, not advisory - you'll fill our gap in Databricks engineering, LLM pipelines and agent development, and build the capability in-house.

Initial 6 Month Contract (Ext) & Remote Working

What you'll do

• Build ingestion pipelines (PySpark) into a governed medallion layer in Unity Catalog.
• Productionise the LLM pipeline - run Claude across the estate on a schedule, with retries and validation gates.
• Build and tune the conversational agent - grounding, guardrails, benchmarking.
• Own evaluation (prompt versioning, MLflow) and CI/CD across dev/UAT/prod.

Essential

• Production Databricks (Unity Catalog, Delta, medallion, Workflows).
• Python/PySpark and strong SQL.
• Production LLM/API integration - batch inference, automated quality checks.
• Conversational / text-to-SQL agents over enterprise data.
• CI/CD for data platforms; Azure.

Nice to have: Databricks Genie / Mosaic AI, MLflow, RAG, Power BI.