Principal AI Engineer - Enterprise Transformation

Job Description

The Senior AI Engineer is the primary builder in the CoE. You will take prioritised workflows from architecture and PRD through to working, evaluated, production-grade AI systems — and, just as importantly, factor what you build into a reusable component library so that the fifth workflow costs a fraction of the first.

This is an applied engineering role with unusually high leverage. You are not building one product; you are building the components, patterns and reference implementations that internal engineers and external partner pods will use to build many. The work spans retrieval, agent orchestration, evaluation, and the unglamorous production engineering — latency, cost, failure handling, observability — that separates a demo from a system the business can depend on.

Where this role sits

Reports to: Head of the AI Centre of Excellence

Technical direction from: Lead AI Architect — you will work alongside them daily, and you are expected to push back when a design will not survive contact with production

Works with: The functional AI Engagement Specialists who own the workflows, PRDs and quality bar for the business functions in scope

Also partners with QAD engineering teams, Data & Platform Engineering, Security & GRC, and external system integrators

Scope: Foundational engineering hire. Expect to set the engineering standard, mentor subsequent hires, and review partner-delivered code.

Key responsibilities

Building AI and agentic workflows

• Build production AI and agentic workflows end to end — from PRD and architecture through implementation, evaluation, release and iteration in production.

• Design and implement agent orchestration: multi-step flows, tool and API invocation, planning and routing, state and memory, retries, fallbacks, timeouts and human-in-the-loop checkpoints.

• Make the judgement call on deterministic control flow versus model-driven control flow — and default to the former wherever it produces the same outcome more reliably and more cheaply.

• Integrate with enterprise systems — ERP, CRM, support, services delivery, knowledge and data platforms — including systems with imperfect APIs and imperfect data.

Retrieval and context engineering

• Own the retrieval stack: corpus onboarding, document parsing, chunking strategy, embedding selection, hybrid lexical and vector search, reranking, metadata filtering and query transformation.

• Implement permission-aware retrieval so that a user or an agent can only ever retrieve what that identity is entitled to see — non-negotiable in an enterprise context.

• Handle freshness and incremental indexing so that retrieval reflects the state of the business rather than a snapshot from onboarding day.

• Measure retrieval quality explicitly and treat it as a tunable subsystem with its own metrics, not as an assumption.

Model selection, tuning and evaluation

• Select, tune and route models across tiers based on measured quality, latency and cost — not on reputation or recency.

• Own prompt engineering, versioning, structured output and context strategy as versioned, tested artefacts under source control.

• Apply fine-tuning, adapters or distillation only where evidence shows the return justifies the operating burden — and be able to make that argument either way.

• Build the evaluation harness: golden datasets, offline evaluation, LLM-as-judge with human calibration, regression suites running in CI, and online quality monitoring with structured feedback capture.

• Work with the Engagement Specialists to turn a business quality bar into measurable criteria — and be honest when a workflow does not clear it.

The reusable component library

• Design, build and maintain the shared component library — SDKs, shared services, templates, reference implementations and documentation — that both internal engineers and partner pods build against.

• Treat internal engineers and SI teams as customers of your library: versioning, backwards compatibility, examples, and documentation good enough that people use it without asking you.

• Review partner-delivered code and designs for conformance, quality and maintainability.

Production engineering and operations

• Meet explicit latency and cost budgets per workflow, using caching, batching, streaming, model tiering and prompt efficiency.

• Build for graceful degradation: provider outages, rate limits, backpressure, partial failures and safe fallbacks.

• Implement observability for non-deterministic systems — full step-level tracing, token and cost telemetry, quality dashboards, and a triage path for incidents where nothing crashed but the output was wrong.

• Own infrastructure as code, CI/CD, environment promotion, secret and credential handling, and testing discipline for everything the CoE ships.

• Participate in the operational support model for live AI workflows, including post-incident review and remediation.

Craft and team

• Set the engineering standard for the CoE and raise it as the team grows.

• Mentor subsequent engineering hires and partner engineers.

• Document decisions and trade-offs so that the next engineer inherits reasoning, not just code.

Qualifications

What we are looking for

Essential

• 6+ years of professional software engineering, including 2+ years building and operating production LLM or GenAI systems — systems with real users and real operating cost, not notebooks or proofs of concept.

• Excellent Python, plus working competence in at least one other language (TypeScript, Go, Java or similar).

• Genuine production retrieval experience — you have built a RAG or hybrid search system, measured it, found it wanting, and improved it.

• Agent orchestration experience with frameworks such as LangGraph, LlamaIndex, Semantic Kernel, Google ADK, Strands, CrewAI or equivalent — together with the judgement to know when a framework is the wrong answer.

• Strong AWS and/or GCP experience, including managed AI services (Bedrock, SageMaker, Vertex AI or equivalent) alongside core compute, serverless, networking, IAM and managed data services.

• API design and enterprise integration, including authentication and authorisation patterns — OAuth 2.0 / OIDC, service-to-service authentication, and delegated or on-behalf-of access.

• Containerisation, infrastructure as code (Terraform, CDK or equivalent), CI/CD and real testing discipline.

• Evaluation rigour — you can define what “good” means numerically for a subjective task, defend the definition, and act on the result.

• Cost and latency awareness as a design instinct, not an afterthought raised by finance.

• Clear written communication — design notes, documentation and honest status.

• Comfort with ambiguity and shifting priorities in an early-stage function.

Strongly preferred

• Vector and hybrid search infrastructure at production scale — OpenSearch, pgvector, Vertex AI Search, Elasticsearch, Pinecone, Weaviate or similar.

• Evaluation and observability tooling — LangSmith, Langfuse, Phoenix, Braintrust, Ragas, DeepEval or equivalent.

• Multi-tenant SaaS engineering, and the data isolation discipline that comes with it.

• ERP, manufacturing or supply chain data experience.

• Classical ML and MLOps background — feature stores, model registries, monitoring and drift.

• Fine-tuning, parameter-efficient adaptation, or model serving and optimisation.

• Experience building internal platforms or SDKs consumed by other engineers.

• Open-source contribution in the AI engineering ecosystem.

The technical environment

A predominantly AWS estate with a meaningful GCP footprint, multi-tenant SaaS products, ERP and manufacturing data, and a heterogeneous set of enterprise systems across CRM, support, services delivery, marketing and data platforms. Engineering capacity is a mix of internal team and external implementation partners.

The AI toolchain is not fixed. You will help select it — and then live with the consequences, which is the correct incentive.

…