Principal Core Infrastructure Engineer - Physical Networking Automation and Tooling

Description

Oracle Cloud Infrastructure (OCI) delivers mission-critical cloud services to enterprises worldwide. The Physical Networking Automation and Tooling team builds innovative solutions that improve the productivity and efficiency of Network Engineering teams through automation, observability, and actionable insights at hyperscale.

As a Principal Core Infrastructure Engineer, you will design, build, and deliver scalable automation frameworks and advanced platforms that leverage AI/ML to drive operational excellence across OCI’s global network. You will develop event-driven data pipelines that process network signals such as failures and power capabilities including anomaly detection, automated remediation, self-healing operations, model training, and real-time inference.

You are passionate about building software that solves real-world operational challenges. You thrive in a fast-paced environment, are comfortable working with complex distributed systems, and value simplicity, scalability, and collaboration.

Key Responsibilities

• Lead the design of distributed, event-driven systems that ingest and correlate network failures, performance, power, capacity, topology, and change data at global scale.

• Architect AI decision systems for anomaly detection, fault diagnosis, capacity forecasting, and remediation. Establish how models and agents use network context, express uncertainty, and escalate decisions to engineers when needed.

• Direct the development of agent workflows that discover and invoke approved capabilities, coordinate tools and services, maintain execution state, and recover safely from partial failures.

• Establish controls for actions affecting production networks, including identity and authorization, approval gates, validation, limits on the impact of failures, cancellation, rollback, and audit trails.

• Define evaluation and observability standards for AI-driven capabilities. Measure diagnostic accuracy, false positives, action quality, latency, cost, and operational outcomes before and after deployment.

•

Set architecture and engineering standards across APIs, backend services, data pipelines, workflow execution, testing, deployment, and production operations.

Minimum Qualifications

• At least 6 to 10+ years of experience in software engineering, network automation, cloud infrastructure, or a related field, including technical leadership of initiatives spanning multiple teams.

• Bachelor’s degree in Computer Science, Computer Engineering, a related technical field, or equivalent practical experience.

• Strong programming skills in Java and Python, with experience building and operating production software.

• Deep experience designing distributed systems, APIs, event-driven data pipelines, workflow systems, or cloud-native services.

• Experience defining technical architecture and leading complex projects from concept through production operation.

• Experience with network operations, data center infrastructure, or other large-scale infrastructure systems.

• Experience building AI/ML or LLM-based capabilities to operational problems, including evaluating their output and defining when human review is required.

• Strong knowledge of Linux, software development practices, CI/CD, observability, and production incident response.

• Demonstrated ability to influence senior stakeholders, resolve architectural tradeoffs, and mentor experienced engineers.

Preferred Qualifications

• Experience developing AI/ML automation workflows with network telemetry, device connectivity, configuration validation, or automated remediation in large-scale networks.

• Track record of architecting agentic systems, including planner and executor patterns, tool orchestration, agent handoffs, state management, checkpointing, retries, and durable workflows.

• Hands-on work integrating agents with enterprise tools through MCP, A2A, or similar protocols, including authentication, tool schemas, context handling, approval gates, and failure handling.

• Knowledge of AI evaluation, quality, observability, and governance practices for production systems.

• Proficiency with Kubernetes, containers, and operating cloud-native platforms(e.g., AWS, Azure, Google GCP, Oracle Cloud) in production.

• Practical use of AI-assisted development tools for coding, testing, review, and debugging.

Responsibilities

Key Responsibilities

System Design & Architecture - System Scalability:

–Lead the development and implementation, and begin to architect, components of scalable distributed systems that support horizontal and vertical scaling to meet system demands, including leveraging distributed state management tools.

–Optimize code and/or systems for large-scale data processing and high-throughput requirements to support hyper-scale systems.

–Define scalability requirements for owned components and ensure design and implementation requirements are met.

–Design systems to scale with elasticity (e.g., effectively scaling both up and down).

–Leverage data plane platforms to effectively handle large-scale data retrieval, storage, and processing.

–Design performance and load testing.

System Design & Architecture - System Reliability Design:

–Build and design fault-tolerant components and systems capable of withstanding in-service updates by implementing redundancy, replication, and automatic failover mechanisms.

–Design systems to effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.

–Implement and optimize approaches to handle network unreliability, including load-shedding, throttling, and rate-limiting.

–Design components and systems that are durable and adhere to service level objectives (SLOs), setting expectations for availability and durability of other computing services within the department.

System Design & Architecture - System Reliability Performance:

–Define key performance indicators (KPIs) and telemetry to identify gaps or issues in running systems.

–Build and customize moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor components and system health..

System Design & Architecture - Correctness / Availability:

–Design and implement functional and correctness requirements for feature sets and/or systems in new or existing systems.

–Design complex test scenarios (e.g., fault-injection, brown-out) to evaluate system correctness.

–Implement data replication and synchronization techniques to maintain data integrity and availability.

Operational Troubleshooting & Incident Management:

–Take a proactive role in diagnosing, debugging, and resolving issues in active components and systems to support ongoing operation, and mentor others in these processes.

–Implement strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.

–Maintain expertise in owned components and systems to ensure effective troubleshooting and performance.

–Meet operational readiness expectations through design and implementation.

–Serve in operational support rotations, providing guidance in incident response and root cause investigations.

Compliance & Security:

–Implement robust security measures to protect data and applications in multi-tenant environments, including encryption techniques and access controls.

–Execute remediation plans to address identified security gaps.

–Ensure cloud infrastructure is in compliance with industry standards and regulations and that documentation is up to date.

Automation & Change Management:

–Develop and maintain automation scripts and tools (e.g., Infrastructure as Code (IaC)) to manage cloud infrastructure.

–Create and adhere to change management plans for patching, updating, and rolling back applications, and begin designing systems and components to allow for automation of these processes.

Core Responsibilities

Planning & Execution:

…