Lead Principal Core Infrastructure Engineer

Description

Provides senior technical leadership for OCI’s network monitoring and observability platform, responsible for the systems that provide visibility into the health, performance, and behavior of OCI network infrastructure.

Architects highly scalable, reliable, and efficient distributed systems for collecting, processing, storing, and analyzing large volumes of network telemetry. Defines technical strategy and engineering standards for telemetry pipelines, real-time stream processing, metrics, alerting, visualization, and network health analytics.

Drives architecture for systems that must operate continuously at cloud scale, with strong requirements for availability, data integrity, latency, scalability, and operational efficiency. Identifies systemic performance and reliability constraints and leads solutions across multiple services and engineering teams.

Serves as a senior technical authority for network observability and distributed systems, leading complex architectural and production issues, influencing cross-organizational technical decisions, and mentoring senior engineers. Partners with network engineering, infrastructure, and service teams to continuously improve OCI’s ability to detect, understand, and resolve network problems.

Responsibilities

Network Observability Architecture

• Define the architecture and technical direction for OCI’s network monitoring and observability platform.
• Architect distributed systems that collect, process, aggregate, store, query, and visualize network telemetry at cloud scale.
• Design scalable telemetry ingestion and streaming architectures capable of handling high-volume and high-cardinality data.
• Define strategies for metrics, events, alerts, topology, network state, and other signals required to understand network health.
• Drive architectures that enable rapid detection, correlation, diagnosis, and isolation of network failures and performance degradation.
• Establish standards for telemetry quality, completeness, freshness, accuracy, retention, and availability.

Scalability & Distributed Systems

• Lead the design of horizontally scalable and elastic systems supporting continued OCI infrastructure and traffic growth.
• Identify performance, throughput, latency, storage, and scalability bottlenecks across telemetry pipelines and drive systemic improvements.
• Architect high-throughput streaming and event-processing systems with appropriate partitioning, backpressure, buffering, aggregation, and failure-handling strategies.
• Define resilient state-management, replication, synchronization, and recovery strategies for distributed monitoring systems.
• Evaluate architectural trade-offs involving consistency, availability, latency, durability, and cost.

Reliability & Operational Excellence

• Establish SLOs and engineering standards for availability, durability, latency, data freshness, and correctness of the observability platform.
• Architect fault-tolerant systems that continue operating through infrastructure failures, network partitions, service disruptions, and software upgrades.
• Define KPIs and telemetry that measure the health of the monitoring platform itself and identify gaps or blind spots in observability.
• Lead diagnosis and resolution of complex production issues spanning networking, distributed systems, telemetry pipelines, and infrastructure.
• Serve as a senior technical escalation point for critical incidents and drive root-cause analysis and systemic corrective actions.
• Design systems and operational practices that support automated deployment, upgrade, rollback, recovery, and minimal customer-visible disruption.

Network Monitoring & Analytics

• Define approaches for monitoring large-scale Layer 2 and Layer 3 network infrastructure and identifying changes in network state, topology, reachability, performance, and device health.
• Drive correlation of telemetry across devices, network layers, and infrastructure services to improve fault localization and reduce time to detection and resolution.
• Develop approaches for identifying abnormal network behavior, telemetry gaps, capacity risks, and emerging infrastructure failures.
• Partner with network engineering teams to translate network behavior and operational requirements into scalable monitoring capabilities.
• Improve signal quality and actionable alerting while reducing noise and unnecessary operational load.

Technical Leadership

• Set technical direction and influence architecture decisions across Network Monitoring and dependent OCI organizations.
• Lead complex and ambiguous initiatives spanning multiple systems and engineering teams.
• Establish architectural patterns, engineering standards, and technical best practices for large-scale observability systems.
• Mentor senior engineers and provide technical guidance on distributed systems, networking, reliability, and observability.
• Evaluate emerging technologies and engineering approaches and drive adoption where they materially improve scale, reliability, performance, or operational efficiency.
• Contribute to technical talent development through senior-level interviewing, candidate assessment, mentoring, and knowledge sharing.

Skills & Technologies

The candidate should have deep expertise in several of the following areas and sufficient breadth to lead architecture across the complete observability stack:

• Distributed Systems & Cloud: Large-scale distributed systems, cloud infrastructure, Kubernetes, service-oriented architectures, distributed state management, high availability, fault tolerance, capacity planning, and performance engineering.
• Observability: Metrics and telemetry architecture, Prometheus, Grafana, time-series data, high-cardinality metrics, alerting, dashboards, SLOs, telemetry pipelines, and monitoring large distributed environments.
• Streaming & Data Processing: Kafka, Flink or equivalent technologies; real-time stream processing, event-driven architectures, partitioning, aggregation, backpressure, data pipelines, and large-scale telemetry processing.
• Networking: Strong understanding of L2/L3 networking, routing and switching, network topology, BGP, LLDP, SNMP, gNMI, network failure modes, and network performance troubleshooting.
• Network Telemetry: Experience designing or operating systems that collect and correlate telemetry from large fleets of network devices using streaming telemetry, counters, events, protocol state, and device health information.
• Programming: Strong software engineering expertise in Java, Go, or comparable systems programming languages, with experience building highly concurrent, performance-sensitive production services.
• Reliability Engineering: SLOs, availability and durability engineering, distributed failure handling, load shedding, throttling, rate limiting, incident response, root-cause analysis, and production readiness.
• Automation: Infrastructure as Code, automated deployment and configuration management, safe rollout and rollback strategies, and operating large infrastructure fleets with minimal manual intervention.
• Security: Secure multi-tenant cloud architectures, authentication and authorization, encryption, vulnerability remediation, and security considerations for infrastructure telemetry and management systems.

Experience building observability or telemetry platforms for large-scale cloud or network infrastructure is strongly preferred.

Qualifications

Career Level - IC5