Principal Site Reliability Engineer

Description

Designs and architects infrastructure and service to ensure reliability and functionality. Forecasts demands and responds to capacity needs. Collaborates with software development teams to develop reliable and scalable infrastructures. Exercises judgment when performing data collection to maintain and optimize operations and reliability. Leverages advanced knowledge to perform incident response and/or maintenance tasks. Provides comprehensive health and performance reporting. Identifies and recommends opportunities for automation. Communicates comprehensive information about services and proactively anticipates and articulates the potential impact of changes. Provides comprehensive support for technology and documents incidents. Conducts advanced experiments with new tools and develops and maintains advanced knowledge of site reliability trends.

Responsibilities

Mandatory

• 5+ years of hands-on experience across site reliability engineering, monitoring, and automation, with demonstrated ownership of production platforms.
• Strong application skills in EBS, Database, weblogic, middleware, ....
• Expertise in Oracle Database and Exadata monitoring, performance diagnostics, and remediation automation.
• Good to have programming skills in Python, shell scripting, integrations, and event-driven workflows.
• Proven experience building monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
• Good to have experience integrating AI/LLM capabilities into applications for incident analysis, contextual recommendations, and operational decision support.
• Experience implementing alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
• Strong experience developing automated remediation workflows with approval controls, execution validation, audit trails, and recovery safeguards.
• Experience integrating monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.
• Strong understanding of distributed systems, Linux, networking, databases, and application performance, with the ability to translate failure patterns into platform capabilities.
• Ability to evaluate platform reliability and AI output quality, measure operational outcomes, and improve capabilities using engineer feedback.
• Excellent communication and collaboration skills, with the ability to work with monitoring teams, service owners, and engineers to onboard services and define requirements.

Good-to-Have

• Experience with agentic AI frameworks, tool calling, retrieval-augmented generation, and vector databases.
• Knowledge of anomaly detection, time-series analysis, and predictive techniques for identifying recurring alarm patterns.
• Experience building centralized notification services, automated escalation calls, and incident communication workflows.
• Familiarity with OCI, Kubernetes, and Infrastructure-as-Code tools such as Terraform or Ansible.
• Experience with observability platforms such as OCI Observability, Prometheus, Grafana or equivalent tools.
• Experience using AI coding assistants to improve operational scripts and validate proposed changes.
• Knowledge of SLOs, error budgets, and incident metrics such as detection time, recovery time, and recurrence rate.