Principal Systems Engineer

About the Group:
At Oracle Cloud Infrastructure (OCI), we build the future of the cloud for Enterprises as a diverse team of fellow creators and inventors. We act with the speed and attitude of a start-up, with the scale and customer-focus of the leading enterprise software company in the world. AI2 Oerations is one of the core organisations within OCI. We are responsible for providing BM Compute power.

The AI2 Operations org comprises of a family of critical foundational infrastructure services that drive OCI’s hardware lifecycle activities
Work with product team on the shared full stack ownership of a collection of services and/or technology areas. Understand the end-to-end configuration, technical dependencies, and overall behavioural characteristics of production services. Responsible for the mitigating critical customer incidents, or deployments or testing required to improve security, performance, availability, and scalability of service. Authority for end-to-end performance and operability. Partner with development teams in meeting SLA to unblock customers. Articulate technical characteristics of services and technology areas and guide Development Teams to engineer and add premier capabilities to the Oracle Cloud service portfolio. Understand and communicate the scale, capacity, security, performance attributes, and requirements of the service and technology stack. Demonstrate clear understanding of automation and orchestration principles. Act as ultimate escalation point for complex or critical issues that have not yet been documented as Standard Operating Procedures (SOPs). Utilise a deep understanding of service topology and their dependencies required to troubleshoot issues and define mitigations. Understand and explain the effect of product architecture decisions on distributed systems. Professional curiosity and a desire to a develop deep understanding of services and technologies.

Responsible for the operation of production environments, including systems, supporting critical business operations. Will perform administration and analysis for multiple production environments and recommend new and novel solutions to improve availability, performance, and supportability. This is an opportunity to bring a combination of deep technical knowledge with administration/analysis knowledge of Oracle’s Cloud Infrastructure to provide support to a wide range of complex production environment problems related to immense growth, scaling, leveraging the cloud, extremely high performance, and high availability requirements.

Qualifications / Experience:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• 8+ years of experience in infrastructure engineering, cloud operations, site reliability engineering, production operations, or a related technical discipline.
• Expert-level Linux administration skills, particularly with Ubuntu and Oracle Linux in large-scale production environments.
• Proven ability to troubleshoot complex infrastructure issues across hardware, operating systems, networking, drivers, firmware, and service dependencies.
• Strong understanding of distributed systems and how compute, networking, storage, and application services interact to affect reliability and customer experience.
• Hands-on experience with automation, scripting, and configuration management using tools such as Python, shell scripting, Ansible, and Terraform.
• Strong knowledge of incident management, service management, root cause analysis, and operational improvement.
• Experience with infrastructure provisioning, configuration, monitoring, maintenance, and host lifecycle management.
• Solid understanding of the OSI model, TCP/IP, HTTP, and structured troubleshooting across multiple infrastructure layers.
• Demonstrated ability to serve as a senior technical escalation point and independently drive complex issues to resolution.
• Demonstrated success using operational metrics—such as mean time to restore service (MTTR), ticket age, incident volume, recurring issue rates, automation coverage, and service level performance—to drive measurable improvements.
• Strong planning and organizational skills, including prioritizing work, managing dependencies, tracking execution, and assessing operational risks.
• Clear written and verbal communication skills, with the ability to explain technical issues, customer impact, risks, and recommendations to technical and business stakeholders.

 

Infrastructure/Networking:

• Operate and improve Linux-based compute infrastructure to support reliable, scalable AI, GPU, and inference workloads.
• Apply strong networking knowledge across TCP/IP, IPv4/IPv6, DNS, DHCP, VPNs, and SSL/TLS to diagnose connectivity and service issues.
• Troubleshoot communication failures between hosts, nodes, and distributed services, identifying dependencies and potential points of failure.
• Use Ansible, Terraform, Python, and shell scripting to standardize infrastructure configuration and automate operational tasks.
• Support datacentre operations, including host provisioning, maintenance, hardware troubleshooting, repair coordination, and decommissioning.
• Collaborate with infrastructure and networking teams to improve availability, capacity, configuration consistency, and operational readiness.

 

Incident Management:

• Lead the technical response and coordination of critical infrastructure incidents, from initial triage through service restoration, recovery validation, and handoff.
• Assess customer and workload impact, clearly distinguishing confirmed issues from symptoms still under investigation.
• Coordinate across the Network Operations Centre (NOC), networking, datacentre operations, GPU and interconnect engineering, hardware repair, and service delivery teams to drive timely resolution.
• Provide calm, clear communication during high-severity incidents, keeping customers and stakeholders informed of impact, progress, risks, and next steps.
• Maintain accurate incident records, including timelines, accountable owners, mitigation actions, decisions, and escalation paths.
• Validate recovery with customers and technical owners before completing incident closure or transitioning to follow-up work.
• Lead root cause analysis and post-incident reviews, ensuring corrective and preventive actions are assigned and tracked to completion.
• Use telemetry, operational trends, and support ticket data to identify recurring issues, prioritize repairs, and strengthen incident response.

 

Technical Leadership:

• Provide hands-on technical leadership for compute and inference infrastructure as a senior individual contributor.
• Guide engineers through complex troubleshooting, infrastructure improvements, and operational decisions.
• Mentor team members, share technical knowledge, and contribute to hiring and onboarding.
• Set high standards for engineering quality, ownership, documentation, and operational discipline.
• Drive the adoption of automation, observability, reliable incident response, and continuous improvement.
• Contribute to architecture and design reviews, balancing reliability, scalability, maintainability, cost, and business priorities.
• Lead technical initiatives across teams, aligning stakeholders on priorities, responsibilities, and delivery plans.

 

Engineering and Automation:

• Develop and maintain automation that improves reliability, reduces repetitive manual work, and accelerates incident diagnosis and recovery.
• Build reusable scripts, operational tools, and workflows using Python, shell scripting, Ansible, Terraform, or similar technologies.
• Standardize infrastructure provisioning, configuration, validation, maintenance, and recovery procedures.
• Create and maintain clear runbooks and technical documentation to support consistent execution and knowledge sharing.
• Partner with engineering teams to turn recurring operational problems into sustainable engineering improvements.
• Support safe infrastructure changes through testing, staged deployment, validation, and rollback planning.

 

Cross-Functional & Stakeholder Engagement:

• Partner with Product, backend service engineering, datacentre operations, networking, and customer support teams to deliver reliable infrastructure services.
• Serve as a technical escalation point for complex production issues and critical infrastructure decisions.
• Translate customer requirements and operational challenges into clear technical priorities and improvement plans.
• Communicate infrastructure health, service risks, dependencies, and delivery progress to relevant stakeholders.
• Coordinate service readiness and operational handoffs, ensuring teams have the monitoring, documentation, and support procedures needed for production.
• Collaborate effectively with globally distributed teams to maintain clear ownership and timely execution.

 

Performance Optimization:

• Benchmark and tune infrastructure to improve performance, reliability, scalability, and resource efficiency.
• Analyze GPU utilization and identify opportunities to improve capacity use and workload efficiency.
• Investigate performance bottlenecks across compute, memory, storage, networking, and service dependencies.
• Improve performance consistency and resource isolation in multi-tenant environments.
• Partner with service and workload owners to diagnose customer performance issues and validate improvements.
• Use performance data to recommend configuration changes, capacity adjustments, and engineering priorities.

 

Monitoring and Telemetry:

• Build and maintain monitoring, dashboards, and telemetry that provide clear visibility into infrastructure health and customer experience.
• Use Prometheus, Grafana, and related tools to track availability, performance, capacity, and service reliability.
• Define actionable alerts and improve alert quality to support early detection and efficient troubleshooting.
• Correlate metrics, logs, events, and support ticket data to identify failures and emerging operational risks.
• Establish and track service level indicators and objectives, partnering with service owners to address reliability gaps.
• Improve monitoring coverage and diagnostic information across GPU hosts, networking, hardware, and supporting services.

 

Preferred / Nice to Have:

• Experience operating large-scale GPU fleets, AI/ML infrastructure, inference platforms, or high-performance computing (HPC) environments.
• Advanced experience with NVIDIA and AMD GPU ecosystems, including GPU diagnostics, performance analysis, and platform troubleshooting.
• Knowledge of high-performance networking technologies such as InfiniBand, RDMA over Converged Ethernet (RoCE), and Remote Direct Memory Access (RDMA).
• Experience with Kubernetes, containers, and workload orchestration in production environments.
• Familiarity with CUDA, NCCL, Slurm, or similar AI and HPC technologies.
• Experience building or scaling GPU cloud platforms, AI factories, or large-scale inference infrastructure.
• Experience providing technical leadership in 24×7 production operations with globally distributed stakeholders.
• Experience supporting multi-region infrastructure deployments or working with hyperscalers, cloud providers, or AI-focused technology companies.
• Experience improving operations through AIOps, event correlation, triage automation, alert noise reduction, and standardized runbooks.
• Experience with patching, upgrades, staged rollouts, change management, operational risk assessment, audit readiness, and compliance requirements.