Senior Infrastructure / Site Reliability Engineer On-Call

Senior Infrastructure/Site Reliability Engineer On-Call

San Francisco in-personFull-timeREAD MORE

We're looking for a Senior Infrastructure/SRE Engineer to join our on-call team for enterprise clients. You'll be the technical expert our clients rely on when things go wrong with extremely valuable clusters—diagnosing complex infrastructure issues, resolving production incidents, and ensuring zero downtime for critical AI workloads. This role requires deep technical knowledge, excellent troubleshooting skills, and the ability to stay calm under pressure.

What You'll Do

• Respond to and resolve production incidents across our clients' infrastructure, including Kubernetes clusters, Ceph storage systems, and bare metal servers
• Diagnose complex issues ranging from pod scheduling problems and CNI networking failures to distributed storage performance degradation and hardware issues
• Handle escalations requiring deep expertise in distributed systems, including etcd cluster problems, Ceph RGW authentication issues, and custom networking setups with Cilium and bare metal load balancers
• Work directly with enterprise clients during incidents, providing clear communication about status, timeline, and resolution steps
• Participate in a follow-the-sun on-call rotation with engineers across time zones
• Document incidents thoroughly and improve runbooks based on recurring patterns
• Collaborate with our infrastructure team on long-term reliability improvements and automation to reduce incident frequency

Preferred Experience

• 5+ years of production experience with Kubernetes in enterprise environments, including deep knowledge of cluster operations, troubleshooting bare metal k8s issues, working with admission controllers, and understanding the control plane architecture
• Strong experience with distributed storage systems—Ceph experience is highly preferred, but deep experience with other systems like Weka, VAST, or similar is highly valuable as well
• Very solid modern Linux systems administration skills
• Experience with bare metal infrastructure management and construction, not just cloud environments. You should be comfortable with IPMI, hardware troubleshooting, networking, etc.
• Experience with at least one CNI plugin in production, preferably Cilium or Calico
• Strong troubleshooting methodology—you know how to systematically narrow down issues across complex distributed systems and can work effectively under pressure during outages

Highly Valued Experience

• Production experience with etcd clusters, including backup/restore procedures and troubleshooting cluster health issues
• Experience with GPU infrastructure for AI/ML workloads, including the NVIDIA Kubernetes operator and GPU stack
• Knowledge of infrastructure as code tools like Ansible, Kubespray, or similar orchestration frameworks
• Experience with high-availability load balancers, service mesh technologies, or IPAM systems
• Background working with AI inference or training companies and understanding their unique infrastructure requirements

What Makes You Successful

• You're calm and methodical during incidents, with good judgment about when to escalate versus when to dig deeper yourself
• You communicate clearly with both technical and non-technical audiences, especially during high-stakes situations
• You have intellectual curiosity about root causes and want to prevent problems in advance

APPLY NOW