Inference Systems Engineer: Linux Kernel Test and Validation

About the role

Introducing Cedana

The Problem

Inference is increasingly the dominant GPU workload and GPUs remain scarce and expensive. Every cold start, idle replica, and node failure is throughput that reduces revenue per GPU/watt. Data centers are bottlenecked by power requirements and GPU availability. Inference providers are measured on tokens per GPU, latency under load, and uptime, yet the complexity of models, hardware, and operations is a continuous and evolving challenge. Cluster productivity directly determines margin, research output, and productivity per MW.

Cedana’s Solution

Cedana mission is to accelerate and expand the ROI of every GPU. We maximize inference performance, reliability, and throughput per GPU with a breakthrough form of real-time GPU virtualization. We capture the full state of a running GPU workload and can manipulate it in real-time to enabling full workload migration, throughput and power optimization, and more. The result is more tokens from the same GPUs, faster time to first token, higher availability, higher power efficiency. Our system operates at the kernel and CUDA driver level, requires no code or config changes, and works seamlessly with Kubernetes, SLURM, and NVIDIA Dynamo. We work with the leading inference providers, and we deploy into neoclouds, enterprises, and research clusters.

The Team

Cedana's founding team has spent over a decade making computation run fast, productively, and reliably for AI. Our team has PhDs and MS from Caltech, CMU, Cambridge University, and have published across NeurIPS, CVPR, and Nature. We published some of the earliest formal methods for guaranteeing convergence in distributed training. At Shopify, we've developed a control plane for robotics fleets used in warehouse automation. We bring repeat founder experience, having built and exited a Series B healthcare AI company.

Backed by Y Combinator, Initialized Capital, Pebblebed (founders of OpenAI and Facebook AI Research), Keith Adams (engineer #20 at VMware, founded HHVM and FAIR at Facebook, Chief Architect at Slack), Venture Guides, Garry Tan, and Gokul Rajaram.

More about Cedana

Cedana has secured partnerships with established AI / HPC systems integrators and OEMs, is in production with leading inference providers, research institutions and enterprises. Check out our co-published work with NVIDIA and measured results on 3rd party benchmarks.

The Role

What you’ll own

You will own the core components of our inference virtualization stack including production, test and validation for kernel development. This includes measurement and optimization. Over time, you will contribute to critical core components of our system, with a focus on new capabilities, performance, and reliability.

What You'll Do

• Monitor and ensure regressions: Quantifying regressions to ensure our development cycle is converging to clearly specified objectives.
• Automate and validate testing: A key priority is to understand and extend our testing platform to establish your understanding of how our system works. Ensure testing is complete, all tests are passing, and our environments match our customers. Reliability and testing is a key part of our culture and you will push our bar higher.
• Measure and optimize platform performance: Get Cedana to the theoretical maximum performance by understanding fundamental bottlenecks. Measure reliability, throughput and performance using our internal tools.
• Design and contribute to key system components : Our solution touches all the major aspects of the OS, kernel, GPU, and CPU. Write design papers to outline your vision for a specific capability and then lead implementation.
• Educate our team: Our team is our best learning resource and we continually educate each other. We huddle and co-pair as needed.
• Excellent communication: You enjoy writing concise and articulate design papers on your code and experiments. You respond to slacks and emails within our internal SLAs.

What we are looking for

• 5-10 years of software engineering experience with Linux Kernel. Experience building internal tooling at a large company with recent experience.
• Kernel-level depth: reads kernel and driver source, and root-causes defects at the kernel/driver/syscall boundary rather than only reproducing them at the surface. Comfortable operating below the abstraction line, at the hardware/software interface.
• Proven expert-level mastery of at least one performance- or correctness-critical Linux systems domain (networking data plane, virtualization/hypervisor, storage and block I/O, scheduling, memory management, or real-time/determinism), plus demonstrated ability to ramp into an unfamiliar low-level subsystem quickly. Depth in one hard domain is a valuable signal that you can reach depth in the next; we are hiring the descent capability, not the specific subsystem. This included some combination of:
  • OVS, DPDK, SR-IOV, RDMA, or high-performance packet processing
  • CRIU + QEMU live migration + VFIO
  • runc / containerd / OCI / namespaces / cgroups v2 / overlayfs
• Performance and latency engineering: characterizes throughput, jitter, and tail latency; uses tracing and profiling tooling (perf, ftrace, eBPF, or equivalent) to localize bottlenecks to a code path.
• Systems-level QE: designs and owns automated test infrastructure, performance and regression harnesses, and reproduces kernel-level races and corner cases. Not manual or UI QA.
• Enterprise Linux distribution environment (RHEL or equivalent): version matrices, backports, customer-grade triage.
• Principal scope: owns test or validation strategy for a subsystem; sets technical direction; mentors.

AI Coding Experience

• Shipped internal tooling at a large organization within the last 3 to 4 years, through the shift from AI assistants being optional to expected.
• Use AI aggressively for speed: scaffolding, test generation, migrations, reading unfamiliar code.
• Treat AI output like a confident junior engineer’s PR: plausible until proven, and you do the proving.
• Can show a case where AI-generated code or analysis was wrong, would have shipped unchecked, how you caught it, and what you changed afterward.
• Hold a stated personal standard for what you verify by hand: anything touching correctness, data, security, or a number someone will act on.
• Bonus: QA, test infrastructure, or release engineering experience, ideally at an organization that mandated AI use for engineers.

Bonus if you have

• Real-time Linux (PREEMPT_RT) and deterministic-path validation.
• Device virtualization beyond NICs: VFIO, vGPU / GPU passthrough.
• Virtualization internals: KVM / QEMU / libvirt.
• Container and Kubernetes networking (CNI, OVN-Kubernetes).
• Upstream kernel or open-source contribution history.
• Telco / 5G / NFV domain context.
• Credentials as proxies, not gates: RHCA / RHCE, ISTQB CTAL-TM.

Logistics

• Remote, US-based.
• Base $160,000–$180,000 + meaningful early-stage equity.

Benefits

• 100% covered medical, dental, and vision insurance for employees and families.
• Unlimited PTO policy.
• Workstation setup budget.
• 401K Plan.

Equal Opportunity Employer

Cedana is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status

About the interview

• Initial interview for fit
• Written component to understand background and motivation. Not a coding test.
• Interviews with engineering team.
• References

About Cedana

Cedana is pause/migrate/resume for compute workloads. We're working on building a global, real-time system for compute. This means a paradigm shift in how we allocate resources to things like high performance computing, numerical simulation and training and running machine learning models. We do so by taking a systems-level and deep-tech approach to these problems, working at the Linux Kernel layer and with hardware.