Example Company
Software engineer
- Location
- San Francisco
- Type
- Full time
- Compensation
- $150k-$200k
- Posted
- 2 days ago
Join a team building useful software. Work closely with engineering and design to turn new ideas into products.
Loading job details
Deep Learning Team Lead
- Location
- Tel Aviv
- Type
- Full time
- Posted
- 34M AGO
About the role
Description
AI21 is looking for a Deep Learning Engineer – Team Lead to join our team.
AI21 is one of the few companies in the world to have trained multi-billion-parameter Large Language Models (LLMs). Building and serving models at this scale requires strong software engineering, deep learning expertise, and cutting-edge infrastructure.
In this role, you will lead a team of engineers working on our LLM training and inference infrastructure. You will collaborate closely with applied researchers to develop and scale new training methods, improve performance, optimize model serving, and make the best use of advanced hardware accelerators.
This is a hands-on technical leadership role combining engineering, research, and team management.
Role and Responsibilities
- Lead, mentor, and manage a team of deep learning engineers, including task delegation, technical guidance, and project execution.
- Develop, improve, and maintain our LLM training framework and supporting infrastructure.
- Optimize training performance, scalability, and model quality through parallelism techniques, training SDK improvements, and new model architectures.
- Research, develop, test, and integrate new training methods and reinforcement learning environments.
- Adapt training infrastructure to support the latest GPU hardware and accelerators.
- Optimize model serving for high throughput and low latency in production and training environments.
- Work closely with applied researchers on critical post-training efforts.
- Design, run, and analyze training and evaluation experiments to improve LLM performance across different tasks.
- Debug complex issues across the training and inference stack, from model behavior to distributed systems and hardware performance.
Requirements
Must-Have Qualifications
- 3+ years of experience developing production software, with strong Python skills and solid engineering fundamentals, including testing, debugging, performance optimization, and working with unfamiliar codebases.
- Proven experience leading and managing engineering teams, including mentoring, delegation, and project execution.
- Hands-on understanding of end-to-end LLM training in PyTorch, including data, model, tensor, and pipeline parallelism, mixed precision, and memory/compute tradeoffs.
- Understanding of distributed training frameworks such as FSDP, Megatron, or DeepSpeed, including their underlying mechanisms. Experience with a specific framework is not required.
- Deep understanding of LLM architectures, including attention variants, Mixture of Experts (MoE), positional encodings, and KV cache.
- Ability to investigate and debug complex issues such as loss spikes, NCCL hangs, log-probability mismatches, and inefficient kernels.
- BSc or higher in Computer Science, Electrical Engineering, or a related field.
Required Specialized Experience
Significant hands-on experience in at least one of the following areas:
- Distributed Training: Scaling training jobs across nodes, working with FSDP, Megatron, DeepSpeed, or JAX-based frameworks, and diagnosing communication and scaling inefficiencies.
- Post-Training and Reinforcement Learning: SFT, DPO, GRPO, online and offline training, rollout–trainer coupling, and frameworks such as verl, slime, prime-RL, TRL, or equivalent in-house solutions.
- Inference Systems: Serving frameworks such as vLLM, SGLang, or TensorRT-LLM, batching, scheduling, KV cache management, and latency/throughput optimization.
- Model Efficiency: Quantization, distillation, kernel-level optimization, speculative decoding, and model export or serialization for deployment.
Nice to Have
- Experience with distributed, containerized GPU training environments such as Slurm, Kubernetes, or similar platforms.
- Experience launching and debugging multi-node training jobs, including container images, storage, scheduling, and resource constraints.
About Us
AI21 is one of the world's leading and most experienced foundational AI labs, with extensive expertise in developing and training large language models.
As companies deploy AI agents at scale, they face growing challenges around performance, cost, and the specialized expertise required to manage these systems.
AI21 helps companies scale their AI agents sustainably by automatically reducing token costs and waste without compromising frontier-model quality.
Our approach combines advanced research and engineering across the entire AI stack, from intelligent model routing and agent harness optimization to model training and inference.
Similar roles
Principal, Product Learning & Developmenttwilio · Remote
ConsultantDeloitte · Philadelphia- Bilingual Senior Regional Media Advocacy Manager (English & Spanish) - RemoteAmerican Cancer Society · Remote
Mortgage Servicing SpecialistNew American Funding
Associate Creative Director (Design)Day One Agency · Los Angeles
Solutions Analyst IV - Encompass AdministratorLayer 6 AI · Charlotte