Senior Site Reliability Engineer

Senior Site Reliability Engineer, GPU Infrastructure

Remote, EMEA or APAC | Full-time | Competitive base + meaningful equity

The company

I’m hiring for an AI infrastructure company that sources GPU compute from dozens of providers and runs it as one managed platform for AI labs and model builders. They have tens of thousands of GPUs under management and a team of around 50. They’re profitable and backed by some of the best-known names in AI. It’s a confidential search, so I can’t name them here, but I’ll tell you everything on a call.

The role

This isn’t a generalist SRE seat. You’ll run large GPU clusters for distributed training and inference, spread across multiple providers and regions, and work directly with the customers pushing them hardest. When a training run slows down, you’ll be the one tracing it from the network fabric through the kernel to the framework.

What you’ll own

• Designing and running multi-provider, multi-region GPU clusters for large training jobs
• Acting as the technical contact for customers, from onboarding through to live troubleshooting and tuning
• SLOs, capacity planning and reliability across a mixed GPU fleet
• Keeping InfiniBand, RoCE and NVLink fabrics healthy
• Monitoring that shows what’s happening with GPU utilisation, interconnect throughput and hardware health
• Automation for provisioning, health checks, self-healing and driver and firmware updates
• Leading incidents where the root cause could sit in hardware, network, orchestration or the ML stack

What you’ll need

• You’ve run large GPU clusters (A100, H100, B200 or similar) in production yourself
• Production experience with InfiniBand, RoCE or NVLink for distributed training
• A good understanding of how big training jobs behave at the systems level: NCCL, CUDA, PyTorch distributed or similar
• Strong Linux, including kernel tuning and NVIDIA driver and CUDA management
• Kubernetes with GPU workloads, or Slurm and other HPC schedulers (either counts)
• Python, Go or Bash for building proper tooling, plus Terraform, Helm, Ansible or similar
• GPU monitoring with DCGM and nvidia-smi
• Experience leading messy incidents where the cause could be anywhere from firmware to application code

Nice to have

• Parallel file systems like VAST, WEKA, Lustre or GPFS
• Profiling and tuning distributed training performance
• Physical cluster design: racks, power, cooling, network topology and burn-in