Vulcan Vector
← All open roles

GPU CLUSTER RELIABILITY

Site Reliability Engineer

Own the reliability of the GPU cluster as a working system: healthy hardware, a functioning fabric, usable storage and research jobs that can make progress.

About Vulcan Vector

GPU capacity is valuable when researchers can use it with confidence. Getting there takes more than servers: it takes a validated fabric, a dependable data path, sensible scheduling and someone who owns the operating details.

Vulcan Vector builds and manages that foundation. Yggdrasil brings bare-metal validation, storage, Kubernetes, Slurm, access and observability into a service for AI startups and research labs. We want researchers to spend their time on experiments while we take care of the systems beneath them.

The engineering challenge

You’ll help operate Yggdrasil across the layers that can interrupt a research run. You should be comfortable using production evidence to narrow down a failure, coordinating the response and building the operational changes that make the next incident less likely.

What you’ll own

  • Useful GPU and cluster telemetry. Connect GPU health, host metrics, fabric signals, scheduler state and workload symptoms. Build Prometheus and Grafana views that help an operator decide what to do next.
  • An incident response people can trust. Take part in a defined on-call rotation for critical incidents. Establish impact, coordinate escalation, keep customers informed and carry corrective work through to completion.
  • A diagnosis across the stack. Investigate unhealthy devices, driver problems, link degradation, storage latency and failed collectives. Correlate hardware and system signals with the affected research jobs.
  • Service objectives with operational meaning. Define availability and responsiveness measures for the service boundary. Use incident history and error budgets to prioritise reliability work and maintenance.
  • Less repetitive operational work. Build health checks, configuration checks and safe remediation procedures. Automate recurring tasks with safeguards that are appropriate for expensive shared infrastructure.
  • Prepared recovery and change paths. Exercise recovery procedures, plan upgrades and validate the cluster after maintenance. Maintain escalation notes and runbooks that another engineer can use.

The experience that matters

  • Production ownership. You have supported a distributed environment through real failures and can explain your part in diagnosis, recovery and follow-up.
  • Strong Linux troubleshooting. You can reason about host networking, memory, disks, processes and drivers, and know when you need deeper evidence.
  • GPU or HPC familiarity. You have practical experience with GPU workloads or HPC clusters, including how node, interconnect or storage faults can surface as job failures.
  • Observability that leads to action. You can build useful metrics, dashboards and alerts, and avoid paging people for signals that have no response.
  • Engineering and communication. You can automate operational work, review changes and write clear incident updates and handovers for colleagues in other time zones.

Other useful experience

  • GPU-specific diagnostics and telemetry, including DCGM, Xid/ECC errors, thermal throttling or NVLink health.
  • InfiniBand or RoCE troubleshooting, RDMA counters and NCCL collective-performance analysis.
  • Kubernetes or Slurm operations, GPU resource scheduling and cluster accounting.
  • Shared-storage performance, dataset loading, checkpoint behaviour or restore testing.

We’re interested in how you think under uncertainty: the signals you trusted, the possibilities you ruled out and the change that made the system more reliable afterward.

What makes the work worthwhile

Work directly on the infrastructure that enables research, with responsibility for both the implementation and its behaviour in operation. Help shape Yggdrasil’s tools and practices through engineering work that gets used.

  • Competitive salary, with compensation discussed early in the process.
  • Technical work spanning physical hardware and the software researchers depend on.
  • Room to improve the way the platform is built, validated and operated.

Location & compensation

This role is globally remote. Hiring arrangements, time-zone overlap and the on-call rotation—including escalation and handover expectations—will be discussed during the process.

Salary will reflect the role, experience and hiring location. We’ll discuss the details directly.

Let’s talk about your work

Our process starts with a conversation, followed by a technical discussion grounded in your experience. We’ll explain the next steps and leave time for your questions. You don’t need every optional skill to start the conversation.

We welcome different routes into systems engineering. Tell us if you need an accommodation during the hiring process.

Apply for this role

START A CONVERSATION

Apply for Site Reliability Engineer

Share your background and a piece of work you’d like to discuss. A profile or résumé link is enough to get started.

Share only information relevant to your application. Privacy notice