Roblox

Senior Software Engineer

2023 — present · San Mateo, California

Agent infrastructure

  • Built the Slack integration and Kubernetes clusters for our internal agent platform. About 1,000 people use it each month, and the agent gets tagged about 2,000 times a day in Slack.
  • Built the Kubernetes sandbox provider and integrated agent-sandbox. A pool of ready sandboxes brought startup below one second.
  • Wrote Temporal workflows to shut down idle sandboxes and resume sessions. Workspace files stay on EBS between visits to the same Slack thread.
  • Built a Slack bot that checks stuck training jobs and answers debugging questions from their logs (6,400 invocations).

GPU inference clusters

  • My teammates and I built the GPU serving clusters behind Roblox’s chat filters. I wrote the Terraform and Kubernetes deployment code that brought up a new cluster in about 30 minutes.
  • Designed load balancing and automatic failover, fixed downtime during NGINX redeploys, and spread replicas across nodes so we could upgrade the whole cluster while it served live traffic.
  • Cut the time to add serving capacity from eight minutes to three by using local NVMe storage.
  • Our new clusters ran larger chat filters at half the cost. The model that detects personal information in chat handled 370,000 requests per second at peak.

Batch inference and training on Ray

  • Adapted KubeRay to Roblox’s schedulers and built the Go job service and dashboard. The platform handled more than 100 jobs a day.
  • Enabled RDMA for distributed training and helped 3D researchers process petabytes of data on Ray.
  • Put a code editor on the Ray head node so researchers could try a change in seconds. They used to rebuild a container and wait over ten minutes. The GPU workers could scale to zero while the editor stayed open.

Labeling 3D assets on idle GPUs

  • Managed GPU allocations across fleets totaling more than 10,000 GPUs.
  • Used spare GPUs to label 100 million assets for a 3D foundation model. Labeling jobs could pause when training reclaimed the GPUs and resume later.

Talks and code