Build Scalable LLM Training Infrastructure

$25-50 USD / hour

Posted half a minute ago

•

Ends in 6 days

$25-50 USD / hour

We are seeking a highly experienced Distributed AI Infrastructure Engineer to design, develop, and optimize large-scale AI training systems. The ideal candidate will have deep expertise in distributed computing, GPU architecture, high-performance networking, and large language model (LLM) training infrastructure. This role involves solving complex engineering challenges related to multi-node GPU synchronization, distributed fault tolerance, memory optimization, and high-performance AI workloads. Key Responsibilities Design and implement distributed training infrastructure supporting 100+ GPUs across multiple compute nodes. Develop and optimize large-scale LLM training pipelines using PyTorch, DeepSpeed, FSDP, and Megatron-LM. Implement advanced parallelism strategies, including tensor, pipeline, and data parallelism. Optimize NCCL communication, GPU memory utilization, and inter-node network performance. Build fault-tolerant training systems with automated checkpointing and recovery. Investigate distributed deadlocks, GPU synchronization failures, and performance bottlenecks. Develop scalable GPU orchestration solutions using Kubernetes, Slurm, or Ray. Implement real-time monitoring and performance analysis for distributed AI workloads. Ensure training consistency, reliability, and efficient resource utilization. Required Qualifications 5+ years of experience in distributed systems, machine learning infrastructure, or high-performance computing. Strong proficiency in Python, C++, and PyTorch. Advanced understanding of CUDA, NCCL, GPU memory management, and distributed communication. Hands-on experience with multi-node, multi-GPU training. Experience with NVIDIA GPU architectures, InfiniBand, and high-speed networking. Strong knowledge of Linux, Docker, Kubernetes, and cloud infrastructure. Proven ability to troubleshoot complex distributed training failures. Preferred Qualifications Experience training or fine-tuning models with 70B+ parameters. Experience with NVIDIA H100/H200 GPU clusters. Knowledge of custom CUDA kernels and Triton. Experience optimizing large-scale AI training efficiency. Familiarity with distributed checkpointing, ZeRO optimization, and hybrid parallelism. Technical Challenge The successful candidate should be capable of diagnosing and resolving complex distributed training failures involving GPU communication deadlocks, inconsistent checkpoints, network interruptions, and unexpected model convergence issues. Our Goal Build a highly scalable, fault-tolerant AI training infrastructure capable of efficiently training next-generation large language models while minimizing computational costs and maximizing system reliability.

Project ID: 40822730

About the project

Open for bidding

Remote project

Active 57 yrs ago

Place your bid

Bid amount Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

It's free to sign up and bid on jobs

内容来自 Freelancer News,版权归原作者所有。

阅读原文 ↗

把喜欢的内容,留在你的新标签页。

打开我的订阅