Build Scalable LLM Training Infrastructure
$25-50 USD / hour
Posted half a minute ago
•
Ends in 6 days
$25-50 USD / hour
We are seeking a highly experienced Distributed AI Infrastructure Engineer to design, develop, and optimize large-scale AI training systems. The ideal candidate will have deep expertise in distributed computing, GPU architecture, high-performance networking, and large language model (LLM) training infrastructure. This role involves solving complex engineering challenges related to multi-node GPU synchronization, distributed fault tolerance, memory optimization, and high-performance AI workloads. Key Responsibilities Design and implement distributed training infrastructure supporting 100+ GPUs across multiple compute nodes. Develop and optimize large-scale LLM training pipelines using PyTorch, DeepSpeed, FSDP, and Megatron-LM. Implement advanced parallelism strategies, including tensor, pipeline, and data parallelism. Optimize NCCL communication, GPU memory utilization, and inter-node network performance. Build fault-tolerant training systems with automated checkpointing and recovery. Investigate distributed deadlocks, GPU synchronization failures, and performance bottlenecks. Develop scalable GPU orchestration solutions using Kubernetes, Slurm, or Ray. Implement real-time monitoring and performance analysis for distributed AI workloads. Ensure training consistency, reliability, and efficient resource utilization. Required Qualifications 5+ years of experience in distributed systems, machine learning infrastructure, or high-performance computing. Strong proficiency in Python, C++, and PyTorch. Advanced understanding of CUDA, NCCL, GPU memory management, and distributed communication. Hands-on experience with multi-node, multi-GPU training. Experience with NVIDIA GPU architectures, InfiniBand, and high-speed networking. Strong knowledge of Linux, Docker, Kubernetes, and cloud infrastructure. Proven ability to troubleshoot complex distributed training failures. Preferred Qualifications Experience training or fine-tuning models with 70B+ parameters. Experience with NVIDIA H100/H200 GPU clusters. Knowledge of custom CUDA kernels and Triton. Experience optimizing large-scale AI training efficiency. Familiarity with distributed checkpointing, ZeRO optimization, and hybrid parallelism. Technical Challenge The successful candidate should be capable of diagnosing and resolving complex distributed training failures involving GPU communication deadlocks, inconsistent checkpoints, network interruptions, and unexpected model convergence issues. Our Goal Build a highly scalable, fault-tolerant AI training infrastructure capable of efficiently training next-generation large language models while minimizing computational costs and maximizing system reliability.
Project ID: 40822730
About the project
Open for bidding
Remote project
Active 57 yrs ago
Place your bid
Bid amount Email addressBenefits of bidding on Freelancer
Set your budget and timeframe
It's free to sign up and bid on jobs
内容来自 Freelancer News,版权归原作者所有。
阅读原文 ↗