Responsibilities
- Analyze and enhance end-to-end distributed training pipelines, covering data ingestion, preprocessing, augmentation, kernel performance, gradient synchronization, and checkpoint management.
- Enhance large-scale offline and batch inference workflows on petabyte-level sensor data through optimized batching, scheduling, quantization, low-precision computing, computational graph improvements, and sustained accelerator utilization.
- Develop performance models and roofline analysis to measure real-world efficiency against theoretical limits, and prioritize optimization efforts by impact and implementation cost.
- Advance multi-node training scalability by refining data sharding, parallelism techniques, collective communication protocols, interconnect usage, and resolving memory bandwidth and kernel fusion constraints.
- Maximize cluster utilization by minimizing GPU idle periods caused by input pipeline bottlenecks, storage and network latency, scheduling inefficiencies, stragglers, and job recovery delays.
- Design and implement benchmarking systems, performance monitoring tools, and regression detection mechanisms to maintain consistent performance amid evolving models and codebases.
- Partner with cross-functional engineering teams to address complex challenges in data handling and computational scale.
- Foster a team environment centered on strong collaboration, technical rigor, and forward-thinking innovation.
Work Arrangement
On-site
Work Arrangement
- The company operates on an in-office model, with full-time staff expected to work from the office five days per week.
- Limited remote flexibility is permitted, such as joining morning meetings from home or leaving early for personal obligations.
- Contract roles are not subject to the same in-office requirements.
- The company maintains physical offices in multiple global locations including Sunnyvale, Washington D.C., San Diego, Ft. Walton Beach, Ann Arbor, London, Stuttgart, Munich, Stockholm, Bangalore, Seoul, and Tokyo.
Other
- Full-time employees are expected to work from the company office five days a week.
- Occasional remote work is allowed for personal scheduling needs, such as attending morning meetings from home or leaving early for family reasons.
- Contract positions are exempt from the in-office requirement.
- The company is an equal opportunity employer and federal contractor.
- For roles based in the U.S., reasonable accommodations are available for disabled veterans and individuals with disabilities or medical conditions.
- Requesting an accommodation does not impact hiring decisions, and applicants are not required to disclose the specific nature of their disability.
