Author: vitorcherulli

The machine learning industry is grappling with a problem that rarely makes headlines but consistently derails production deployments: data pipeline failure. Gartner’s July 2024 forecast projected that at least 30% of generative AI projects would face abandonment after proof of concept, with poor data quality and inadequate infrastructure visibility among the leading causes. For organizations building perception models — systems that must interpret human movement, expression, and gesture with millisecond precision — the consequences of pipeline failures are immediate and expensive.

Rahul Rathi encountered this problem at scale during his tenure at Meta, where he was responsible for delivering the data infrastructure that powered the face tracking and eye tracking systems at the heart of Quest Pro, Meta’s flagship mixed reality headset launched in October 2022. The measurement framework he designed to solve it — treating the entire pipeline as a connected, instrumented system rather than a collection of discrete stages — was subsequently adopted as the internal standard for data-centric machine learning development across Meta’s perception teams, influencing how multiple engineering organizations approached pipeline measurement for production-critical AI models. That same methodology now informs his work governing frontier model training infrastructure at Microsoft AI, where he currently serves as a Principal AI Data Ops Technical Program Manager.

Measuring What Others Overlooked

Face tracking and eye tracking systems require extraordinary data precision. A single corrupted dataset can degrade model behavior in ways that are invisible during training but immediately apparent to users — avatars displaying expressions that feel subtly wrong, or gaze models that fail under real-world lighting conditions. Traditional machine learning pipelines measured data volume and final model accuracy but provided little visibility into degradation occurring between collection and training. Teams frequently discovered quality problems only after dedicating weeks to training runs using compromised inputs.

Rathi’s response was to redesign how the pipeline itself was measured. He introduced a data funnel framework that defined specific stages tracking data flow, quality degradation, drop-off rates, and readiness from initial acquisition through model training. Each stage included quantitative thresholds determining whether data met requirements for progression. Teams could identify bottlenecks and quality regressions far earlier than existing monitoring permitted, shifting measurement upstream to prevent the costly late-stage rework that had characterized previous programs.

“We defined intermediate indicators revealing where data degraded, stalled, or failed to meet readiness criteria,” Rathi explains. “Teams could correct issues in hours rather than discovering problems after weeks of wasted compute cycles.”

The practical outcome at Quest Pro was measurable. The framework accelerated model readiness for the October 2022 launch and subsequently became the recognized internal standard across Meta’s perception teams. Engineers working on hand tracking, Codec Avatars, and augmented reality applications implemented the same funnel-based approach — establishing a consistent methodology across Meta’s computer vision data infrastructure that represented a departure from conventional practice. The pattern of building measurement upstream into the pipeline, rather than relying on downstream model accuracy as the primary quality signal, influenced how subsequent programs were structured from their inception.

The broader significance sits within a well-documented industry challenge. The global VR headset market reached $12.6 billion in 2024, with analysts projecting growth to $38.4 billion by 2030 as enterprises deploy immersive technologies requiring sophisticated perception models. The competitive pressure on perception model quality — and therefore on the data infrastructure supporting it — is only intensifying.

Infrastructure Economics Demand New Approaches

Quest Pro represented one application of Rathi’s measurement thinking. His subsequent work at Microsoft AI addresses challenges operating at vastly different scales. Training large language models and multimodal systems demands coordination across thousands of GPUs consuming electricity measured in megawatts. Industry estimates place OpenAI’s compute spending at approximately $3 billion for training in 2024, with inference costs adding another $1.8 billion. Individual H100 GPUs carry purchase prices near $25,000 per unit. Even at cloud pricing that declined from peak rates near $10 per hour to approximately $3 per hour across 2024 and 2025, training frontier models requires commitments measured in tens of millions of dollars.

These capital commitments heighten the cost of inefficiency dramatically. Industry research has documented that GPU idle time in large-scale training environments commonly represents 20 to 40% of total compute capacity — waste that compounds across thousands of GPUs operating continuously and remains largely invisible without the right measurement infrastructure.

Rathi developed granular compute efficiency metrics that expose GPU underutilization, idle time, and scheduling inefficiencies across distributed training systems. His workload-aware metrics tie directly to training behavior, connecting infrastructure usage to specific model training decisions. The practical outcome was a fleet-wide GPU utilization improvement from 75% to 95% across Microsoft AI’s compute infrastructure — a 20-percentage-point gain that, across a GPU fleet of the scale Microsoft AI operates, represents a capital efficiency improvement measurable in the hundreds of millions of dollars based on publicly available GPU pricing and cluster scale data.

“GPU availability, utilization efficiency, and orchestration have become critical bottlenecks,” Rathi notes. “We lead cross-functional programs coordinating GPU cluster readiness, Kubernetes and SLURM-based scheduling, and distributed training workflows across research, data engineering, and infrastructure teams.”

The methodology is structurally parallel to his earlier work at Meta. Where the Quest Pro framework tracked data degradation across pipeline stages, his compute metrics track utilization degradation across training workflows — shifting organizations from reactive cost control to proactive efficiency engineering while preserving model performance and delivery timelines.

A Field Divided on Priorities

Not everyone embraces infrastructure optimization as the primary lever for improving machine learning systems. Dr. Andrew Ng, Adjunct Professor of Computer Science at Stanford University and founder of DeepLearning.AI, has made a widely noted distinction between model-centric and data-centric approaches to AI development. Through his advocacy for the Data-Centric AI movement and published course materials at DeepLearning.AI, Ng has argued for shifting the industry’s focus toward systematic data quality improvement rather than model architecture iteration alone — a position that aligns with Rathi’s pipeline measurement philosophy. At the same time, Ng and others have noted that measurement systems optimized for current architectures may not generalize as the field evolves, and that organizations risk over-investing in infrastructure tuned to today’s problems.

The concern has real-world grounding. DeepSeek’s V3 model reportedly achieved dramatic reductions in training costs through architectural innovations rather than hardware optimization. Software advances can and do outpace infrastructure efficiency gains, suggesting the two approaches are complements rather than substitutes.

Rathi acknowledges the tension directly. “Visibility into resource utilization enables faster experimentation,” he responds. “Whether training transformers or future architectures, teams benefit from understanding where compute goes and why systems underperform expectations. Measurement is not an alternative to architectural innovation — it is what makes architectural innovation faster.”

Industry analysts project AI infrastructure spending will reach approximately $7 trillion by 2030, with inference workloads consuming an increasing share of total compute. The organizations best positioned to operate at that scale will be those that have built the measurement infrastructure to govern it.

Implications for AI’s Industrial Future

Data pipeline complexity will intensify as AI adoption expands beyond technology companies. Healthcare organizations training medical imaging models face stringent privacy requirements that complicate data collection. Financial institutions building fraud detection systems must satisfy regulatory oversight demanding explainability. Manufacturing companies deploying computer vision for quality control require models robust to real-world variation absent from controlled training sets. Each domain introduces specialized requirements that generic infrastructure cannot address without measurement frameworks adapted to their specific constraints.

The machine learning market reached an estimated $113 billion in 2025, with analysts forecasting growth to $503 billion by 2030. Gartner’s analysis of generative AI project failures points directly to inadequate data quality and infrastructure visibility as primary contributors to abandonment — exactly the gaps that Rathi’s frameworks address. Real-world deployments encounter messy data, inconsistent formats, and unexpected edge cases that carefully curated training sets rarely capture, making upstream measurement not a luxury but a precondition for production reliability.

His work also carries governance implications extending beyond efficiency. The European Union AI Act demands technical documentation proving compliance with safety and fairness standards. Measurement infrastructure that surfaces bias in training data, detects distribution shifts, and provides audit trails documenting development decisions is becoming a regulatory requirement, not merely an engineering best practice. The infrastructure Rathi has built — providing auditable evidence of data quality decisions, pipeline behavior, and compute governance across training workflows — represents exactly the kind of technical documentation these frameworks require. Organizations building frontier AI systems without it will face increasing difficulty demonstrating compliance as enforcement scales.

“We help advance responsible AI practices by embedding quality, efficiency, and governance considerations into platform and infrastructure decisions,” Rathi observes. “Continuous improvement of operational rigor across the AI lifecycle requires visibility into what is actually happening inside these complex systems.”

A Consistent Methodology Across Scales

The trajectory from Quest Pro’s perception models to frontier language models illustrates how Rathi’s measurement approach scales across applications. Techniques developed for tracking face data quality translate directly to monitoring the massive datasets used in language model training. Efficiency metrics originally designed for GPU utilization in computer vision workloads apply equally to transformer training runs consuming weeks of continuous compute. The fundamental principles remain consistent even as specific implementations adapt to new contexts.

As training costs escalate, compute efficiency metrics have transitioned from operational considerations to strategic imperatives. Organizations that can demonstrate concrete return on infrastructure investment — through measurable utilization improvements and deferred capital expenditure — are better positioned to justify continued AI investment to boards increasingly scrutinizing spending.

“Looking ahead, my work helps prepare Microsoft AI for future industry needs by shifting compute platforms from reactive scaling to proactive, efficiency-driven design,” Rathi reflects. “This foundation supports sustainable AI growth, faster experimentation, and responsible deployment as model sizes and demand continue to increase.”

Through his contributions across data-centric machine learning and large-scale compute efficiency, Rahul Rathi has advanced the operational foundations upon which production AI systems depend. The frameworks he developed — first under the competitive pressure of a flagship consumer hardware launch at Meta, and now at the frontier of large language model development at Microsoft AI — have moved from internal tools to adopted standards. In an industry where the gap between research capability and production reliability remains one of the defining challenges, his work addresses that gap at the infrastructure layer where it is hardest to solve and most consequential when left unaddressed.