Tag: Kubernetes

In this interview, Chandrasekhar, a seasoned software and platform engineering leader with nearly two decades of experience, shares his journey from building telecommunications billing systems to designing the infrastructure that powers one of America’s largest banks. He discusses the principles that have guided his work, the challenges of building platforms at scale, and how he integrates AI into quality engineering while maintaining compliance in regulated environments.

Q: Could you briefly introduce yourself and share your journey into software engineering and platform engineering?

A: I started as a Java developer in telecommunications, writing billing systems for British Telecom. That environment is unforgiving—processing errors have direct financial consequences for customers, so correctness at the design stage is not optional.

What pulled me toward platform engineering was a pattern I kept seeing across every organization I worked in: teams solving the same infrastructure problems in isolation, building their own tooling, making the same avoidable mistakes. The London 2012 Olympics work sharpened this. When systems serve a global audience simultaneously, the shared infrastructure underneath the application matters as much as the application itself. I started building for that layer deliberately. Twenty years on, that is still the core of what I do.

Q: Over the past two decades, what have been the most significant lessons you’ve learned from building enterprise-scale software systems?

A: The first is that what works for one team breaks at fifty. Scale does not give you more of the same problems—it surfaces assumptions nobody noticed they had made. Decisions that felt reasonable in year one become liabilities in year five, and the engineers who made them are often long gone.

The second came from Weight Watchers. The SmartPoints algorithm touched millions of daily decisions about what people ate. A subtle error there was not a software bug; it was a public health issue. I carry that framing into financial services. The downstream consequences of a bad deployment are real and human, and that changes how seriously you take the platform work upstream of every release.

The third is harder to articulate but matters as much: staying longer than is fashionable. The full lifecycle of a platform—design, adoption, scaling, failure, recovery, evolution—takes years to live through. I have seen that arc on more than one system and it has fundamentally changed how I build.

Q: You developed an automation framework that became the standard across thousands of projects at a major financial institution. What problem were you trying to solve and what impact has it delivered?

A: The core problem was that quality had no common language across the organization. Teams were running inconsistent configurations, different frameworks, different data strategies, different reporting formats. You could not compare quality signals across teams because the signals themselves were incompatible. Leadership had no reliable picture of delivery health at the portfolio level.

I redesigned the framework by embedding it directly into CI/CD pipelines, ensuring that any team capable of executing a build could also execute automation. I also rebuilt the entire test result structure throughout so that logging and failure reporting would be consistent regardless of which team or application was running the tests. The impact is measurable: the framework now operates as the standard across more than 2,500 enterprise projects, quality gates are automated rather than manually enforced, and the duplicated infrastructure investment across teams has been largely eliminated.

Q: How do you ensure software quality and governance across an organization with tens of thousands of engineers without slowing down innovation?

A: The honest answer is that speed and governance only conflict when governance is badly designed. I have been on both sides of that. I have watched teams grind to a halt because every release needed a manual sign-off from three people. I have also built systems where compliance happened automatically and nobody had to slow down to think about it.

What I did was embed the quality checks directly into the pipeline so they run without anyone asking them to. Tests fire automatically. Policy gates trigger automatically. Audit records get created automatically. When doing the right thing is also the fastest thing, engineers just do it. It is a design decision I made deliberately. The governance is still there—it just does not require a human in the loop at every step. In a bank that ships software every day, that is the only way it works in practice.

Q: Can you tell us about the cloud-native execution platform you’ve built and the challenges of operating it at such a large scale?

A: When I took on this problem, teams were running tests on static infrastructure and execution time was a significant bottleneck. Test suites had grown to the point where sequential execution was taking hours. By then automated testing stops serving its purpose—the feedback arrives after the team has already moved on and the value is lost.

I built the platform on Kubernetes so workloads run in parallel across containerized environments. In a regulated bank, though, making execution fast is only part of the problem. It also has to be auditable, and those two requirements pull in opposite directions. I originally built on AKS for orchestration, autoscaling and Azure-native integration. I am now expanding it with OpenShift support, which adds image signing, namespace isolation and role-based access controls that compliance teams require and that can be independently verified across environments.

The numbers reflect what that shift produced. The platform now executes over 137,000 validation jobs and approximately 875,000 automated tests within a thirty-day period, with an average execution time of around 26 minutes. Jobs that previously took hours now finish in under half an hour. But throughput is not the metric I care most about. It results in trustworthiness. Teams have to be able to act on what the platform tells them and that requires a level of reliability that does not happen by accident.

Q: What role is AI playing in modern quality engineering and how are financial institutions leveraging it to improve software delivery?

A: I have applied AI to strengthen the automation systems I built, while preserving the control and audit mechanisms that regulated environments require. I introduced a GenAI-based approach to extend automation coverage to approximately 800 applications that previously had none, generating tests calibrated to each application’s complexity and integrating them automatically into the test execution platform.

I built predictive models into automation workflows that surfaced failure patterns and dependency behavior under load that manual review would have missed. In one instance, a predictive model I built identified a dependency timing issue across seemingly unrelated failures, which significantly reduced investigation time and helped teams reach the root cause faster. But I did not automate decisions based on those outputs. In a regulated environment the model provides insight—the decisions and audit trails remain under human control. The capability is real; the implementation has to be redesigned around compliance requirements before it can run in production.

Q: As organizations continue their cloud transformation journeys, what are the biggest mistakes you see teams making when adopting cloud-native technologies?

A: The biggest one I keep seeing is teams treating migration as a hosting decision when it is actually an architectural redesign. I have watched teams lift applications that were built around shared memory and persistent connections, drop them into containers and then spend the better part of two years debugging failures that their old architecture never produced. The containers did not break anything—they just made visible what was always there.

The second mistake is skimping on observability—and I ran into this myself. Early in building our cloud-native platform we had a category of failures that looked completely random. Nothing in the logs connected them. I added distributed tracing and it turned out to be a container warm-up timing issue that only showed up under certain cluster loads. Without that instrumentation I would still be guessing. In a regulated environment that kind of unexplained failure is not just an ops problem—it creates a compliance problem. I built observability from the start after that.

Q: What does effective platform engineering look like in a highly regulated environment such as banking and financial services?

A: The platform has to answer a regulator’s question before it is asked. What was deployed? When? By whom? What validations ran before it reached production?

Early in my platform work, I had a release that cleared every quality gate—tests ran, coverage thresholds were met, the pipeline went green. It was stopped anyway because the test execution logs were not stored in a format that satisfied the audit requirement. The capability was there; the evidence trail was not. That incident changed how I design permanently. I now ensure every automation workflow captures logs, inputs, outputs and decision points in a structured, reviewable format. Traceability is a first-class requirement from day one, not something retrofitted after a compliance review.

Effective platform engineering here also means risk and compliance teams are design partners, not downstream reviewers. The institutions that do this well end up with platforms that are both faster to use and easier to audit, because the compliance path and the delivery path are the same path.

Q: Your research work has received recognition from global institutions and researchers. How do you bridge the gap between academic research and real-world engineering challenges?

A: The research grows directly out of production problems. While building the data quality dashboard platform, I kept hitting a question that existing literature did not answer well: how do you make quality signals meaningful to non-engineers at enterprise scale? I worked through it, published the findings and that paper has since been cited by researchers at Delft University of Technology, institutions across China and others across fifteen countries. But the more important outcome was that the design patterns I developed for the paper became the foundation for how I structured quality reporting across the platform itself. The research and the engineering fed each other directly.

As a practitioner, I am accountable for systems that run in production under load in regulated environments. I cannot theorize around a failure—I have to fix it. The research pushes me to evaluate whether the failures I encounter reflect broader, generalizable patterns. That combination produces reusable solutions with documented reasoning behind them, not one-off fixes and not just theory. I have over 100 citations from researchers across fifteen countries and that reach consistently tells me the problems I am solving inside one institution are problems the global research community recognizes as important.

Q: Looking ahead, what technology trends do you believe will have the biggest impact on enterprise software development and platform engineering over the next five years?

A: The thing I am watching most closely is AI moving from something that helps individual developers to something that sits inside the delivery pipeline itself. In the platforms I run I am already working through early versions of this: AI-assisted test generation, predictive failure analysis, automated coverage decisions. It works, but it creates a question I do not think the industry has answered yet: how do you audit a system that is partly generating itself, at enterprise scale, inside a regulated institution? That is something I am actively designing around right now.

The London 2012 experience comes back to me when I think about where this is heading. Those systems had to be correct before they went live. There was no patching it during the opening ceremony. AI in the pipeline raises the same question at a much larger scale and with far less transparency about why decisions were made. The institutions that work out governance early will have a real advantage. The ones that leave it for later will find out why that was a mistake.