"The Future is Where Any Business Gets Insights From Their Data Easily" says Aniruth from Databricks

At HackerNoon, we value stories about building groundbreaking tech. These questions aren’t just about AI mechanics—they’re about the innovation, challenges, and creativity of bringing cutting-edge tools to life, a perfect fit for our community of tech leaders, builders, and future-minded readers.

Introduction

My name is Aniruth. I work on the storage team at Databricks, where we work on enabling customers to save large amounts of data in an open, scalable format with our Data Intelligence Platform. Specifically, I work on our interoperability efforts with Delta Lake and Apache Iceberg.

HackerNoon: What’s the main problem your AI product is designed to solve, and what is effective about your approach to solving it?

Aniruth: Databricks unifies data and AI to give customers actionable intelligence—what we call data intelligence. This includes ingesting large amounts of data, ETL, large-scale storage, business intelligence queries, and AI workloads. The techniques used in machine learning in the past decade have been around since the 1980s; the rise of big data made it feasible to run algorithms at scale.

Techniques like few shot prompting or RAG rely on high-quality data. Models that have better data win against those with better architectures. Databricks has put significant investment into leading efforts in the data space, pioneering the lakehouse architecture with open data formats and open governance, where customers are able to get the best insights with the best performance from data lakes.

What criteria did you use to select the specific AI models for this product, and how does your company make decisions like this?

We use AI models in a number of ways in the product - such as Llama 3 for the AI Assistant. We believe in an open data and AI ecosystem and encourage our customers to use any model of their choice. We help make sure customers have end-to-end governance across the AI lifecycle regardless of the models they use, so they can focus on making their models purpose-built for their use cases.

How do you ensure your product delivers accurate and unbiased results to users?

We have spent significant effort and investment in prioritizing accuracy and unbiased answers for AI usage within our products, and continue to conduct frequent testing.

What is your day in the life like as a product manager?

The data and AI space is rapidly evolving, so it’s very important to keep up to date. My day can include talking to customers, market analysis, putting together a product requirements document, or preparing marketing materials. My favorite part is getting to make diagrams illustrating how things will work, as it’s pretty fun to transform an idea into a visual.

What’s the next big breakthrough in AI that everyone should be watching for?

There’s a lot of big breakthroughs coming soon. One that I’m particularly interested in is the hyperpersonalization of content. For the past decade, ads have been fine-tuned to the specific watcher. Some elements of content have been tuned, such as what thumbnail Netflix shows a user, but the actual content (the video itself) has not been. It’ll be interesting to see how directors/producers balance telling the story they want to and the user’s interests.

What’s been the biggest challenge you’ve faced in bringing this AI product to market, from concept to launch?

I work on large-scale data storage, which can be very confusing to understand. We have various AI optimizations on data, but there’s often questions about when these optimizations are run, how they work, what they don’t cover, etc. With these kinds of questions, it is important to ensure we have clear, consistent messaging about what we are building and why. I’ve found that explaining the cause of limitations resonates very well with customers.

How do you envision AI evolving to better understand and respond to human emotions, and what challenges or opportunities do you see in that area?

Multimodal models are going to get significantly better in the coming years, which will change our primary mode of interaction with AI. Figuring out human emotion is significantly easier from visual or audio information compared to text. I think there's an opportunity to create more natural interactions in a wider array of scenarios.

What performance metrics or KPIs do you track to gauge the success of this AI product?

We typically want to see good feedback and usage. I talk to customers pretty often to get a sense of how and why they think about our products, which is key to explain why we see certain trends in metrics.

How do you approach designing a product experience that’s not only functional but also engaging and memorable for users?

Large-scale data products are notoriously challenging to use. Simple examples are easy to set up, but production workloads typically involve confusing configuration and code. It’s been a high priority for me to build out functionality that customers need, while making the product very simple to use.

The future is where any business gets insights from their data easily. In the current world, data-driven business insights are typically restricted to the biggest companies - but even they would prefer a simpler experience.

In the long-term, how do you think AI will help the individual reach one’s full potential?

For the individual, I’m very excited about AI integrations into hardware. Up to now, we’ve largely seen AI in software applications like websites. There’s many larger applications of building devices that utilize AI, and we’re already starting to see some of those implications in cars and phones.

Where do you see the product evolving in the next few years, and what features are you most excited to add?

Databricks is on a path towards becoming increasingly simple and more powerful at the same time. There are a lot of efforts we are working on across the board, from making big-scale data and compute easy to work with to improving performance on queries and workflows. Personally, I think we have some exciting features coming soon throughout the product that make workflows way easier with AI. Examples include AI-generated comments on data, AI code suggestions in the notebook editors, and AI interfaces to chat with data (for example, Databricks AI/BI Genie).

What’s your perspective on AI’s impact on jobs, and how do you address those concerns in your product strategy?

There are concerns over whether AI would reduce the number of jobs. Our products are designed to increase valuable insights, which often come in conjunction with users. For example, with AI/BI Genie, users can create interfaces on their own data. This is a magical experience, where users can ask questions and get answers specific to them. In fact, users can even check the SQL being used to confirm it’s what they’re looking for. This is collaborative with analysts, reducing the time it takes them to go from idea to insight.

What user feedback surprised you and led to changes in your roadmap, strategy or product experience?

One major surprise for me was the complexity that some of the larger companies have. This introduces requirements into the product that I wouldn’t have considered on my own. A common example is thinking of migration strategies when introducing a new product. Typically, large companies will have either put together existing technologies (commonly open-source software) or have built custom software to solve the problem that our new product offering tackles. It usually takes a bit of time to understand why and how these are put together to ensure we have a solution that covers all possibilities.