paint-brush
What AI Will Do To Data Scienceby@docligot
687 reads
687 reads

What AI Will Do To Data Science

by Dominic LigotOctober 27th, 2024
Read on Terminal Reader
Read this story w/o Javascript

Too Long; Didn't Read

Reflecting on the highly manual and contentious nature of data science and the inevitable shift to AI
featured image - What AI Will Do To Data Science
Dominic Ligot HackerNoon profile picture

I recently conducted a class on using data science for cybersecurity, focusing on the analysis of packet capture data—a somewhat technical and traditionally dry topic. The approach I shared drew from my experience in cybersecurity within financial institutions, covering core steps like exploratory data analysis, preprocessing and transforming log data, and identifying anomalies through a combination of clustering and graph network analysis.


One surprising aspect was the time I spent preparing for this session—a fraction of what I'd usually invest. AI played a significant role in streamlining the process. I used Claude to assist with coding, developing the outline, and even creating the slides. In total, the entire course was ready within 48 hours.


The session turned out to be engaging. The participants, primarily CISOs who typically don’t code, found the exercises, crafted with AI’s assistance, to be intuitive and hands-on. My goal was to immerse them in working directly with data and code. They especially appreciated the chance to explore manually what modern cyberthreat surveillance and SIEM platforms typically automate, gaining insights into the processes happening "under the hood."


My key takeaway from the class was surprisingly counterintuitive: data science, as we know it, will eventually be replaced by AI. This view might seem premature—or perhaps ahead of its time—but it’s a perspective that warrants discussion.


Warning: some of this might trigger people.

Sexiness is Baggage

For over a decade, data science has been celebrated as the “sexiest job of the 21st century.” Yet as AI rapidly advances, it’s becoming clear that the field’s underlying challenges are harder to overlook. The advent of powerful generative AI could very well be the tipping point for a discipline that, in retrospect, may have been more loosely defined and overhyped than initially acknowledged.


At its essence, data science combines computer science, statistics, and business acumen, offering organizations the promise of actionable insights from vast amounts of data. This skillset is undeniably valuable in today’s data-driven world. However, beneath its polished image, the field faces significant issues. What is often labeled as data science frequently turns out to be a patchwork of loosely related tasks that don’t always align neatly, and many professionals in the field struggle with the full breadth and complexity that the discipline demands.


The rise of AI-driven tools capable of handling data analysis, modeling, and insight generation could force a shift in how we view the role and future of data science itself. As AI continues to simplify and automate many of the foundational tasks within data science, the field may face a reckoning on what it truly means to be a data scientist in the age of intelligent automation.

The Cracks

Many data scientists, despite wielding sophisticated coding skills and digital tools, engage in work that is surprisingly manual and prone to error. Data preparation, cleansing, and analysis involve tedious, time-consuming tasks that are repetitive and mechanical. In fact, a significant amount of data science labor goes into preparing datasets—a task that often feels more like drudgery than the exciting, discovery-driven science it is made out to be. This problem is compounded by the fact that many who enter the field are, at best, amateurs. Having taken a few online courses in Python or R, these "data scientists" are often unprepared for the rigors of the role. Data science isn't just coding. It involves deep analysis, contextual understanding, and the ability to present insights to non-technical audiences. In truth, it is more of a research job, requiring a blend of creativity and analytical thinking that many in the field simply don’t possess.


Furthermore, many data scientists have developed a sense of entitlement, expecting high salaries and lucrative packages just by virtue of their title. This attitude is turning off companies, especially in sectors where cost efficiency is paramount. I have met firms that once rushed to hire data scientists but are now reconsidering. Why pay high wages to someone who spends most of their time wrestling with data cleaning, when AI can do it faster, better, and at a fraction of the cost?

AI Who?

As I personally experienced writing the class, Generative AI has evolved into a powerful force in the very areas where data science is weakest. Tasks like data preparation, cleansing, and even basic qualitative analysis—activities that consume much of a data scientist’s time—are now easily automated by AI systems. What’s worse (or better, depending on where you stand) is that AI is faster, more accurate, and less prone to human error or fatigue.


For many data scientists, this can be terrifying. After all, these tasks represent the bulk of their day-to-day work. Data cleansing, for instance, is notoriously time-consuming and prone to mistakes, but AI can now accomplish it with a few clicks and near-perfect precision. Data scientists often complain about these grunt tasks, yet they are fundamental to their roles. As AI systems improve, the need for humans to do these jobs dwindles. It’s no surprise that much of the vocal criticism against AI comes from data scientists themselves. They see the writing on the wall and fear for their jobs.

The Pettiness

To make matters worse for data scientists, the field hasn’t made significant progress in recent years. Despite its meteoric rise in popularity, data science is still plagued by inefficiencies, errors, and a lack of clarity on what exactly it should entail. It was once believed that more sophisticated tools and better training would evolve the field, but this hasn’t materialized to the extent expected. In contrast, AI has steadily improved. Machine learning algorithms, natural language processing, and generative models are rapidly evolving, leaving traditional data science in the dust.


Again, the high salary expectations of data scientists compound the issue. Companies that might have once tolerated inefficiencies are now realizing that AI can replace a lot of the grunt work without the hefty price tag attached to human labor. With AI becoming more adept at performing key tasks like analysis, forecasting, and even presentation, the manual nature of data science is becoming increasingly redundant. Many companies will realize that what used to require a team of data scientists can now be handled more efficiently by AI-powered tools.

The Shift

The reality is that data science, as traditionally defined, is on the brink of obsolescence. With generative AI advancing at an astonishing rate, the demand for human data scientists in their current form will likely decline. This doesn’t imply that humans have no role in data-driven decision-making, but the classic “data scientist” role may soon be a concept of the past. What’s needed now are professionals skilled in collaborating with AI, harnessing its capabilities while concentrating on strategic thinking and complex problem-solving at a higher level.


AI isn’t the end of analytics, insights, or decision-making—it represents their evolution. The current field of data science risks becoming obsolete if it does not evolve in step. AI is already revolutionizing industries, and data science must adapt or risk being overtaken by this wave. Ultimately, the question may not be whether AI will eliminate data science but whether data science ever fully delivered on its promises.


Or perhaps the distinction doesn’t even matter if we finally move beyond the “data science” hype and embrace AI as the next logical progression.



About Me: 25+ year IT veteran combining data, AI, risk management, strategy, and education. 4x hackathon winner and social impact from data advocate. Currently working to jumpstart the AI workforce in the Philippines. Learn more about me here: https://docligot.com