As AI becomes adept in a growing range of computing tasks, the people working to develop it could soon find themselves out of a job. As artificial intelligence enters more and more of today’s technologies, aspiring algorithm engineers face a unique dilemma. On the one hand, they must compete to be qualified enough in AI to have development prospects; on the other, they risk becoming obsolete at the hands of the same AI models they are competing to work on. AI today has become such a popular field among developers that those returning to class reunions who have not pursued it risk becoming the butt of their classmates’ jokes. Far from being the new masters of an increasingly automated world, though, algorithm engineers are among those whose work could soon be replaced by the same development tools they are building to simplify it. As competition rises for a diminishing number of human jobs, it is more important than ever that emerging developers strategize to maximize their career opportunities. In this article, Ant Financial’s senior algorithm engineer Zhao Yiming(赵一鸣) surveys the AI battlefield and offers his advice for newcomers hoping to prevail in algorithm work for the long term. The AI Boom Today In recent years, an avowed interest in artificial intelligence has become de rigueur in computing circles. Simultaneously, HR desks at technology companies are stacked with resumes full of keywords like AI, parameter tuning, CNN, LSTM, and image recognition. Taking the latter as an example, even primary school students now have the opportunity to work with microcontrollers and computer vision technology. Throughout the industry, AI departments are shifting from research-centric institutions on the periphery of business to front and back office structures; those on the front end are now business departments with clear KPIs and intense external competition, while those on the back end are as integrated into company infrastructures as fundamentals like databases. Whereas a monastic life of research was once the expectation for AI specialists, practical deployments are now prevailing with unforeseen immediacy. Despite how quickly these developments have emerged, clear signs of a crisis in two parts are already apparent. First, a flood of cheap labor from recent graduates and trainees has saturated today’s market, ensuring a future of intense competition for human algorithm engineers. Second, the algorithms that engineers work on every day are bound to eliminate an untold number of their creators from the talent pool. Given these two patterns, the AI talent market is bound to become a hotly contested battlefield in the near future. When Children Can Code: Origins of a Crisis Steady development of tools and frameworks has made coding for model designs increasingly concise. A decade ago, using C++ and matrix libraries to achieve gradient descent from scratch involved thousands of lines of code, generally demanding at least a high school education. Today, dozens of Keras and graphical model building tools are available to help, and even elementary students can design a usable binary classification model. Powerful class libraries have a tendency to consume more disparate knowledge points and mask internal complexity, bringing considerable inertia to the user process. While many invest in mastering the use of models, master-level practitioners will tend to use underlying technologies that make more logical sense. The nature of deep learning itself makes a sharp divide between developers with true mathematical talent and those without. Unlike SVMs and decision trees, strict mathematical arguments require extremely high-level abstraction skills, due to the high number of nonlinear, complex hierarchical relationships and input signals involved. The reasons why a method is sound or the types of data it is suited to often evade its authors, and many state-of-the-art approaches in use are the result of raw experience and tricks passed by word of mouth among developers, as opposed to rigorous theory. Even the reasons why batch normalization is effective (while using formulas that require only a middle school math background to understand) are the subject of much debate. While a small handful of top experts can delve into models and perform the mathematical analysis and proof to explain their workings, most people in the industry enter the long platform development stage of algorithm engineering soon after making a start, following the dubious path of “parameter tuning” to uncertain ends. The AI learning curve In the AI learning curve pictured above, the leftmost portion of the curve indicates the steep but brief entry stage in which study of basic matrix theory, calculus, and programing is essential. Following this, a long period of mid-level development on platform development leads steadily toward greater and greater challenges. While user-friendly tools can shorten the entry period and reduce much of the learning curve for these two stages, the rightmost part of the curve invariably remains steep, separating top experts from a large field of less advanced professionals. In short, getting started in algorithm engineering is easy, but mastering the practice is extremely difficult. A similar curve could be used to describe the respective income levels of AI talents at each stage of their progress. The previously mentioned crisis in the AI talent market most directly impacts those in the platform stage, as they have little understanding of theory and a high dependence on tools, as well as being highly replaceable. When the current boom in AI has eventually passed, those who have not progressed from this stage will be left with little genuine skill or knowledge to proceed from. As the market evolves and new hires continue to prove capable of providing code for workable models, the outlook remains bleak as to whether companies will be willing to invest in the small margin of performance senior engineers can add to their businesses — in many cases, a mere one percent. The Current State of Obsolescence Somewhat ironically, the first to be replaced by AI models are most likely to be the same algorithm engineers who build them. Algorithmic posts are generally less resistant to replacement than engineering posts. Current technology makes it nearly impossible to generate a set of software apps or services, due to the complexity of business requirements. (If this were possible, we would already have reached the era of autonomous AI.) According to the nature of the data involved, automatically generated end-to-end models are gradually becoming available for utilization at the industry level in various fields, namely image and speech applications and advertising recommendation (where models can be applied directly). As theory and experience improve, humans are becoming increasingly replaceable in these areas. As automatic feature generation and optimization advances, feature engineers are increasingly unjustifiable in their positions. Similarly, parameter tuning engineers are bound to be out of business as deep network technology rises to business challenges and parameter search becomes more convenient under AutoML. The data reflow and prediction links that formerly took great effort to build have become basic components of a company, meaning data engineers are prone to being laid off. The refrain of the moment for many in a position to hire has become, “If a machine can do it, why do I need you?” Current trends in AI research papers indicate advertising recommendation models are maturing quickly. Many standalone techniques have been brought together in complete methodologies, thus entering the platform stage of algorithm engineering. The next field to be overtaken should be image generation, while text generation remains the last dry land for algorithm engineers, given its inherent abstraction and ambiguity. Nevertheless, this too should undergo an explosive breakthrough within the next five years. Staying Relevant: A Guide to Optimizing Career Development Artificial intelligence has enjoyed some five years of spotlight to date. While its future remains uncertain, it is bound to become a more polarized field. Basic functions will likely remain in the domain of general programmers, while research on more complex models and even independent AI will become the dominion of senior experts. At technology companies, demand for traditional software development and product design far exceeds that for AI algorithms. These algorithms are a luxury, rather than a crucial form of assistance in moments of crisis. Even the best algorithms cannot remedy a backward business model, and when the economy of such a model declines it is the people contributing to fancy but irrelevant algorithms who are dismissed first. While top algorithm experts need not worry about such developments, the vast majority will struggle to find a reliable course and optimal career path. The following sections offer advice that algorithm engineers should find useful in establishing a sense of direction. Master key principles and underlayers Anyone seeking a future in algorithm engineering should read core codes such as TensorFlow’s at least once in their entirety, with the most important such codes depending on their concentration areas. Even where there is no strict basis in theory for a particular effort, no one should ever code blindly. Potential pitfalls to avoid in a career include dependence on the ease of use that tools can offer. Engineers should instead become familiar with the character of each function in the toolbox and gain enough sense of the data flowing in models to quickly remove unreliable parameters at the time of tuning. One major gain of algorithm design is its scientific spirit and experimental thinking, which is difficult to cultivate by way of engineering. When reading papers, for example, it is a mistake to think that the introduction, model design, and results of an experiment are sufficient for exploration when the AB experimental design is more likely to form the core of the paper; specifically, this is where it can be determined whether the experimental design is rigorous, the sample unbiased, the core effect reasonable, and the conclusion provable. Behind a line of code or a parameter modification you are likely to find hard thinking and experimentation, and algorithm design requires rigorous and meticulous thinking; even those who will not work on algorithms in the future will find such experiences an invaluable asset. Become oriented with the field The algorithm engineering field, the business associated with it, and the people engaged in it are all essential to understand as early as possible. AI is simply a tool, and its level of abstraction means it will not soon become a panacea across different fields. If you cannot compete with AI experts in depth, it is important to develop a strong data sensitivity in your service field and become a cross-industry expert. Many successful cases in areas such as AI + Finance, AI + Medicine, and AI + Sports illustrate the importance of becoming familiar with both the data and the humanity behind a specific application setting or business area, which machines will not be able to diminish in the near future. Many opportunities for work following peak crowding in the talent market are likely to come by way of these cross-industry combinations. Even senior experts face some amount of confusion today. As people from algorithm backgrounds grow and diverge in their career paths, this problem seems one common thread of experience that will challenge engineers in years to come. Some in the rising generation of engineers born during the 1980s have already reached positions as vice presidents for major companies, while others have taken the lead of smaller teams or continue to work at the grassroots level in startup environments. Faced with the likelihood of a coming crisis, their paths should offer unique advantages and opportunities alongside the challenges the entire field will face together. (Original article written by Zhao Yiming赵一鸣) Alibaba Tech First hand and in-depth information about Alibaba’s latest technology → Facebook: . Twitter: . “Alibaba Tech” “AlibabaTech”

Staying Relevant in the Age of AI: An Alibaba Guide for Algorithm Engineers

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development

10 Things Everyone Should Know About Machine Learning

10 Repositories that Will Transform the Way You Approach Technical Interviews

10 (Free) Data Structure and Algorithm Courses Junior Developers Should Explore

10 Data Structure & Algorithms Books Every Programmer Should Read

The Noonification: How to Develop a DSL in Kotlin (12/12/2023)

A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development

10 Things Everyone Should Know About Machine Learning

10 Repositories that Will Transform the Way You Approach Technical Interviews

10 (Free) Data Structure and Algorithm Courses Junior Developers Should Explore

10 Data Structure & Algorithms Books Every Programmer Should Read

The Noonification: How to Develop a DSL in Kotlin (12/12/2023)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps