Behind the Scenes on 11.11: A Night in the Life of an Alibaba Engineer

Diligent technicians are rarely under the lime light. In this article, we look at what it is like to work at Alibaba during the 11.11 Global Shopping Festival for O&M engineer Ren Ruxian.

Alibaba’s 11.11 Global Shopping Festival, set each year on China’s November 11 Singles’ Day, has garnered widespread global attention in recent years. On the eve of this year’s festival, which grossed a record-breaking 30.8 billion dollars, announcements declaring it could be seen in renowned public spaces from New York’s Times Square to the lawn of the Eiffel Tower in Paris, and even reflected in London’s Thames River.

As it gains recognition, some may wonder what it is like to work behind the scenes of such a massive festival. Today we follow the event through the eyes of Ren Ruxian, an on-duty O&M engineer who worked overnight to ensure the online festivities went on.

Wind turbines mark the landscape of windy and remote Zhangbei County in China’s rural Hebei Province

Guarding the Silent Vent

Removed from the bustling urban and online shopping scene, Ren Ruxian dutifully monitors the remote “silent vent” known as Alibaba’s Zhangbei Data Center, located 300 kilometers outside Beijing city.

Known for its music festivals, Zhangbei County is also home to Alibaba’s technological heart, where online transactions are processed for the 11.11 Festival. The center holds the key to the festival’s success, with each blinking monitor light in its servers representing billions of data points being processed online.

Ren Ruxian monitors a facility at Zhangbei Data Center

Oblivious to the billions of consumers engaged in the annual shopping frenzy, Ren cares only about the rows of machines in front of him.

“These machines are like my wife,” he said.

In stark contrast to the hordes of enthusiastic shoppers online, Ren did not buy a single item during the last 11.11 Festival. Instead, he worked three days straight with fellow team members who likewise endured the stress on high alert. Under their fast-paced daily work conditions, they relied on short texts to communicate efficiently.

The center’s servers are far busier than usual during the festival, and specialized equipment is required to keep the servers stable as billions of worldwide transactions are processed simultaneously. To handle the volume, the center houses the world’s largest immersion and air server cooling systems, which together address the risk of servers overheating at such a large data center.

Ren lifts a submerged server component from the server immersion cooling system for inspection

To prevent blackouts, the data center has also invested heavily in a diesel power generator system. The center’s maintenance staff’s voices are drowned out by the loud roar of dozens of diesel power generators running concurrently, forcing them to communicate mostly in sign language. Because China’s power grid network has become more comprehensive, the diesel fuel housed at the data center can probably last a lifetime, according to Ren.

“However, we still need these generators,” he said. “A blackout would cost us tens of thousands of yuan per second.”

Ren monitoring equipment manually during a routine check

Compared with human interactions, Ren is more accustomed to communicating with machines. Despite the installation of intelligent devices to assist equipment monitoring, Ren prefers the traditional five-sense approach of viewing, smelling, and listening to a piece of equipment. Through his senses, Ren can differentiate the sounds and scents of dozens of machines.

In the course of a shift, the engineer can become so accustomed to the company of machines that he once found it awkward to interact with people at a super market after a 6 am to 11 pm shift.

“I suddenly forgot the logic of human interaction,” he said.

Taking the Watch for Night Patrols

Ren wipes a machine clean with a piece of blue cloth during a shift

During his frequent lone night patrols, Ren takes in the scents and sounds of the center’s humming machines. As he walks by dusty ones, he takes out a piece of cloth out to wipe them clean.

Engineers often dread receiving emergency calls during late night shifts. At these times, Ren’s brain must be on high alert and jump into emergency response mode, but he must remain calm at the same time. Engineers often collaborate in teams to analyze emergency situations. Yet, there are also times when engineers must work independently to speedily address emergency issues. Sometimes, only one person is required at the emergency site.

Ren walks alone on the cold and quiet streets of The Zhangbei Data Center campus

Zhangbei Data Center is quiet and cold at night. Zhangbei County is a large, windy plain with strong winds that last on average for six months of the year, and a reported average temperature of 2.6℃. The first snowfall often occurs in September, and temperatures can reach minus 38.6 ℃. A person’s beard will turn into a short stub of frost after a short 200-meter outdoor walk. It’s under these extreme weather conditions that Ren leaves the duty room each night to complete patrols in the cold empty streets of the data center.

After a long night shift, Ren eats breakfast at Zhangbei Data Center’s new canteen

At the end of a long night shift, Ren feels happiest when the first beam of sunlight shines on his face in the morning.

Ren remembers a freezing winter when the data center was still under construction. It was so cold that people were still shivering in their down jackets and heavy military jackets. The heater in the temporary shed even broke down, and people were exhaling frosty white clouds in the shed. To generate body heat, the engineers ran, shouted, and laughed as they patrolled the data center under construction.

Although, the data center improved its facilities with the addition of a new canteen in October 2018, Ren misses eating spicy hot pot at food stalls nearby the data center with other colleagues.

Ren grabs a tool box as he runs towards a designated site during an emergency drill

Intensive Workouts on the Job

Misconceptions about engineers’ daily routine include that they are glued in front of the computer screen all day or idling away in equipment rooms. This is far from the truth. Maintaining servers at the data center is physically demanding and time consuming, and despite having an annual gym membership Ren has only visited the gym 10 times throughout the past year. His workout routine has mainly been his server maintenance routine.

Preparations for the 2018 11.11 Festival started as early as last year. On the day of the event, police vehicles and fire trucks line up outside the data center in case of an emergency, and engineers are drilled daily on emergency protocols. Ren often has to sprint to different server locations during emergency drills while carrying his tool box.

Daily maintenance routines also require a lot of physical activity. When Ren Ruxian first started nightly patrols of the campus, he often walked 40,000 steps each day. Eventually, this was reduced to 20,000 to 30,000 steps. With the aid of automated operation and maintenance systems, the number of steps he walks is now around 5000 per day.

Ren falls asleep during a train ride from Beijing to Zhangbei

Pursuing the Engineer Dream

Ren moved out of China’s bustling city of Beijing for desolate Zhangbei County in early November 2018. The train ride from Beijing to rural Zhangbei spans 300 kilometers and takes 3 hours. For two consecutive years, Ren would travel from Beijing to Zhangbei every 10 days, and often stayed in Beijing for only four or five days each month. Before November 11, 2018, Ren moved out of his Beijing apartment to a new house in Zhangbei with all his belongings packed in two suitcases.

The terminal stop of the train trip from Beijing to Zhangbei is actually Ren’s home town in Inner Mongolia.

“When I hopped off the train, I thought it’d actually be great to take this train home,” said Ren. Home is one of the destinations of the train trip, and on the opposite end his aspirations. In a word, “protection” is the shared concept between these two extreme destinations.

Ren walks towards an office building at Alibaba Zhangbei Data Center

Born in 1989, Ren is highly adaptable to Zhangbei County’s quiet wilderness. His hobbies are singing and listening to music. His songs reflect his state of his mind, and sometimes he sings gently, others loudly and excitedly. Whenever Ren finds the freezing weather in Zhangbei unbearable, he reminiscences about life in Beijing, and pictures the busy SoHo district, Central Business District (CBD), and warm, crowded metro in his mind.

Nothing is exciting in Zhangbei County. The county is not chasing after the high stock growth curves or large-sum transactions seen in large financial centers. Similarly, Ren is not chasing after exhilarating city sights or sounds, preferring the tranquility of Zhangbei that helps him feel grounded.

Engineers and technicians like Ren are rarely in the spotlight, and most people are unaware of their existence. As the 11.11 Festival’s chief technology director sums up beautifully, “Technicians are most successful when no one can sense their presence.” So next time you snatch the latest online bargain, try to imagine the team of industrious engineers like Ren working silently behind the scenes to ensure each transaction goes smoothly.

(Original article by Fu Guannan符冠男)

Alibaba Tech

First hand and in-depth information about Alibaba’s latest technology → Facebook: “Alibaba Tech”. Twitter: “AlibabaTech”.