A story of how one CTO took the company to the brinks of its existence and another almost managed to save the ship I have been sitting on this post for some time now — I wasn’t sure if this is a story that others would find interesting or useful. But after speaking with Jeff Meyerson on the podcast I realised there might be a lot of lessons other engineers and would-be founders can learn from what transpired at Yubl and avoid making the same mistakes. Software Engineering Daily _http://traffic.libsyn.com/sedaily/ServerlessBurningMonk.mp3Podcast: Play in new window | Download After raising $18…_softwareengineeringdaily.com Serverless Startup with Yan Cui On a side note, one of my submissions to Hacker News about also got some traction a while back. As you can see from the comments some readers tried to extrapolate a causality between Yubl shutting down to our decision to use AWS Lambda. things we learnt about AWS Lambda at Yubl Anyone who was there knows that was not the case, and although I had previously touched on the it’s a story that needs to be told in its entirety for one to understand the burning shipwreck it was and how close we came to saving it. Had one of the investors followed through with the investment he verbally agreed on, things could have turned out very differently and those involved will get the accolade they deserve for turning things around in a short time. context around Yubl’s closure But alas, there was no fairy tale ending. Instead we have a cautionary tale of how one CTO took a well-funded startup to the brinks of its existence, and for another CTO to (almost) turn things around through ruthless, decisive hiring & firing. A tale that I will tell with characters from beloved TV series Game of Thrones in order to protect the people involved. : Disclaimer I joined Yubl in April 2016 so I wasn’t present during the reign of the first CTO, the stories from that era are first-hand accounts by those who were there and lived to tell the tale. Prologue (~Aug 2013) A group of fresh university graduates were literally put into a room to think up ideas for a new startup. After some market research and brainstorming, the idea of a new social networking app is born — an app where users can interact with your post via interactive buttons that are part of the post rather than via meta-actions (liking, retweeting, etc.). The idea is pitched to private investors, (non-trivial amount of) investments are raised, and soon a commercial entity is formed around this idea. The app will be known as , or, o r social ubb e. YUBL Y U B L The Littlefinger Era (2014 to ~Aug 2015) joined the company as consulting CTO on the strength of his remarkable CV. On paper, looks the perfect fit for this young, ambitious and well-funded startup looking to take on the established giants in the field — the , and Petyr “Littlefinger” Baelish Petyr Baelish Facebook Instagram Twitter. It’s no small feat. Luckily we now have a fancy office in the heart of London and has assembled an elite marketing team to help us take on the big boys! The only problem is that, we don’t have a product yet… Petyr Baelish The engineering team would be situated far away (at one point the engineering team was in a different building!) from the “product people” so they can focus on doing what their do best. would act as the bridge between the teams and provide regular status updates to the investors to let them know that everything is on track. Petyr Baelish Requirements are passed down to the engineering team in the form of a 300 page document, lovingly crafted by the same eager young minds that had conceived the very idea the company is founded on. Meanwhile, the engineering team scrambles to build more and more features. We need a timeline feature similar to Twitter. Check. Users should be able to follow/unfollow other users. Check. Users can search for other users by first name and/or last name. Check. We need a Suggested Users feature to recommend people for you to follow. Check. We need a Nearby Users feature to show you nearby users for you to follow. Check. and the list goes on, and on. Amidst all the excitement, we also secured contracts with various outsourcers: a company to moderate contents on our platform (for the paltry sum of £10k/month!) a company to provide us with contract PHP developers to develop a CRM/CMS system long term contract with MixPanel at £3k/month … 18 months flew by. No one outside the engineering team had seen the product yet. Despite all the reports from that we’re making good progress and are on track, the investors grew weary. Petyr Baelish And yet, all the engineers are also singing the same tune. If and his engineers are to be believed, everything is rosy, right? Petyr Baelish One of the lead investors, (the original ) did manage to find the truth he seek from the diligent and honest QA engineer . Jeor Mormont Lord Commander Podrick Payne From what and told me, this is how that conversation went. Jeor Podrick : “So tell me, how are we really doing?” Jeor : “We are f**ked” Podrick : “No, that’s not true. We are really really f**ked!!” Podrick : “The app is untestable, nothing works! And the devs are not remotely interested in fixing bugs, when I bring bugs to the server team they just blame the client, when I bring them to the client team the client team just blame the server, nobody wants to find the actual problem.” Podrick It sounded unbelievable to me, I have never ever worked at a place where developers would just keep passing the buck and not take any ownership of their work! Yet, that was a recurring theme when I spoke to folks about what happened during _Petyr Baelish’_s reign — distrust, incompetence, arrogance and downright disrespect and hostility towards one another. Two of the Node.js engineers would forbid the other server developers to look at the code they’re working on. It would later transpire that they spent months building their own framework for doing pub-sub via RabbitMQ. What’s worse, the system they conjured together was so complicated they had no way of debugging and making sense of it. How bad was it? Well, see for yourself… Here, the handler for a RabbitMQ message would decide which of the modules named , , , , and so on would process the message by giving each a chance to run a predicate function against the message. The reason the modules have to be named this way is to enforce precedence. 1 2 3 4 To call it the worst, over-engineered, ill-conceived piece of crap would be an insult to crap. I’ve seen bad code in my time, but this takes the cake. Try figuring out what happens when the handler receives a message from RabbitMQ… The feature in question, however, is crucial for retention — when you post a piece of new content it should be distributed to your followers by background workers. The QA team had a detailed spec of how it should work and a suite of test cases, the system failed most of the test cases. We reimplemented the feature from the ground up after consulting with the product team to understand how the feature is supposed to work. The rework took just over 2 weeks with most of the work carried out by one developer. The reimplemented feature passed all of QA team’s test cases. The feature was rewritten using services AWS offers out of the box, was far simpler, and was covered by a suite of automated tests that were created with the help of the QA and product team. The culture of a startup stems from the founders and the management team, and judging by the evidences on offer one must question the leadership of both (the CEO at the time) and , specifically at their decisions to bring in engineers with such destructive qualities into the company. These bad apples were brought in at a great expense too, all of them were on pretty high day rates even by today’s market standard. Aerys Targaryen Petyr Baelish Angered by what he heard, took immediate actions. Within days both and are gone. Jeor Petyr Baelish Aerys Targaryen was dismissed in a spectacular and public style, and was gently pushed aside as took over the reigns as CEO. Petyr Baelish Aerys Targaryen Jeor Mormont _Baelish’_s close allies soon fled the company, and with _Baelish’_s removal there is now a power vacuum that needs to be filled. In comes as the new head of engineering. Edmure Tully We would later discover some questionable hires by that brought forth questions about his integrity: Petyr Baelish , a tester turned contract Scrum Master after a crash course (well, technically he is certified) was on the payroll for 2 months on a staggering £900/day. When quizzed about his role by he openly admitted that he was doing nothing. Well, at least he’s honest. Pycelle Daenerys , a server engineer who allegedly wrote the “whole” backend system in C++ although he was never in the office and no one has seen any evidence that this backend system exists in any form anywhere. What we had, was an invoice for £60k. In the email thread that shared (to prove that was in fact under Yubl’s employment) was a conversation between the two men where was coaching on how to make his invoices seem more believable. I don’t believe was ever paid that £60k. Jaqen H’ghar Petyr Baelish Jagen Baelish Jagen Jagen The Edmure Era (~Sep 2015 to Jan 2016) If _Petyr Baelish’_s reign was marred by questionable dealings and a lack of visible results, then ’s reign is the personification of desperation and ineptitude, with devastating consequences. Edmure Everything I have heard about paints the picture of a nice guy, a good engineer, and someone who genuinely wants to be everyone’s friend. Whilst he makes a perfectly fine teammate, he lacks a crucial skill required of leaders — knowing when to say no. Edmure From my experience, all the best leaders I have worked with are able to hold both the carrot and the stick and use the appropriate tool to motivate/push his team to get the job done. Those are also the qualities that I look for in the leadership team in any company I join. is perhaps the most famous example of a leader that is able to effortlessly combine both approaches and apply them depending on the player and the situation. Sir Alex Ferguson To ’s defence, this is his first time in the hot seat, and thanks to previous regime’s failures he is put under immense pressure to deliver . The investors have already spent millions and seen nothing in return, they want to see results right away. Edmure immediately Given where the engineering team was at, and the upheaval that has just happened, there was no way it could be done and the reasonable thing to do would be to explain the situation to and ask to revise the plan accordingly. is a perfectly reasonable man and a successful businessman, and he would have understood had communicated clearly the absolute mess they were in at the time. Jeor Mormont Jeor Edmure Unfortunately for him, and for Yubl as a whole, he did the one thing he shouldn’t have — he said Yes. His plan? Hire more developers. It must have been like an early xmas present for recruiters and contractors alike. Here is a startup with a good idea and a lot of money, desperate, willing to pay above market rates to bring people in by the bucket load… At the peak of this hiring craze, Yubl had 15 mobile developers for each mobile platform — that’s a total of 30 mobile developers! All of them expensive contractors. But wait! Why would do that? We know bringing more people on board a software project late on is only going to slow things down, and we have known that since 1975 thanks to the . Ironically, when later confronted by , would use the as argument to support his decision to bring in an army of developers to help get the project over the line… Edmure Mythical Man-Month Daenerys Edmure Mythical Man-Month After firing one CTO and spending a lot of money bringing in all these contractors, expected to see progress. Yet, good news were hard to come by and after a continued lack of progress (especially given the investment and support) it was clear that things had to change, again. Jeor In comes as the new CTO. Daenerys Targaryen The Daenerys Era (Jan 2016 to Nov 2016) In his own words, is a man (I know I know, is a woman in the TV show) who does not tolerate bullsh*ters. Daenerys Daenerys In his first few days on the job, interviewed everyone in the tech team and fired most of the newly hired contractors. In addition, a few of the bad apples who were either bullsh*ters or disruptive forces were also removed. Daenerys After just a few days the size of the technology team was down by nearly half, and the reduced communication overhead allowed those remaining to better focus on getting the app ready for launch. Aside from trimming out the fat in the staffing, was also acting as the . He discussed with the product team and trimmed down the launch feature set to a more manageable level. At the time, the essential features in the app were still barely functioning but the engineers were spending a lot of time working on new features such as VOIP. Daenerys Bad Idea Terminator _Melissa Perri presents how to destroy bad product ideas before they go to market while achieving the business goals and…_www.infoq.com The Bad Idea Terminator In a minor miracle, managed to launch the app within 7 weeks of taking over as CTO. After more than 2 years sitting on a shelf, most people had lost hope. The CEO even offered the option to start over if he thought the ship could not be salvaged. Daenerys Daenerys But things still aren’t great. The app is still buggy. The mobile platforms are not in feature parity. The backend infrastructure is not going to scale. We are down to 2 server developers ( had resigned at this point, after a failed attempt to get fired). There are still a few questionable characters in the mix due to contractual constraints with their agencies. Edmure Daenerys needed to surgically reconstruct the server team and he needed people who are doers, people he can trust. Daenerys I was introduced to through a mutual friend, I had the skillset he needs and he can offer me the new challenge that I’m after. Soon after I joined a few other people knew and trusted from his previous company also followed suit. Daenerys Daenerys What followed was an intense 6 months of transforming the company from both a product and engineering point of view. I am immensely proud of the work me and my team accomplished in such a short time, our efforts did not deserve the bitterly disappointing end we received. I have discussed our work during the podcast and at various , and I have many of our learnings on my blog. Software Engineering Daily talks shared The folks at AWS were very impressed with our work and were in the process of writing up a case study on us when the company was abruptly shut down. _theburningmonk Server Architect Developer @ Yubl. I like C#, F#, Erlang, Go, Dart, Elm, PostSharp and occasionally dabs…_theburningmonk.com Yubl's road to Serverless architecture By the end, we had attracted closed to 1 million installed users, and had around 200k MAU. The VC funds we were speaking to said we were much further along than other social networks after 6 months, but they won’t step in without seeing 12 months of live data. Epilogue (Nov 2016 — present) Since Yubl closed down, we have all gone our separate ways. My biggest disappointment is that the server team (the one that got together in the era) had been broken up. In the short time we worked together we had developed a great rapport and we worked so effectively as a team. Daenerys We tried to stay together as a team but none of the avenues we pursued panned out. After speaking to quite a few companies — by now I have developed a much better sense of what a good start up looks like and the sort of environment I would like to work in — I decided to go back to the mobile games industry and joined Space Ape Games. Space Ape Games is a really weird company in all the good ways — it defies many norms amongst tech companies and does many good things where other companies do badly. During my interview process I was really impressed by my conversation with the founders — they have a clear vision for the company and clearly defined the company’s culture, every recruit is hired for that fit. The company’s mission statement is clearly communicated regularly, and there is great transparency and honesty in the way the founders communicate with the rest of the company. It’s no surprise that, Supercell decided to after speaking to over a hundred game studios all around the world. The two companies bear many resemblance in how they’re structured and the way they operate with small, autonomous game teams. acquire Space Ape Games _Clash of Clans maker Supercell has acquired 62 percent of London-based Space Ape Games for $55.8 million. The deal is…_venturebeat.com Supercell acquires 62% of London's Space Ape Games for $55.8 million I’m currently working on a secret new IP which I can’t talk about just yet, but hopefully it’ll continue to progress through our creative funnel and become publicly playable in the near future!

Maker

Amazon

Twitter

YouTube

Yubl’s road to Serverless — Part 4, Building a scalable push notification system

Understanding homoiconicity through Clojure macros

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

I have met every character from Game of Thrones, all while working for a little known startup…

I have met every character from Game of Thrones, all while working for a little known startup…

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

12 months to become an Indie Hacker

19 Stories To Learn About Failure

4 Celebrity Endorsed Startups That Failed

Six Crucial Startup Lessons I Would Share With My Younger Self

A Brief Timeline Of Kodak's Descent Into Oblivion

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

12 months to become an Indie Hacker

19 Stories To Learn About Failure

4 Celebrity Endorsed Startups That Failed

Six Crucial Startup Lessons I Would Share With My Younger Self

A Brief Timeline Of Kodak's Descent Into Oblivion

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps