The results from Headlight‘s attempt to uncover New York’s most brilliant product managers _Sign up as a competitor or employer for_👩🏾💻 👨🏻💻 NYC Engineering Tournament (Apr 20–23) Product management is one of the most coveted and difficult to land roles in tech. It is a multidisciplinary role which requires technical know-how, product insight, organizational chops, and people skills—and can bring huge success to a company in certain hands, and major harm in the wrong ones. For candidates, it can be hard to definitively prove your ability to do the job. For employers, it can be paralyzing to hire for a role that’s so important yet so hard to define, measure, or articulate. This means most offers go to people who already have the the title “Product Manager” on their resume, with experience at a well-known tech firm or educational institution, or to an internal hire (usually an engineer or designer) with the hope of coaching them on the skills they’re missing. product management This makes it almost impossible for people to break into the field. I faced these challenges myself in breaking into product (you can read the whole story in ) and that’s part of what inspired me to co-found , a performance hiring platform. We help employers recruit, screen, and hire engineers, product managers, marketers, and other roles. FastCompany Headlight Source: The State of Tech Hiring 2017 Helping PM’s Get Hired for Ability, Not Pedigree One of our company’s key tenets is that we should hire for what people can actually do, and not get distracted by the shiny things they have or don’t have on their resume. The truth is, decades of research has shown that work samples are on the job than years of experience or education. better predictors of success Our has found that rather than people / culture skills, the biggest reasons most tech candidates don’t get jobs is lack of (or perceived lack of) technical skills. While take-home assignments are gaining popularity in screening developers, PM candidates haven’t had as many opportunities to demonstrate their ability to succeed in the role. own research on tech hiring In December 2017, we released 9 PM take-homes across B2B, Consumer, and E-Commerce business scenarios. At the end of 2017, Headlight launched , a free library of assessments / mini-projects that employers could use to screen PM candidates. We got some great feedback from folks who tried out these assignments but what we found was that they were still heavily relying on resumes and thus cutting down their candidate pool way too early. PM Take-Homes I had run a one-day product workshop a few months earlier and found an enormous hunger within the NYC tech community for more opportunities to develop their skills and get recognized for their work. So we decided to take things further and designed perhaps what was the first ever online product manager tournament. The NYC Product Tournament In January of 2018, we announced the NYC Product Tournament in partnership with , a psychology-based hiring tech platform. The goal of the tournament was to help identify who were the top PM’s in the greater NYC area and help tech employers find their next product hire. Day100 The Product Challenge We wanted to create a product challenge that was interesting, meaty, and open-ended, to give people a flexibility to interpret and address the problem in their own way. We had a little fun with it and turned the challenge into a bit of a Marvel fan-fiction. Participants were put in the role of the newly hired Head of Product for Stark Labs — an incubator program inside the (fictional) Stark Industries. One of the portfolio companies was struggling with the pilot of their FaceID competitor: SecureSmile. The original pilot had SecureSmile put inside ATMs and consumers were asked to setup their smile data so they could get cash just by smiling After the pilot saw limited adoption, the company is now wondering if it should iterate on the experience and try again or enter a new vertical like luxury apartment buildings, partnering with a smartphone manufacturer, or doing something else entirely The challenge asked competitors to develop a product plan (in presentation format) that would be shown to Tony himself when he arrived in the office a few hours later. They were given a with about 15 different slide formats (big quote, list, 2x2 framework, etc) that they could use to get started. About half of finishers elected to use this and that’s why some of the decks have a similar style. template deck Here‘s a 5 minute video of the product challenge we presented to competitors. They had three hours to submit their presentation. Click here to jump to the final presentations. Prizes To incentivize folks to participate, beyond just the glory of victory of course, we put together some prizes for top finishers. From a tech reading set of by , by , and by , to the latest innovation from one of the world’s most successful product organizations: Airpods, to a roundtrip flight anywhere inside the United States, we wanted to have this tournament to have some tangible stakes and outcomes. Reset Ellen K. Pao Hooked Nir Eyal Sapiens Yuval Noah Harari Evaluation Criteria We talked before about the challenge of figuring out who might succeed in the product role as being difficult because it requires so many different competencies. For the sake of the tournament, we broke that down into four categories: How well do they define a clear and compelling market and business case for this product? Do they appreciate the competitive landscape? Business Opportunity: How well do they understand customer needs, pain points and world view? How would they use research to inform this understanding? Customer Insight: How well do they actually articulate a product solution? Do they understand the design and technology issues at play? Have they defined what success looks like? Product Development: How strong is your execution plan for taking this product to launch and beyond? Are you able to organize the work in a practical and efficient way? Project Management: Psychology Assessment by Day100 Since we were experimenting with a new kind of hiring process where we hid candidate’s names and even resumes, we wanted to give employers a chance to learn more about candidates, so we turned to our friends at Day100, a psychology-based hiring technology platform that works off the Big Five personality framework that is most widely accepted by psychologists. Each competitor completed a Day100 assessment, which was reviewed against an “ideal PM temperament” that we created at Headlight. This wasn’t perfect as every company and manager has a different idea of what they’re looking for, but it at least helped reveal more about what a candidate was like. Judges To bring it all together, we needed to have capable and respected folks reviewing the work of the participants. We assembled a panel of folks from four fields: product, engineering, marketing, and design. By having multiple judges review the work, we’d have a more balanced and healthy view of the participant’s ability. We recruited judges from different size companies — startups, midsized tech co’s, and publicly traded firms — and in different industries — ecommerce, SaaS, media, developer tools, and social networking. I have to give a huge shout out to , , , , , , , and for giving up their Sunday to churn through dozens of competitor submissions, providing feedback and debating the merits of top finishers to get to our final list. #YouTheRealMVP Cliff Kim Kate Zasada Veronica Ray Frantz Joseph Jordan Obi Joel Califa Sam Yeh Victor Gamez The Competition On January 4th, we announced the NYC Product Tournament. We shared the tournament with folks in our networks, on private Slack and Facebook groups, on social media, and through our email lists. In just over two weeks, we had 153 PM’s register to compete! Who were these folks? Most of them were already product managers, with a majority of the remainder in a role adjacent to product like engineering or product marketing. They were nearly evenly split between men and women and 75% of them were between 25 and 34 years old. About half were Asian-American, another 35% were White, and 16% were Black or Latinx. Making tech a more inclusive place where people of all backgrounds can thrive is something we care a lot about and we’ll continue to work on making our tournaments accessible, especially to traditionally underrepresented groups. One final stat — the majority of respondents were actively looking for a new job (60%) while the remainder were open to new opportunities. This isn’t far off from our survey findings which indicated that 75% of tech workers had either changed jobs or seriously considered a move in the past year. Employers We reached out to a couple dozen NYC employers with PM openings and about 14 folks said they were interested in seeing top candidates. Final Results Over the course of Jan 20–21, we had 53 people work on the Product Challenge. Each competitor was given only three (3) hours to review the prompt, develop a plan, and put a presentation together to convey their ideas. Every submission was reviewed by at least two judges, and the top scoring presentations (normalized for tough vs easy graders) were identified. Because we emphasized blind review, judges had nothing to base their judgement on except for the candidate’s work. We even code named these candidates with planets from Star Trek so their race and gender were hidden. The top 14 of competitors were identified, which represented the top quartile of finishers. We had all judges look through the top 14 finishers and together they all independently voted for their Grand and First Prize winners. So far, one of those PM’s have gone on to be hired by an employer via the Tournament. Analyzing our Top Finishers While I’d love to tell you about why all our top finishers (top 25% of all competitors!) are awesome, I’ll stick with just our top four for brevity’s sake. This competitor proposed a new product line geared towards music festivals. Our judges felt like there was a bit of a leap there and wanted more details on the business model. That said, they laid out the feature set very clearly, with effort / impact estimations on the MVP. Their KPI’s for success were also well-considered and overall the concept was strong. 🏅 First Prize: Benthos. This competitor proposed a new product line around luxury apartments. Judges felt they made a good case for the new market and appreciated the user personas (even if they were probably made up given the 3 hour timeline). The presentation also included good thinking on revenue models and used the presentation template well, though the project plan could have used more details. 🏅 First Prize: Onara. — This competitor proposed a new product line around cruise ships. The presentation featured some solid research on how different organizations are using facial recognition and included a 2x2 matrix analysis of different market opportunities. That said, the actual cruise ship proposal, though creative, did not get adequate justification and the product plan was light. 🏅 First Prize: Finnea Prime Our overall winner chose to stick with the banking market. They presented thoughtful ideas for why the initial pilot failed and had solid thinking around customer segmentation and a clear product concept. They were also one of the few who identified risks with the project and how to mitigate them. The only real ding on this one was that the deck was extremely text-heavy with very little visual presentation — but since the focus was on the strength of their ideas, this didn’t hurt Talax much in the end. 🏆 Grand Prize: Talax. All Presentations Below you can see an of all our top finishers. Interested in meeting any of them? Shoot us a note at team@headlightlabs.com with your company and a relevant job description and we’ll be in touch! Airtable Lessons Learned We started this journey wondering if it would be possible to run a hackthon-esque competition for product managers and I think the answer: definitely! Overall our competitors found the product prompt to be clear, appropriately challenging and interesting. That said, there were definitely still areas we could improve: We wanted to give the challenge the same kind of time pressure that a could have. Three hours is clearly a tight time-frame and I think that stressed people out. We also could have extended the timeframe in which you were able to compete from just the weekend. I wrote a note to many of the people who never submitted anything and the vast majority of responses indicated that an unexpected event came up that weekend that prevented them from competing. Our extends the timeframe to 4 hours and doubles the number of days it’s open (Friday — Monday). Give people more time / flexibility. hackathon Engineering Tournament We tried to give our judges a sense of the range of submissions and thought our rubric / evaluation criteria was clear, but once they actually started grading, they found it hard to determine actual scores. Sometimes a presentation’s business thinking would have been a 3 stars for a senior PM but 5 stars for a junior PM. Give judges more context. As a two-person team, we were trying to do a lot, between introducing top finishers with employers, supporting our users on the Headlight platform, and talking with investors. That said, I was struck by how many competitors kept asking when they were going to feedback from judges (which we had to organize and collate) and also how many people opened and clicked through to the presentations of our top finishers. It’s clear that learning how they could improve (as well as studying what top finishers did) was a key reason for why many people entered. People crave feedback. Due to a poor employer UX (literally a spreadsheet with links) and not enough context on our candidates, only 5 of the employers actually took the time to review all the candidates and request introductions. That’s why I made the Airtable to better showcase the work. We’ll also be asking top finishers to write a small personal statement about what their career goals are to help employers understand what they’re interested in / looking for in their next role. Employers need convenience and context. All in all, the Product Tournament was a lot of work, but I’m really happy we were able to pull it off. Probably the best feedback is that 81% of competitors would recommend something like this to their colleagues / peers. We’re excited to take what we’ve learned from this Tournament and apply it to our next event — the . NYC Engineering Tournament 2018 If you’re interested in having better onsite interviews and reducing time spent in the screening process, check out Headlight . And please give this piece 👏 so others can find it too! Have thoughts, questions, feedback? Find me at @jasonshen . Further Reading _The NYC Engineering Tournament is an online programming event for fullstack and mobile devs running April 20-23, 2018…_www.headlightlabs.com Compete in the NYC Engineering Tournament 2018 _Forget résumés. Candidate challenges are the new tool to find top talent. Hackathons and boot camps are great ways to…_www.workforce.com It's Tourney Time - for Hiring - Workforce Magazine _How to make take-home assignments more fair, pleasant, and effective for employers & candidates_medium.com Designing Better Take-Homes _Product Management deals with all the bullshit that ensures a product will ship. PMs are responsible for strategy…_hackernoon.com Product Management Life - Hacker Noon
