MVP Match

mvpmatch

Today, I am speaking with Levin Wense, Co-Founder & Managing Director of MVP Match, which connects startups with top remote tech talent. As the world continues to adapt to a more virtual work environment, the ability to find and retain high-quality remote tech talent has become increasingly crucial. We will explore the benefits and challenges of hiring remote tech talent and gain insights from Levin on effectively sourcing, interviewing, and retaining the best remote tech talent. So, without further ado, let's begin the interview and hear what Levin says about this vital topic. Tell Us About Yourself and Why You Created MVP Match I'm Levin Wense, the co-founder and CEO of MVP Match, a top-tier tech talent marketplace launched in 2020. My background is in shaping early-stage tech startups. I held business development roles at Redi (mobile food ordering) and GoButler (AI-powered concierge services) before joining MVP Factory in 2017 as the first employee. Now, it's a well-established Berlin scaleup that designs and builds digital products for clients using agile teams of highly-skilled tech freelancers. Throughout my work, especially as VP for product & talent at MVP Factory, I've witnessed many companies struggle to build successful product teams – first, to identify the correct headcount and skill mix, and second, to find these people. It's a problem when great products don't come to market because the right team isn't there to execute, but it happens daily. On the other side of that coin are the talents themselves. They're trying to sift through and find suitable projects and roles, often landing unfulfilling jobs that don't support their professional growth. And traditional tech recruiting isn't working to close any of these gaps. There was a need for a new model of talent matching led by experts who really understand product building from both a client and talent perspective. So I co-founded MVP Match to do that. What Advantages Can Remote Talent Offer Compared to In-Person Talent? In a nutshell: more and better opportunities for companies and talent. Many companies still need to adapt to the future of work. They are either resisting the change or waiting to see what happens. But the future is obvious: remote-first and agile. Smart companies are quickly moving in this direction or have already arrived. And they are landing the best talent because they are open to recruiting from any location. For instance, our pool of 7,000+ senior-level tech talent is spread over 120 countries. They work in freelance and permanent roles for our clients across the EU and UK. These clients range from small scaleups to global enterprises. As long as the vetting and matching are done well, and projects are managed right, and there's no reason that talent, and even whole teams, can't be distributed anywhere. For talent, working remotely means a quality of life upgrade. It's the ability to grow your career, work on the most exciting and challenging projects, and live where / how you want. These are the conditions that the best tech talent wants, and they don't need to compromise. Can You Tell Us Some Common Mistakes Companies Make When Hiring Decentralized Teams and How They Can Avoid Them? The most common mistake I see is one that affects all companies hiring teams: needing to know how to design a team for your project. By that, I mean not starting with a clear view of the tasks, relationships, and processes you need to successfully get your project off the ground. If you do that first, you should know what roles to hire for. (We often get called in to support the team design piece before the hiring piece.) How Do You Establish and Adjust Work Culture Through Team Augmentations? We work with many companies that manage agile, fluid product teams. They augment their core teams with freelance talent depending on the project stage. Here are a few key things they do to establish and keep a strong work culture. First, they have great project team leads. In addition to day-to-day task oversight, these leads make everyone feel valued and supported (regardless of employment status). Second, they have a solid onboarding process that introduces new team members quickly to people, expectations, tools, and procedures. Third, if they're hiring freelancers, they prefer experienced ones. This typically means talent who has worked at multiple companies on multiple project teams. People who know how to quickly embed in a team, communicate well, document their processes, etc. The ‘Great Resignation’ Indicates Employees Have the Upper Hand. Is the Current Market Downturn Shifting the Power Dynamics Back Towards Employers? I believe we'll see some overall power shifting in 2023 in the job marketplace. This is because workers are less likely to change jobs in times of economic uncertainty. Naturally, that gives employers more leverage over pay, benefits, and requirements, such as where employees work. I also think employers will opt for more liquid workforce options, like freelancers, where they can flex up or down depending on their needs and financial constraints. However, I don't foresee a power shift in the employer's direction when it comes to tech hiring. The recent layoffs in Big Tech grab headlines, but they represent only a tiny percentage of people working in the sector globally. As for senior-level tech talent – developers, engineers, designers, project managers – the demand from companies for both freelance and permanent roles is still fierce. These people will continue to be able to command very competitive wages and work conditions, like remote-first. What’s Your Take on ChatGPT and Its Involvement With R&D? Will It Replace Software Engineers, Make Their Work Better, or Won’t It Have Any Effect? The use of AI in work is inevitable, not just for developers but for all kinds of knowledge workers. My take is that ChatGPT and similar AI models will make the software engineer's work better and more productive. How good you are at promoting an AI chatbot is quickly becoming a critical programming skill. Smart engineers will use it to do repetitive, tedious tasks faster, like writing, debugging, and explaining code snippets. What chatbots can't do – senior engineers know this – is understand the needs of the business and figure out scalable solutions to meet those needs. How Has Developer Demand Evolved Over the Last Year, and What Do You Predict From the Sector in 2023? There have been some notable changes over the last 12 months. For one, the rise in popularity of Web3 is definitely visible and therefore drives the need for talent with experience in blockchain or crypto. Golang is also quickly becoming a standard favorite technology that is being sought after more frequently. The same can be said for Flutter, the go-to framework for mobile. Security is becoming an integral part of development, shifting the requirement for tech talent. Projects that started three to four years ago needed security as a top priority. Now, companies are specifically looking for talent that can plug that gap. Q3 and Q4 of 2022 were unsettling for many industries – it's difficult to predict the outlook for 2023. However, I do think that, given layoffs and economic uncertainty in the near future, the need for remote tech teams will continue to grow and likely at a record pace, especially in expanding sectors like climate tech. Thank you for speaking with me today, Levin!

Hiring Remote Tech Talent: Overcoming the Challenges, Maximizing the Benefits

-𝙷𝚘𝚠 𝚍𝚘 𝚢𝚘𝚞 𝚝𝚎𝚊𝚌𝚑 𝚊 𝚋𝚘𝚢 𝚑𝚘𝚠 𝚝𝚘 𝚍𝚛𝚘𝚠𝚗? There are mismatched socks everywhere you step; it’s like a pandemic of socks. You walk on your toes, careful not to step on any. Today is the day you learn how to swim. You are excited; it shows in your demeanor. “Oga, hurry up, na. We go leave you o.” You hear the voice of your bosom friend and classmate since the beginning of JSS1, the boy who sticks by you like glue. “No dey rush me, I dey come.” You thunder back as he sighs and tells you to hurry it up. Your room is a mess; it always is. You never remember where the ballpoint pens you so dearly love are kept; you never remember if your bag is on the top of a pile of unwashed clothes. You never remember anything, or rather, you choose to forget. “Oboi, this one goes mad o. Shey, you fit swim?” One of your friends asks, and you are tempted to roll your eyes. It is not a question you grace with a response, so you keep mute. The others come to your rescue in a flash. “No dey do like beans na, you know say we wan teach am how to swim. Sometimes this Otis dey do like person wey him brain dey glitch.” Ayo says, looking at Otis with a look of contempt. Ayo is the boy you look up to, the one most girls would kill to talk to. He looks like a cross between an American and a Nigerian, even though he keeps reminding everyone who decides to listen that he’s a full-fledged Yoruba boy. Otis is the guy everybody ignores, even you. He keeps repeating irrelevant details to fill up the silence. “No mind Otis. Na why him dey do like Otu. E no sabi anything. I go teach you how to swim, no worry.” You nod in acceptance, staring at the boy who speaks with such confidence you can’t help being shell-shocked. He’s another guy, you rever, who’s somehow good at whatever he does. Kanye is his name—the boy who does it all. It doesn’t matter what; he’s good at it. The girls used to like him until they got annoyed with him since they consider him a 𝑘𝑛𝑜𝑤-𝑖𝑡-𝑎𝑙𝑙 or rather, 𝐼𝑇𝐾. So, he sticks with the boys now. “I no mind am before na. Na so he dey do. We don know am. No worry, we get him password for hand.” You say with a laugh, as the others chorus. Four boys go to the swimming pool on a blistering hot day. Even the gods of misfortune could see nothing wrong with that idea. Or so you thought. The pool is a large one; even standing at 5’8, you can tell there are parts of the pool you absolutely must not enter. Girls are squealing like ducks out of the water, but you ignore them. It’s how girls are: noisemakers. Even in your school, it’s no different. In a bid to be seen, they often try to outshine the boys who do whatever they do with no regard for who’s watching. You do not like girls. You used to, but now you don’t. You are startled out of your reverie when Otis jostles you forward with a casual smirk on his face. He looks at you and winks while Ayo and Kanye watch you and Otis fool yourselves. The pool looks inviting as some girls leave it, clapping hands and whispering into their ears. You know how they are—gossip, all of them. You cannot remember where your hatred of women stems from, but you do know that it is from a valid place. You hear it’s called misogyny nowadays. Humans have names for everything, even inconsequential things. “So, how we go do am? Who goes in first?” Ayo asks, one of his hands raised. “Who cares? Make everybody enter jare.” Otis enters the conversation, using his body to push you aside from where you were standing. You stand at the pool's edge before you fall, your hands flailing in the air. The thought that you were just 5’8 feet tall in a pool over 7 feet deep strikes you as ironic. Nature does have a sense of humor, a wicked one. You smile, and you plunge into the water. The water cradles your head as you begin to see the beauty in the darkness of the water. The scenery changes; you lie in the arms of the one who broke you, staring into those deep brown eyes that seem to house all the pain in the universe. “I never taught you how to swim, did I?” She says, a sad smile on the edge of her lips. “You? How are you here?” You ask, fighting back tears. She looks as beautiful as you remember from the past, a phantom with no place to call home. “Does it matter? I guess they call this a place between worlds, right? So, you’re dying. How does that make you feel?” She speaks, and you close your eyes momentarily, wondering why she keeps picking you apart. “Why must you be the last thing I see before I die? Haven’t you done enough? You’re not even dead!” You scream at her face, but the only thing that comes out of your mouth are bubbles. “Dead or alive, I’m the one you love above all others. How does it feel to have your last moments with the only girl your family and friends abhor?” She grins at you, and you begin to see how messed up everything is. She isn’t your friend, she isn’t your family. She’s nothing. Yet, she’s everything. “Haven’t you done enough? Why do you keep doing this?” You say, knowing the fight has been drained out of you. “Because I can.” You can hear voices through the depths, but you refuse to heed them. You want to talk to her a little bit longer. “Wake up, dammit!” You hear a voice pulling you out of your private world with her, a voice that seems to break with pain. You don’t ask; you follow. “Thank God. Yes, that’s it. Breathe in, breathe out. Don’t worry about it. You’ll be fine, you’ll be fine…” The voice gets a little more transparent, and you open your eyes groggily to see the panic-stricken faces of Ayo and Kanye hovering over you. They fuss over you as you cough, water coming from your mouth. Your tiredness is overwhelming; your whole being begs you to go back to sleep. But in a haze, you notice that Otis is nowhere to be found. “Where is Otis?” You ask, looking around. “He’s fine. He was the one who went after you.” Ayo replies as though it is obvious. “Okay, where is he?” You ask again, and Ayo and Kanye exchange nervous glances before they try to find Otis. But you don’t say anything because you know more than anyone else. The same way you know you can never be with 𝑉𝑒𝑟𝑎, 𝑡ℎ𝑒 𝑤𝑖𝑡𝑐ℎ as most of your friends call her. You know where Otis is, and your conversation with him once rings loudly in your head. “Omo me, I just dey o. I no sabi swim. I remember when we go bar beach sef. I just dey play with sand. Me wey get that phobia for water. Wetin be the name again, sef?” Otis asks you as you hold back a laugh before replying. “Hydrophobia. The fear of water.” “Yes! Na im be that. Omo, no tell anybody sha. I wan get over am my way.” “No wahala. Nobody go hear.” You remember; you remember it all. You remember the promise you made not to tell anyone. You don’t have to look for Otis; nobody does. You know where he is. He has become one with the water. You do not remember 𝑡ℎ𝑒 𝑤𝑖𝑡𝑐ℎ anymore; everything else flies by in a haze. Did you attend Otis’ funeral? You do not remember. Your friends try to comfort you, but it is a wasted effort. You didn’t exactly love Otis; he was more like a pesky annoying fly that buzzes around and makes a fuss. You do not care about the death of a fly, yet your heart fills up so much that you fear it’s going to burst. You do not cry; crying is for babies. You stare dry-eyed at the future of the boy who never saw 16. They say the state of a person’s room reflects the state of that person’s mind. That’s why yours is squeaky clean, not a pair of socks in sight. Perhaps you do it for Otis, or maybe it’s for yourself. Maybe your room reflects your mental state, a world empty of who you once were. There are no mismatched socks anymore, just memories of the boy who learned to drown in the guise of swimming. “You can not swim. Why, then, did you do it?” You ask the empty air. But no response comes. Not one. You’re alone with the boy you taught how to drown. A fly, that’s all he was. A pesky fly. Why, then, do you cry?

Mismatched Socks

Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References Acknowledgments This work was partly supported by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) as part of Germany’s Excellence Strategy – EXC 2050/1 – Project ID 390696704 – Cluster of Excellence “Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of Technische Universität Dresden, and by Bundesministerium für Bildung und Forschung (BMBF) and German Academic Exchange Service (DAAD) in project 57616814 (SECAI, School of Embedded and Composite AI). References Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., . . . others (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. Baheti, A., Lu, X., Brahman, F., Bras, R. L., Sap, M., & Riedl, M. (2023). Improving language models with advantage-based offline policy gradients. arXiv preprint arXiv:2305.14718. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., . . . others (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., . . . others (2023). Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31. Coste, T., Anwar, U., Kirk, R., & Krueger, D. (2023). Reward model ensembles help mitigate overoptimization. Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., . . . Sun, M. (2023). Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377. Deng, H., & Raffel, C. (2023). Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. arXiv preprint arXiv:2310.09520. Ethayarajh, K., Choi, Y., & Swayamdipta, S. (2022, 17–23 Jul). Understanding dataset difficulty with V-usable information. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, & S. Sabato (Eds.), Proceedings of the 39th international conference on machine learning (Vol. 162, pp. 5988–6008). PMLR. Feng, X., Wan, Z., Wen, M., Wen, Y., Zhang, W., & Wang, J. (2023). Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179. Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., . . . others (2023). Bridging the gap: A survey on integrating (human) feedback for natural language generation. arXiv preprint arXiv:2305.00955. Gao, L., Schulman, J., & Hilton, J. (2022). Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760. Gilbert, T. K., Dean, S., Zick, T., & Lambert, N. (2022). Choices, risks, and reward reports: Charting public policy for reinforcement learning systems. arXiv preprint arXiv:2202.05716. Glaese, A., McAleese, N., Tr˛ebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., . . . others (2022). Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375. Janner, M., Li, Q., & Levine, S. (2021). Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 1273–1286. Kiela, D., Thrush, T., Ethayarajh, K., & Singh, A. (2023). Plotting progress in ai. Contextual AI Blog. (https://contextual.ai/blog/plotting-progress) Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023). Understanding the effects of rlhf on llm generalisation and diversity. Knox, W. B., Hatgis-Kessell, S., Adalgeirsson, S. O., Booth, S., Dragan, A., Stone, P., & Niekum, S. (2023). Learning optimal advantage from preferences and mistaking it for reward. Knox, W. B., & Stone, P. (2008). Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th ieee international conference on development and learning (pp. 292–297). Lambert, N., Amos, B., Yadan, O., & Calandra, R. (2020). Objective mismatch in model-based reinforcement learning. In Learning for dynamics and control (pp. 761–770). Lambert, N., Gilbert, T. K., & Zick, T. (2023). Entangled preferences: The history and risks of reinforcement learning and human feedback. Lambert, N., Pister, K., & Calandra, R. (2022). Investigating compounding prediction errors in learned dynamics models. arXiv preprint arXiv:2203.09637. Lambert, N., Tunstall, L., Rajani, N., & Thrush, T. (2023). Huggingface h4 stack exchange preference dataset. Retrieved from https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences Lambert, N., Wilcox, A., Zhang, H., Pister, K. S., & Calandra, R. (2021). Learning accurate long-term dynamics for model-based reinforcement learning. In 2021 60th ieee conference on decision and control (cdc) (pp. 2880–2887). Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., . . . Hashimoto, T. B. (2023). Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. GitHub. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., . . . others (2022). Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. This paper is under CC 4.0 license. available on arxiv

Objective Mismatch in Reinforcement Learning from Human Feedback: Acknowledgments, and References

Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References 6 Conclusion This paper presents the multiple ways by which objective mismatch limits the accessibility and reliability of RLHF methods. This current disconnect between design a reward model, optimizing it, and the downstream model goals creates a method that is challenging to implement and improve on. Future work mitigating mismatch and the proxy objectives present in RLHF, LLMs and other popular machine learning methods will becomes easier to align with human values and goals, solving many common challenges users encounter with state-of-the-art LLMs. This paper is under CC 4.0 license. available on arxiv

Objective Mismatch in Reinforcement Learning from Human Feedback: Conclusion

Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References 4 Understanding Objective Mismatch The objective mismatch in RLHF emerges from three broad causes: First, common practice in RL engineering dictates that as long as reward is increasing the model is improving. Second, the evaluation methods available for models trained with RLHF is often incomplete relative to their downstream use-cases. Third, the assumption that the reward model trained is a suitable reward function for optimization. For these reasons, objective mismatch emerges as the assumption that downstream evaluation will be correlated with the reward model score for the current policy, which is not proven. 4.1 Origins of mismatch Objective mismatch in RLHF is the result of the interactions between three different sub-components, rather than just the two (i.e., dynamics model and policy) from MBRL: It is a balance of the reward model training, the goal of getting a calibrated reward function, the policy training, the process of uniformly extracting information from the reward model, and the often bespoke evaluation techniques used for RLHF models, the process of fairly evaluating a multi-use model. There exists an interface between each pair of these three that provides an axis for erroneous assumptions regarding the true optimization problem, but the importance of each link is not uniform for mitigation. Uniformly extracting the information from the reward model into the policy and avoiding the reward hacking inherent to RL (Pan, Bhatia, & Steinhardt, 2022) that can result in overoptimization of reward models (Gao et al., 2022) is central to RLHF. A good reward model may not be one that is empirically easy to train a policy with high reward from, but rather one that is well correlated with downstream evaluation metrics. Common practice in RLHF, especially with larger models where gradients are less stable, is to spend additional compute in search of “stable” training runs with increasing reward, which induces further likelihood of mismatch. Reward model training ↔ policy model training While relatively little work and resources exist for the study of state-ofthe-art reward models, the matching of the reward signal they provide to the intended use-case of the final policy (via evaluation metrics) is central to solving the objective mismatch issue. The reward models are trained on aggregated datasets to maximize agreement of the model on a held out set of data, which in practice often only maximizes at 60-75% accuracy (Bai et al., 2022; Ouyang et al., 2022). Given the complex task encompassed in reward modeling, it is unlikely that the models converge to 100% accuracy, but studying the sources of this performance delta could indicate sources of mismatch. New tools are needed for evaluation of reward models that better match their conceptual underpinnings as a representation of human values for solving the alignment problem (Leike et al., 2018) and as a practical realization as targets for optimization Lambert, Gilbert, and Zick (2023). Reward model training ↔ evaluation tools The third link contributes the least to the emergence of mismatch, but is the easiest axis to visualization potential signs of objective mismatch. Directly matching RL training with any additional evaluation metrics is technically challenges. In MBRL, such a solution could be by using a differentiable simulator (R. Wei et al., 2023), but with the complexity of RLHF such solutions are less obtainable. Exploring any types of regularization or calibration of training with respect to final evaluations is viable as research directions, but this area of study is best suited for visualizing signs of objective mismatch, as shown in Fig. 3. Policy model training ↔ evaluation tools 4.2 Investigations and solutions There is already emerging research on many potential causes of mismatch in RLHF, yet further work can be inspired by solutions from the broader RL literature. Many of the solutions to objective mismatch in MBRL will not apply directly because in MBRL they have a true reward from the environment, and for that reason research is needed to understand the outputs of reward models. Here follows a series of investigations to expanded to mitigate objective mismatch: There are many axes by which a reward model is expected to behave in order to be a reasonable approximation of a reward function, but they are typically not studied. Reward models need to be assessed for consistency, robustness to adversarial attacks, calibration across distributions, and more, as discussed in Lambert, Gilbert, and Zick (2023). Understanding reward models performance is the foundation of solving the mismatch problem. Evaluating reward models will be a indirect but useful path to measure the varied preference datasets used for open RLHF models. • Reward model evaluation In order to solve limitations of reward models across better evaluation techniques, numerous new training methods will be developed. Early research has already shown reward model ensembles can help mitigate overoptimization (Coste et al., 2023). Further research is warranted to integrate techniques that have improved performance of model-based RL algorithms, such as probabilistic loss functions for the dynamics models and planning (Chua, Calandra, McAllister, & Levine, 2018), calibrated probability estimates (Malik et al., 2019) during training the reward model as a classifier, and other solutions (R. Wei et al., 2023). Additionally, links should be explored between the reward models of inverse reinforcement learning (IRL) (Ng, Russell, et al., 2000), the subfield tasked with learning a reward function from agent behavior, and those of RLHF. Early research also shows reformatting the reward model training to better match preference learning literature could improve performance (Knox et al., 2023). • Reward model training methods High-quality datasets are a bottleneck slowing progress in open RLHF research, given the large costs required to acquire them. There are a few datasets available, but they are unproven in their ability match the performance of the best models. The Stack Exchange preference dataset (Lambert, Tunstall, Rajani, & Thrush, 2023), Stanford Preferences Dataset of Reddit content (Ethayarajh, Choi, & Swayamdipta, 2022), UltraFeedback synthetic preference data (Cui et al., 2023), WebGPT internet browsing (Nakano et al., 2021), learning to summarize (Stiennon et al., 2020), and Anthropic HHH dataset (Askell et al., 2021) serve as a strong foundation for research, but exploration is needed. • Reward model training datasets techniques Increased compute can be spent at inference time to improve the performance of RLHF models by utilizing the values returned by the reward model (Deng & Raffel, 2023). Feng et al. (2023) explores this through Monte Carlo tree search generation, yet many more methods can be explored across the planning literature. • Value-guided sampling The most popular evaluation technique for chat-tuned RLHF models is preference percentage versus other top models on evaluation prompt sets (as done in open RLHF models including Llama 2 (Touvron et al., 2023) and Dromedary-2 (Sun et al., 2023)). This evaluation mechanism, while well-motivated in the popular use-cases of the models, suffers from bias and reproducibility challenges. The prompts can easily be chosen to support the model designed by the authors, and the prompts are often not released or aggregated into a future benchmark. Expanding the reproducibility and consistency of these practices will be important to creating robust practices for RLHF. • Human-centric NLP evaluation e As discussed in Sec. 2.1, the optimizers used for RLHF are most often those from previous RL literature. Now there is an opportunity for expansion of RL algorithms into the niche of RLHF, where conditions are highly specialized through the expansive actions space and bandits formulation. New algorithms are a step in the right direction, such as T. Wu et al. (2023) modifying the PPO algorithm for pairwise preferences or Baheti et al. (2023) proposing an offline RL algorithm for full-completion actions. • RL optimizers for languag Other creative mismatch solutions will exist, such as work integrating the LLM policy, reward model, and transition function into a single model (Xu, Dong, Arumugam, & Van Roy, 2023). Methods such as this need to be evaluated across many scales to confirm that they are still numerically stable with the larger state-of-the-art models where powerful emergent behaviors exist. • Other solutions 4.3 Mismatch of next-token prediction The original training object used in popular language model architectures, autoregressive next-token prediction also suffers from an objective mismatch problem, as almost all LLM evaluation techniques evaluate the entire output rather than individual tokens. While this is true, the development signal that the next-token prediction loss provides is more orthogonal to the goals of RLHF. In RLHF, and most related work in RL, the reward signal is interpretted as a direct indicator of performance. This assumption creates a much more unintentionally nuanced research setup, warranting the specific study of its impacts. In MBRL, the learning of a dynamics model is also often done via one-step transitions, with recent work studying autoregressive models (Janner, Li, & Levine, 2021; Lambert, Wilcox, Zhang, Pister, & Calandra, 2021), where the compounding error of multiple one-step predictions is well known as a deeply related issue to objective mismatch (Lambert, Pister, & Calandra, 2022). In the case where mismatch becomes a fundamental problem of LLMs, similar solutions could be investigated. This paper is under CC 4.0 license. available on arxiv

Understanding Objective Mismatch

Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References ABSTRACT Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to prompt and more capable in complex settings. RLHF at its core is providing a new toolkit to optimize LLMs other than next-token prediction, enabling the integration of qualitative training goals. The attempted match between user preferences and downstream performance, which happens in a learned reward model, results in an optimization landscape where training and evaluation metrics can appear correlated. The apparent correlation can lead to unexpected behaviors and stories of “too much RLHF.” In RLHF, challenges emerge because the following sub-modules are not consistent with each other: the reward model training, the policy model training, and the policy model evaluation. This mismatch results in models that sometimes avoid user requests through false safety flags, are difficult to steer to an intended characteristic, or always answer in a specific style. As chat model evaluation becomes increasingly nuanced, the reliance on a perceived link between reward model score and downstream performance drives the objective mismatch issue. In this paper, we illustrate the cause of this issue, reviewing relevant literature from model-based reinforcement learning, and discuss relevant solutions to encourage further research. By solving objective mismatch in RLHF, the LLMs of the future will be more precisely aligned to user instructions for both safety and helpfulness 1 Introduction Reinforcement learning from human feedback (RLHF) is a powerful tool for integrating qualitative styles and values into large machine learning models (Bai et al., 2022; Christiano et al., 2017; Ouyang et al., 2022). RLHF was popularized with its use to integrated human values into large language models (LLMs) for aligning chat tools (Schulman, Zoph, Kim, & more, 2022). In doing so, RLHF has become an important technique in the process of making models better at responding to user requests, often referred to as instruction-tuned, steerability, chat-models, etc. RLHF methods typically operate in a two step process following the training of a base language model, first they learn a model of human preferences that acts as a reward function, and second they use this model within a reinforcement learning (RL) optimization loop. In the RLHF process, these two steps are often executed independently, with an accurate reward model being trained on human preference data and then the RL optimizer is used to extract maximum information into the chat model. A common challenge of modern LLMs trained with RLHF is difficulties in extracting intended behaviors from the model. Sometimes, the models decline benign requests for safety reasons and other times they need clever prompt tuning to extract full performance. In this paper, we detail a fundamental challenge in modern RLHF learning schemes: the objective mismatch issue. In RLHF, three important parts of training are numerically decoupled: the design of evaluation metrics, the training of a reward model, and the training of the generating model. This mismatch between the reward model and the RL training is visualized in Fig. 1, yet other links exist between the goals of evaluation and simulating human values. Specifically, there are many avenues to better align reward model training to the literature in preference quantification (Lambert, Gilbert, & Zick, 2023) and fundamental optimization challenges need to be solved in RLHF practices (Casper et al., 2023). ChatGPT, the most popular model trained with RLHF, shows signs this limitation through issues such as verbosity, self-doubt and question refusals, repeated phrases, hedging, and more (Schulman, 2023). These traits of overoptimization are results of the subtle proxy objective problem that objective mismatch provides a frame for studying and solving – the reward model attributes excess value to phrases that do not contribute to user benefit, which the RL optimizer exploits, such as safety flags. On the hand, the current training setups are not fully aligned with evaluation tools because the RLHF’d models still need sophisticated prompting techniques such as “thinking step by step” (J. Wei et al., 2022) or “take a deep breath” (Yang et al., 2023) to reach maximum performance. Solving objective mismatch will remove the need for these advanced techniques and reduce the likelihood of out-of-scope refusals from an LLM. The phrase objective mismatch originates from model-based reinforcement learning (MBRL), where an agent iteratively learns a dynamics model that it later uses to solve a control task (Lambert, Amos, Yadan, & Calandra, 2020; R. Wei, Lambert, McDonald, Garcia, & Calandra, 2023). In this context, the mismatch is between learning an accurate dynamics model rather than one that is optimized for high task reward. In RLHF, the problem is related, but with added complexity, as the reward model is optimized for preference data over a closed distribution, which does not match the end users. Second, the task of open-ended language generation is less specific to a notion of reward than that of RL control policies. For these reasons, as we explore in this paper, the objective mismatch issue is more nuanced and critical to RLHF. In this position paper, we make three contributions: • Clearly explain the origins and potential manifestations of objective mismatch in chat-tuned LLMs, • Connect related work from NLP and RL literature around objective mismatch, • Propose directions of study to solve the mismatch and foster better RLHF practices. This paper is under CC 4.0 license. available on arxiv

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

সারিবদ্ধকরণ সিলিং: মানব প্রতিক্রিয়া থেকে শক্তিবৃদ্ধি শিক্ষায় উদ্দেশ্যগত অমিল

Die Ausrichtungsobergrenze: Objektive Diskrepanz beim verstärkenden Lernen aus menschlichem Feedback

El techo de alineación: desajuste objetivo en el aprendizaje por refuerzo a partir de la retroalimentación humana

Le plafond d’alignement : inadéquation des objectifs dans l’apprentissage par renforcement à partir de la rétroaction humaine

संरेखण छत: मानव प्रतिक्रिया से सुदृढीकरण सीखने में उद्देश्य बेमेल

アライメントの天井: 人間のフィードバックからの強化学習における目標の不一致

정렬 천장: 인간 피드백을 통한 강화 학습의 객관적 불일치

O Teto de Alinhamento: Incompatibilidade de Objetivos na Aprendizagem por Reforço a partir do Feedback Humano

Потолок согласования: объективное несоответствие в обучении с подкреплением на основе обратной связи с человеком

Hizalama Tavanı: İnsan Geri Bildiriminden Takviyeli Öğrenimde Hedef Uyuşmazlığı

Mức trần liên kết: Sự không phù hợp khách quan trong việc học tăng cường từ phản hồi của con người

对齐上限：来自人类反馈的强化学习中的目标不匹配

The European Union's proposed Cyber Resilience Act (CRA) aims to bolster c standards for products with digital elements. While the intention is commendable, its current form could inadvertently stifle the open-source community, which forms the backbone of modern software development. It was a main topic at Open Source Summit Europe this year, where the slogan could be found on stickers and slideshows everywhere. ybersecurity #FixTheCRA The Hippocratic Oath and Open Source: A Mismatch Doctors take the Hippocratic Oath, vowing to do no harm and to act in the best interests of their patients. This oath is a commitment to uphold specific ethical standards, ensuring that the patient's well-being is always the priority. The CRA, in its current form, seems to impose a similar oath on open-source developers, holding them accountable for the software they release. However, the comparison between doctors and open-source developers starts to break down when we consider scale and control. A doctor typically has a , allowing them to diagnose, treat, and follow up. In contrast, an open-source maintainer might release software that is downloaded by millions, with no direct relationship or even knowledge of the end-users. Moreover, once the software is out, the developer often has no control over how it's used, modified, or integrated. direct relationship with their patients The CRA's Impact on Open Source Commercial Support of Open Source: The CRA's current text only excludes open-source software (OSS) that has no commercial activity around it. This vague definition could mean that if the main contributors to an OSS project are employed, the project might be considered commercially tainted. Such a stance could discourage commercial entities from supporting open-source projects, leading to a potential decline in the quality and quantity of open-source contributions. Vulnerability Disclosure: The CRA would require vulnerabilities to be disclosed within hours of discovery, even if no fix is available. This could jeopardize coordinated disclosure efforts, where researchers give vendors time to develop security patches before vulnerabilities are disclosed publicly. Such a move could inadvertently expose software to more risks. Zero-Days and the CRA: Zero-day vulnerabilities are software vulnerabilities unknown to those who should be interested in mitigating them. If the CRA mandates immediate disclosure of vulnerabilities without allowing time for a fix, it could lead to an increase in zero-day exploits. Attackers would have information about vulnerabilities before patches are available, giving them a significant advantage. Compliance Overload: The CRA could impose a heavy compliance burden on open-source developers, especially those working on critical products. Many open-source projects operate without the financial or legal resources to navigate complex regulatory landscapes. The potential legal risks could deter developers from contributing to open-source projects. Conclusion While the CRA's intentions are noble, its potential impact on the open-source community could be detrimental. The open-source ecosystem thrives on collaboration, innovation, and shared responsibility. Imposing stringent regulations without considering the unique dynamics of this community could stifle innovation and hinder global collaboration. It's crucial to strike a balance that without compromising the spirit of open source. ensures cybersecurity On a personal note, this article was in part inspired by the NHS Doctors Strike in the UK this week. It’s made me reflect very deeply on how much harm can come from putting too much burden, with too little support, on people in critical services. More Resources to Understand #FixTheCRA: [ ] [ ] [[Digital Strategy EC](https://https://[Digital Strategy EC)] [ ] DevOps Linux Foundation TechCrunch Title Image developed by , inspired by this article’s title words. deepai.org

The Cyber Resilience Act (CRA) and Open Source: A Mismatched Prescription

1. HackerNoon: What is your company in 2–5 words? Aaron Lee: We offer AI-driven call & chat answering to SMB. 2. Why is now the time for your company to exist? Providing real time 24/7 customer engagement, lead qualification and conversion has been the biggest pain point for SMB as they don’t have the time nor the resources to do it. Traditional call answering services can only handle simple tasks like take a message and gather contact information. With advancements in technology, particularly AI and automation, Smith.ai evolved to leverage AI to answer calls and chat with human-in-the-loop for escalation, expanding services to integrate with CRMs, provide scheduling assistance, payment collection, and deliver enhanced customer support tailored for businesses of all sizes, especially SMBs. 3. What do you love about your team, and why are you the ones to solve this problem? AI is fast moving but at the current stage, we don’t believe it can handle 100% of the calls and chats as human conversation is nuanced and complicated. Humans can easily detect caller’s emotion but it’s challenging if AI only looks at the transcription, e.g. “That’s great” could be either positive or sarcastic. We believe having a group of highly trained agents + highly tuned AI based on each individual business (think about it as Voice GPT for your business) is the best way to achieve the business outcomes: converting callers to customers. My background is in software engineering, computer graphics with GPU and was the former CTO of The Home Depot which allowed me to work with SMB on a daily basis for years. Our team is unique as we have AI expertise, deep domain knowledge, as well as huge appreciation to the pain points that SMB are facing. 4. If you weren’t building your startup, what would you be doing? I am very excited about entrepreneurship, helping other startups succeed, and have been angel investing for a long time as a side hobby. I probably would be more active in early stage investments if I wasn’t building my startup. 5. At the moment, how do you measure success? What are your metrics? We track customer engagement (number inbound/outbound calls or chats) as one of the core KPIs. Another metric we track is Quality, we treat any internal/external complaint/mistake/expectation mismatch as a concern, basically anything that is not satisfactory to our team or clients would be counted as one and we have been able to maintain a 99.7% correctness for a very long time. This metric has a direct correlation to our churn which is extremely low, which drives the flywheel of great reviews https://www.trustpilot.com/review/smith.ai and more referrals. 6. In a few sentences, what do you offer to whom? Smith.ai provides AI-driven virtual receptionist services, including call and chat answering, tailored for small and medium-sized businesses (SMBs). Our offerings encompass 24/7 live virtual receptionists, AI voice assistants, outreach campaigns, and AI-powered web chat, all designed to enhance customer engagement and support. We serve a wide range of industries, including law firms, home services, marketing agencies, medical and wellness, and many more. 7. What’s most exciting about your traction to date? The most exciting thing about Smith.ai is we have answered tens of millions of calls and chats with extremely high accuracy and satisfaction (99.7%). The most important thing is we help our SMB clients grow by converting callers to customers with the latest AI technologies which is impossible if they were to implement on their own. 8. Where do you think your growth will be next year? Given the uncertainty in inflation and macroeconomics, we believe the trend of doing more with less by leveraging AI and outsourcing will remain front and center for our clients. The fact that we can drive instant satisfaction and revenue growth at a much lower cost would be a huge tailwind for our exponential growth next year. 9. Tell us about your first paying customer and revenue expectations over the next year. We started the company back in October 2015 and I am excited to say that the majority of them are still with us today and have increased their spend with us 10X. We recently launched our AI-first Voice Assistant and it’s a total game changer. One client said “whoever put this together is a bloody genius.” - this is the magical moment when the product is so good that wows our clients. We believe we will have a significant jump in revenue next year as we continue to deploy this technology to 30M SMB in the US. 10. What’s your biggest threat? Our biggest threat is really execution. The market is huge and demand is clear. We have clients asking us for new features on a daily basis and it’s up to us to push the envelope of AI with humans fast enough to meet their needs. Our assessment of the current AI capabilities is that they may struggle with nuanced and complex conversations, especially in emotional or context-heavy interactions. We believe our hybrid human-AI approach is the best in order to bridge the gap. For appointment booking or sales conversion, we believe human touch offers the best business outcome. This startup founder interview template is based on HackerNoon Founder & CEO David Smooke’s ten questions for startup founders. Would you like to take a stab at answering some of these questions? The link for the template is HERE. 1. HackerNoon : What is your company in 2–5 words? HackerNoon Aaron Lee : We offer AI-driven call & chat answering to SMB. Aaron Lee Aaron Lee 2. Why is now the time for your company to exist? Providing real time 24/7 customer engagement, lead qualification and conversion has been the biggest pain point for SMB as they don’t have the time nor the resources to do it. Traditional call answering services can only handle simple tasks like take a message and gather contact information. With advancements in technology, particularly AI and automation, Smith.ai evolved to leverage AI to answer calls and chat with human-in-the-loop for escalation, expanding services to integrate with CRMs, provide scheduling assistance, payment collection, and deliver enhanced customer support tailored for businesses of all sizes, especially SMBs. Smith.ai 3. What do you love about your team, and why are you the ones to solve this problem? AI is fast moving but at the current stage, we don’t believe it can handle 100% of the calls and chats as human conversation is nuanced and complicated. Humans can easily detect caller’s emotion but it’s challenging if AI only looks at the transcription, e.g. “That’s great” could be either positive or sarcastic. We believe having a group of highly trained agents + highly tuned AI based on each individual business (think about it as Voice GPT for your business) is the best way to achieve the business outcomes: converting callers to customers. My background is in software engineering, computer graphics with GPU and was the former CTO of The Home Depot which allowed me to work with SMB on a daily basis for years. Our team is unique as we have AI expertise, deep domain knowledge, as well as huge appreciation to the pain points that SMB are facing. 4. If you weren’t building your startup, what would you be doing? I am very excited about entrepreneurship, helping other startups succeed, and have been angel investing for a long time as a side hobby. I probably would be more active in early stage investments if I wasn’t building my startup. 5. At the moment, how do you measure success? What are your metrics? We track customer engagement (number inbound/outbound calls or chats) as one of the core KPIs. Another metric we track is Quality, we treat any internal/external complaint/mistake/expectation mismatch as a concern, basically anything that is not satisfactory to our team or clients would be counted as one and we have been able to maintain a 99.7% correctness for a very long time. This metric has a direct correlation to our churn which is extremely low, which drives the flywheel of great reviews https://www.trustpilot.com/review/smith.ai and more referrals. https://www.trustpilot.com/review/smith.ai https://www.trustpilot.com/review/smith.ai 6. In a few sentences, what do you offer to whom? Smith.ai provides AI-driven virtual receptionist services, including call and chat answering, tailored for small and medium-sized businesses (SMBs). Our offerings encompass 24/7 live virtual receptionists, AI voice assistants, outreach campaigns, and AI-powered web chat, all designed to enhance customer engagement and support. We serve a wide range of industries, including law firms, home services, marketing agencies, medical and wellness, and many more. 7. What’s most exciting about your traction to date? The most exciting thing about Smith.ai is we have answered tens of millions of calls and chats with extremely high accuracy and satisfaction (99.7%). The most important thing is we help our SMB clients grow by converting callers to customers with the latest AI technologies which is impossible if they were to implement on their own. 8. Where do you think your growth will be next year? Given the uncertainty in inflation and macroeconomics, we believe the trend of doing more with less by leveraging AI and outsourcing will remain front and center for our clients. The fact that we can drive instant satisfaction and revenue growth at a much lower cost would be a huge tailwind for our exponential growth next year. 9. Tell us about your first paying customer and revenue expectations over the next year. We started the company back in October 2015 and I am excited to say that the majority of them are still with us today and have increased their spend with us 10X. We recently launched our AI-first Voice Assistant and it’s a total game changer. One client said “whoever put this together is a bloody genius.” - this is the magical moment when the product is so good that wows our clients. We believe we will have a significant jump in revenue next year as we continue to deploy this technology to 30M SMB in the US. 10. What’s your biggest threat? Our biggest threat is really execution. The market is huge and demand is clear. We have clients asking us for new features on a daily basis and it’s up to us to push the envelope of AI with humans fast enough to meet their needs. Our assessment of the current AI capabilities is that they may struggle with nuanced and complex conversations, especially in emotional or context-heavy interactions. We believe our hybrid human-AI approach is the best in order to bridge the gap. For appointment booking or sales conversion, we believe human touch offers the best business outcome. This startup founder interview template is based on HackerNoon Founder & CEO David Smooke’s ten questions for startup founders. This startup founder interview template is based on HackerNoon Founder & CEO David Smooke ’s ten questions for startup founders . HackerNoon Founder & CEO David Smooke ten questions for startup founders Would you like to take a stab at answering some of these questions? The link for the template is HERE. Would you like to take a stab at answering some of these questions? The link for the template is HERE. The link for the template is HERE.

Stories, News & Mentions about MVPMATCH

Mvpmatch’s Stories on HackerNoon