Loba Wellness

THE PLATTNER STORY

200 Black Women in Tech to Follow on Twitter in 2022

200 mujeres negras en tecnología para seguir en Twitter en 2022

200 femmes noires en technologie à suivre sur Twitter en 2022

टेक में 200 अश्वेत महिलाएं 2022 में ट्विटर पर फॉलो करेंगी

2022 年に Twitter でフォローするテクノロジー業界の黒人女性 200 人

200 mulheres negras em tecnologia para seguir no Twitter em 2022

200 phụ nữ da đen trong công nghệ để theo dõi trên Twitter vào năm 2022

2022 年将在 Twitter 上关注 200 名科技黑人女性

Oh Yeah, I've Heard of Algolia Before

For the last couple of years or so, developments regarding AI and Natural Language Processing seem like they have significantly accelerated. There are more companies, more research papers, more products and more capital invested targeting AI every day. Both expectations and promises have inflated, making categorization and analysis more difficult. To help make more sense of things, in this article I want to talk through the trends and constants that brought us to where we stand today. I also want to clarify both “language models” and what happens when they are “large”, the main architectures and model families, and all the other factors that influence the building of an LLMs not related to the architecture and choice of pre-trained base. Finally, I want to touch upon the future expectations relating to LLMs. I will make frequent use of the excellent recent survey by MInaee et al. ( ) and summarize relevant takeaways.` https://arxiv.org/abs/2402.06196 Some background… Language modeling is a decades-old research domain, with the earliest notable attempt being Claude Shannon’s investigation into it, where he measured how well simple n-gram language models predict natural language text. Since then, language modeling has become fundamental to many natural language understanding and generation tasks, ranging from speech recognition, machine translation, to information retrieval. Hence, language models are probabilistic models of language that enable us to predict, classify and generate natural language based on specific inputs. This task is highly non-trivial due to, among many other reasons, data-sparsity, ambiguity of natural language, and long-range dependencies within the datasets. Early models that showed a degree of practical utility were usually highly task specific. They were usually one among many models in highly engineered and improvised Natural Language Processing pipelines. Modifications and variations of neural networks such as Recurrent Neural Networks, LSTM (Long Short-term Memory) networks and GRU (Gated Recurrent Unit) Networks achieved notable results in this era, becoming viable in applications. Yet, operating these in production environments cost-effectively was very challenging and required very specific experiential knowledge available to only a small number of specialists. While word2vec did indeed establish foundational principles for representing semantics and context within vector representations,  thee “big bang” moment, with respect to language modeling, was arguably the 2017 publication of the first Transformer paper. This paper introduced the “Attention” concept, which is essentially a series of matrix operations computed by a Transformer network that helps approximate statistical dependencies among language “tokens” over a wide range of distances. By this time, the “pre-training” - “fine-tuning” approach had already gained some popularity. But the more powerful parallelization capacity of the Transformer architecture enabled a significant improvement in training efficiency, essentially making these models more scalable with respect to training data and model size compared to previous state-of-the-art models. Accordingly, larger and larger models getting (pre)trained with bigger and bigger portions of the web as training data led to more task-agnostic Pretrained Language Models being used as bases upon which different finetuning methods and datasets were applied. Some of the first Pretrained Language Models were ChatGPT’s predecessors like GPT-1 and GPT-2, along with BERT (which achieved SOTA results in translation benchmarks). LLMs are large-scale, pre-trained, statistical language models based on neural networks. “Large” here refers to not just the model size and dataset size but also certain “emergent” properties that are not found in smaller models. These emergent abilities include in-context learning, instruction following and multi-step reasoning. With in-context learning LLMs learn a new task from a small set of examples presented in the prompt at inference time, with instruction following LLMs can follow the for new types of tasks without using explicit examples, and with multi-step reasoning LLMs can solve a complex task by breaking down that task into intermediate reasoning steps. GPT-3 is widely considered as the first “Large” Pretrained Language Model.  It was not only much larger than previous Pretrained Models, but also demonstrated for the first time emergent abilities that are not observed in previous smaller models. GPT3 showed the emergent ability of in-context learning, which means GPT-3 can be applied to any downstream tasks without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieved strong performance on many NLP tasks like translation and question-answering as well as several ones that require on-the-fly reasoning or domain adaptation. Thus, the “Large” Language Model era was born. OpenAI’s success was followed by rival model families released by Meta (Llama) and Google (PaLm) along with a collection of other experimental open-source pretrained LLMs with different modifications and datasets. The Growing Model Universe , distinguishes itself from GPT models by being open-source, providing model weights to the research community under a noncommercial license. This has fostered rapid growth in the LLaMA family as it has been widely adopted for developing better open-source LLMs or task-specific ones. Initial releases in February 2023 ranged from 7B to 65B parameters, surpassing GPT-3 in performance on various benchmarks. LLaMA employs the transformer architecture with minor modifications such as using SwiGLU activation, rotary positional embeddings, and root-mean-squared layer-normalization. LLaMA , a series of foundation language models by Meta introduced its first model in April 2022, remaining private until March 2023. This initial model, boasting 540 billion parameters, is transformer-based and trained on a large text corpus across various language tasks. Utilizing the Pathways system for efficient training on multiple TPU (Tensor Processing Unit) Pods, PaLM achieved state-of-the-art few-shot learning results across numerous benchmarks. Subsequent U-PaLM models, ranging from 8B to 540B scales, undergo continual training with UL2R, resulting in computational savings. Flan-PaLM models, instruction-finetuned on a vast number of tasks and data, substantially outperformed previous models, with Flan-PaLM-540B achieving notable performance gains compared to PaLM-540B. The (Pathways Language Model) family, developed by Google, PaLM Finally, there have been models that have been developed by researchers sing different pretraining and finetuning approaches. The most significant ones being, Flan: Wei et al. introduced Flan, which enhances zero-shot learning abilities by instruction-tuning language models on a collection of datasets described via instructions. The model significantly improves zero-shot performance on unseen tasks and is fine-tuned on over 60 NLP datasets, resulting in the creation of Flan. Gopher: Rae et al. presented Gopher, analyzing transformer-based language model performance across various scales, up to a 280 billion parameter model. Gopher achieves state-of-the-art performance across diverse tasks, with architectural details provided in the study. ERNIE 3.0: Sun et al. proposed ERNIE 3.0, a unified framework for pre-training large-scale knowledge-enhanced models. ERNIE 3.0 fuses auto-regressive and auto-encoding networks, facilitating adaptation for both natural language understanding and generation tasks. RETRO: Borgeaud et al. enhanced auto-regressive language models with RETRO, conditioning on document chunks retrieved from a large corpus based on local similarity with preceding tokens. RETRO achieves comparable performance to existing models on the Pile dataset, despite using fewer parameters. GLaM (Generalist Language Model): Du et al. proposed GLaM, utilizing a sparsely activated mixture-of-experts architecture to scale model capacity with reduced training costs. The largest GLaM variant boasts 1.2 trillion parameters, achieving superior performance across various NLP tasks while consuming less energy and computation flops compared to GPT-3. LaMDA: Thoppilan et al. introduced LaMDA, a family of Transformer-based models specialized for dialogue tasks, trained on public dialogue data and web text. Fine-tuning with annotated data and enabling external knowledge consultation lead to significant improvements in safety and factual grounding. These models represent advancements in various aspects of language modeling, from performance enhancement to specialized applications like dialogue and zero-shot learning. Each model addresses specific challenges within the field and contributes to the ongoing development of large language models. Although a lot of these models and model families bring unique advantages and drawbacks, they are instances of three essential architectural classes with variations on training processes, dataset collection and preparation, and fine-tuning methods. 3 main LLM architectures that are here to stay The Transformer approach introduced a paradigm shift in natural language processing, primarily due to its novel self-attention mechanism optimized for parallel computation on GPUs. At its core, the Transformer architecture consists of an encoder and a decoder. The encoder comprises a stack of identical Transformer layers, each housing two sub-layers: a multi-head self-attention layer and a position-wise fully connected feed-forward network. On the other hand, the decoder, also composed of a stack of layers, incorporates an additional sub-layer for multi-head attention over the encoder's output stack. In the , attention functions play a pivotal role, essentially mapping queries and a set of key-value pairs to an output. These functions, operating on vectors representing queries, keys, values, and outputs, compute weighted sums of values based on the compatibility of queries with corresponding keys. Notably, linear projections are employed to project queries, keys, and values into different dimensions for improved performance. Additionally, positional encoding is introduced to imbue the model with information about token positions within sequences, whether absolute or relative. Transformer model Beyond the foundational Transformer architecture, subsequent developments have led to various model families, each catering to specific NLP tasks. exemplified by BERT, are adept at comprehending entire sequences and excel in tasks like sentence classification and named entity recognition. These models are typically pre-trained by masking words and training the model to reconstruct the original sentences. Encoder-only models, such as those embodied by GPT, Llama and PaLM are tailored for text generation tasks, with attention layers accessing only preceding words. Pre-training for decoder-only models typically involves predicting the next word in a sequence. In contrast, decoder-only models, which integrate both encoder and decoder components, are commonly referred to as sequence-to-sequence models. In this architecture, encoder attention layers can access the entire input sequence, while decoder attention layers focus solely on preceding words. Pre-training for encoder-decoder models involves complex objectives, such as predicting masked text spans. These models are particularly suitable for tasks like summarization, translation, and generative question answering, where generating new sentences conditioned on input is essential. Encoder-decoder models, Beyond the architecture, what else is influencing performance of these models? Although model architecture and the fine-tuning paradigm get most of the attention regarding model building and performance optimization, there are many more aspects of the building process that influence performance outcomes. Techniques such as filtering and deduplication are crucial for enhancing data quality and improving model performance. Filtering involves removing noise, handling outliers, addressing imbalances, and preprocessing text data. Deduplication eliminates duplicate instances or repeated occurrences of the same data in a dataset. Data Cleaning: : Tokenization involves converting text into smaller parts called tokens. Popular tokenization methods include BytePairEncoding, WordPieceEncoding, and SentencePieceEncoding, which address challenges such as out-of-vocabulary words and corrupted text. Tokenization Techniques like Absolute Positional Embeddings, Relative Positional Embeddings, Rotary Position Embeddings, and Relative Positional Bias are used to preserve sequence order and capture positional information in LLMs. Positional Encoding: Pre-training involves training LLMs on large amounts of unlabeled text data using approaches like autoregressive language modeling and masked language modeling. Model Pre-training: Fine-tuning LLMs for specific tasks using labeled data improves performance. Instruction tuning aligns model responses with human expectations by providing task-specific prompts. Fine-tuning and Instruction Tuning: : Alignment techniques like reinforcement learning from human feedback (RLHF), reinforcement learning from AI feedback (RLAIF), and Direct Preference Optimization (DPO) aim to steer LLMs towards human goals and preferences. Alignment Strategies such as greedy search, beam search, top-k sampling, and top-p sampling are used for text generation with LLMs, each with its advantages and trade-offs. Decoding Strategies: Techniques like optimized training, low-rank adaptation (LoRA), knowledge distillation, and quantization aim to reduce the computational and memory requirements of LLMs without compromising performance. Cost-Effective Training/Inference/Adaptation/Compression: Overall, these techniques play crucial roles in improving the quality, efficiency, and effectiveness of large language models. Trends this year and some areas of interest Going forward, the focus of the research and business communities will be directed more towards deployability, robustness and efficiency. Beyond the flashy demos, benchmarks relating to predictability and robustness will attract more attention. Efficiency, both in terms of model latency but also in terms of training and inference costs, will attract greater scrutiny, pushing research efforts towards those metrics. Smaller and More Efficient Language Models (SLMs): There is a growing interest in developing SLMs as a cost-effective alternative to LLMs, especially considering the inefficiencies and high costs associated with larger models. Techniques such as parameter-efficient fine-tuning (PEFT), teacher/student learning, and distillation are being used to create SLMs from larger models, enabling better efficiency for specific tasks. Prominent works in this direction include Phi-1, Phi-1.5, and Phi-2 from Microsoft, which exemplify efforts to explore smaller, task-specific models. New Post-Attention Architectural Paradigms: While transformer blocks have been dominant in LLM frameworks, there's a growing exploration of post-attention architectural paradigms. Alternative approaches like State Space Models (SSMs) and mechanisms such as Mixture of Experts (MoEs) are gaining traction. MoEs, for instance, allow for the training of extremely large models that are only partially instantiated during inference, leading to efficiency gains. The Monarch Mixer proposes a novel architecture using sub-quadratic primitives, enhancing hardware efficiency. Improved LLM Usage and Augmentation Techniques: Advanced prompt engineering, tools usage, and augmentation techniques are addressing shortcomings like hallucination in LLMs. LLMs are replacing other machine learning systems in various applications like customer service and content recommendation, emphasizing personalization and context analysis. Security and Ethical/Responsible AI: Research efforts are directed towards ensuring the robustness and security of LLMs against adversarial attacks and vulnerabilities. Addressing ethical concerns and biases in LLMs is a priority to ensure fairness, unbiased behavior, and responsible handling of sensitive information. As LLMs are increasingly deployed in real-world applications, efforts to mitigate potential threats and ensure responsible AI usage are crucial. If you are building a model, what is your differentiation strategy? As can be seen, despite the enormous progress made in various benchmarks since 2017, the main architectural direction of models has not changed considerably since then. The diversity of models and their various strengths and weaknesses should not blind us to the fact that success in language modeling applications hinges on keeping our eye on metrics that go beyond mere “next token accuracy”, and incorporate business and application level success metrics. Crucially, we have also seen that pre-training and model selection is but one part of a large chain of processes that make up model building. Aspects like alignment strategy, dataset curation and pre-processing, and deployment optimization are important in differentiating a production-level model serving customers from a toy model running on our laptop for our own amusement. As model building is democratized with easy access to hardware and data lies in abundance with every business, what is truly differentiating in the AI stack?

Takeaways From “LLM: a Survey” - Where are You Differentiating?

Hey Hackers, Revend has been nominated in HackerNoon's annual awards in Antwerp, Belgium. Startup of the Year Please vote for us here: . https://hackernoon.com/startups/europe/europe-antwerp-belgium Read more about us below to understand why we deserve your vote. Are you also participating in SOTY 2023? If so, . click here to fill out this interview Meet Revend Founded in early 2022, Revend is a company dedicated to bringing continuous peace of mind to e-commerce operators worldwide via real-time monitoring of your webshop. In the years prior to co-founding Revend, my CTO Alex Reis was looking for ways to react quicker to operational issues that impacted the revenue of the scaling e-commerce stores he was responsible for running. Missed issues - or issues caught too late - often cost tens of thousands in lost revenue and time spent digging through dashboards after the fact. Frustrated by the lack of solutions he found, and motivated by the value that a properly adjusted tool could provide, Alex brought in former colleagues and fellow tech entrepreneurs Peter Wellens (me) and Bert Baeck. Together, we got the first prototype of Revend into the hands of 30 e-commerce companies, gathered feedback to improve the product, and secured an investment to further grow Revend. Today, our team and user base are accelerating at a rapid pace. Revend’s monitoring ensures peace of mind around the clock for e-commerce stores in all shapes and sizes. My Role My name is Peter Wellens, a three-time founder, including co-founder at Revend. My role at the company is that of co-founding CEO. In practice, I’m the first point of contact for many of our new users, partners, and investors supporting us in pursuing our mission. How We're Disrupting/Improving the e-Commerce Industry Revend is firmly focused on serving the e-commerce community. Business owners, marketers, and platform managers in this community are required to be extremely up-to-date at all times, with an onslaught of new tools and software releasing constantly. Staying on top of the performance of your web store in such an environment can be tough. This is where Revend can help. Revend is disruptive in the sense that it is both very easy to implement and highly focused on business users (as opposed to IT users) as its main audience. Its main point of contact with you is via notifications on , Teams, or email. Slack Existing monitoring tools are either quite complex and time-consuming to set up and maintain, or offer no real-time view of the operational KPIs that a marketeer wants to track (e.g. product views, add-to carts, funnel conversions, checkouts). Revend solves all that and uses advanced AI to make sure you get a self-adjusting model that is carefully tailored to the performance of your exact webshop. Standing Out from The Crowd We knew from the outset that just having great and accurate monitoring was not going to be enough to stand out, so from day one, we made sure that: Made for , not (just) developers - Revend monitors your buying journey, not your backend or frontend systems. business people monitoring - we make sure you are the first to know if there is a problem with your shop in minutes, not hours or days. Real-time Turnkey - Easy, no-hassle , no custom integration or long implementation project required. 5 minute setup My Predictions on the e-Commerce Industry in 2023 I talk to e-commerce founders, solution providers, and agencies on a daily basis. My  predictions for 2023, based on those conversations so far, are the following: In a cookieless world, will become a webshop’s lifeblood for the entire funnel. In order to keep doing meaningful marketing automation, a plan to consistently harvest this data will be(come) crucial. first-party data When selling via , the ownership of data will become more of a hot topic. When the marketplace simply doesn't offer you access to the full data story, the question becomes: does making more money on marketplaces offset the blindness to your own customers' data in the long run? marketplaces is here to stay. I’ve seen both successful agencies and brands use it to their advantage, and there seems to be no going back. Headless or composable e-commerce could see growth for Gen-Z and younger audiences for the upcoming Black Friday, as social media giants continue to invest heavily in it. Whether they can break into the European and US markets (where they have had meager results so far) remains to be seen. Live e-commerce What word defines the state of e-Commerce in 2023? The word I would choose is “Flowing”. There’s never a moment of stasis, everything is constantly moving, updating, and shifting. One of many examples is the forced move to GA4, which has meant quite to overhaul in terms of tracking and monitoring for a lot of e-commerce players. Why we decided to participate in HackerNoon's Startup of the Year awards After more than a year in stealth mode, our team is ready to show what we’ve been working on, and we’re proud of it. Happy users, a clear value, and an easy-to-use product. HackerNoon is the perfect platform to reach like-minded people who can evaluate and discuss the topics we are interested in. Final Thoughts Whether you are an e-commerce operator who just heard from us, a startup aficionado browsing HackerNoon, or a -supporter from day one, I thank you from the bottom of my heart for reading this far. Revend I promise that our team and I will continue doing our best to serve the e-commerce community as best we can, and your support for staying on that path is much appreciated. Revend is nominated as Startup of the Year in Antwerp, Belgium. Vote for us today!

Maintaining Your Web Store and Peace of Mind - Interview with Startups of the Year Nominee, Revend

Mantenimiento de su tienda web y tranquilidad - Entrevista con el nominado a Startups del año, Revend

Maintenir votre boutique en ligne et votre tranquillité d'esprit - Entretien avec le nominé Startups de l'année, Revend

अपने वेब स्टोर और मन की शांति को बनाए रखना - वर्ष के नामांकित व्यक्ति, रिवेंज के स्टार्टअप के साथ साक्षात्कार

ウェブストアの維持と安心 - Startups of the Year ノミネート、Revenge へのインタビュー

Mantendo sua loja virtual e paz de espírito - Entrevista com o candidato a startups do ano, Revend

Duy trì cửa hàng trực tuyến và sự an tâm của bạn - Cuộc phỏng vấn với Ứng cử viên khởi nghiệp của năm, Revend

维护您的网上商店并让您高枕无忧 - 采访年度初创公司提名者 Revend

The Scandinavian peoples hold to-day a position

THE STUDY OF THE CRIMINAL

THE FIGHT IN THE PALACE

LOBA WELLNESS

shoploba.com

-- employees

Since n.d.

COMPANY RANKING

LOBA WELLNESS

EVERGREEN INDEX #6742

Representing this company? Click here to claim and customize this page!

Loba Wellness WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

LOBA WELLNESS

shoploba.com

-- employees

Since n.d.

COMPANY RANKING

LOBA WELLNESS

EVERGREEN INDEX #6742

RELATED COMPANIESRANK

STORIES ABOUT LOBA WELLNESS

Representing this company? Click here to claim and customize this page!

Loba Wellness WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!