MetalBear

Transductive Learning for Textual Few-Shot: Limitations, Acknowledgements, & References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pierre Colombo, Equall, Paris, France & MICS, CentraleSupelec, Universite Paris-Saclay, France; (2) Victor Pellegrain, IRT SystemX Saclay, France & France & MICS, CentraleSupelec, Universite Paris-Saclay, France; (3) Malik Boudiaf, ÉTS Montreal, LIVIA, ILLS, Canada; (4) Victor Storchan, Mozilla.ai, Paris, France; (5) Myriam Tami, MICS, CentraleSupelec, Universite Paris-Saclay, France; (6) Ismail Ben Ayed, ÉTS Montreal, LIVIA, ILLS, Canada; (7) Celine Hudelot, MICS, CentraleSupelec, Universite Paris-Saclay, France; (8) Pablo Piantanida, ILLS, MILA, CNRS, CentraleSupélec, Canada. Table of Links Abstract & Introduction Related Work API-based Few-shot Learning An Enhanced Experimental Setting Experiments Conclusions Limitations, Acknowledgements, & References Appendix A: Proof of Proposition Appendix B: Additional Experimental Results 4 An Enhanced Experimental Setting 4.1 Datasets Benchmarking the performance of FSL methods on diverse sets of datasets is critical to evaluate their generalization capabilities in a robust manner as well as their potential for real-world applications. Previous work on FSL (Karimi Mahabadi et al., 2022; Perez et al., 2021) mainly focuses on datasets with a reduced number of classes (i.e., K < 5). Motivated by practical considerations we choose to build a new benchmark composed of datasets with a larger number of classes. Specifically, we choose Go Emotion (Demszky et al., 2020), Tweet Eval (Barbieri et al., 2020), Clinc (Larson et al., 2019), Banking (Casanueva et al., 2020) and the Multilingual Amazon Reviews Corpus (Keung et al., 2020). These datasets cover a wide range of text classification scenarios and are of various difficulty[4]. A summary of the datasets used can be found in Tab. 1. 4.2 Model Choice The selection of an appropriate backbone model is a critical factor in achieving high performance in few-shot NLP tasks. To ensure the validity and robustness of our findings, we have included a diverse range of transformer-based backbone models in our study, including pretrained using the closed task (Taylor, 1953). We consider two different sizes of the RoBERTa model, namely RoBERTa (B) with 124M parameters and RoBERTa (L) with 355M parameters and DistilRoBERTa, a lighter version of RoBERTa trained through a distillation process (Hinton et al., 2015), for a total of 82M parameters. Three different sizes of RoBERTa based models (Liu et al., 2019b). Similar to BERT, RoBERTa is Three sentence-transformers encoder (Reimersand Gurevych, 2019). Following (Muennighoff et al., 2022), we consider MPNET-base (Song et al., 2020), MiniLM (Wang et al., 2020), and Albert Small V2 (Lan et al., 2019). Multilingual models. To address realistic multilingual scenarios, we rely on three sizes of XLMRoBERTa (Conneau et al., 2020, 2019): base (B), large (L) and XL (XL). text-davinci model: to mimic the typical setting of API-based models, we also conduct experiments on text-davinci, only accessible through OpenAI’s API. 4.3 Evaluation Framework Prior research in textual FSL typically involves sampling a low number of tasks, typically less than 10, of each dataset. In contrast, we utilize an episodic learning framework that generates a large number of N-shots K-ways tasks. This framework has gained popularity through inductive metalearning approaches, such as those proposed by (Finn et al., 2017b; Snell et al., 2017; Vinyals et al., 2016; Sung et al., 2018a; Mishra et al., 2017; Rusu et al., 2019; Oreshkin et al., 2018), as it mimics the few-shot environment during evaluation and improves model robustness and generalization. In this context, episodic training implies that a different model is initialized for each generated few-shot task, and all tasks are compiled independently in parallel. This approach allows to the computation of more reliable performance statistics by evaluating the generalization capabilities of each method on a more diverse set of tasks. To account for the model’s generalization ability, we average the results for each dataset over 1000 episodes, with the N considered classes varying in every episode. For each experiment, we consider the F1-Score. [4] Datasets are available in Dataset (Lhoest et al., 2021)

Transductive Learning for Textual Few-Shot Classification: An Enhanced Experimental Setting

This paper is available on arxiv under CC 4.0 license. Authors: (1) Berrenur Saylam; (2) Ozlem Durmaz ¨ ˙Incel. Table of Links Highlights Abstract & Introduction Federated Learning: Collaborative and Privacy-Preserving Model Training State of the Art Federated Learning Software Platforms and Testbeds Sensors and Edge-Sensing Devices Federated Learning Applications on Sensing Devices Discussion Conclusions, Acknowledgment, and References 2. Federated Learning: Collaborative and Privacy-Preserving Model Training This section delves into the field of FL, a method created to produce accurate models while protecting the privacy of people and institutions. As mentioned, it proposes a collaborative learning paradigm in which numerous devices take part in training a shared model under the direction of a central server. This section investigates the nuances of this collaboration procedure and its various architectural strategies. FL is a notion that respects privacy. Firstly the server distributes a model with randomly initialized parameters to the participating devices. Each device uses its local data to train the Collaborative Model Training: shared model, ensuring no raw data leaves the device. Instead, the server receives only the modified model parameters. The model averaging method is often used to reach a stopping point in this iterative procedure Wang and Preininger (2019); Rieke et al. (2020) (a visual scheme of the process can be seen in Figure 2). By aggregating the updated parameters from all devices, a global model is obtained, embodying the collective knowledge of the participants while preserving data privacy. 2.1. Architectural Categories FL encompasses three main architectural categories: horizontal, vertical, and transfer federated learning (FTL) Yang et al. (2019). These categories are determined based on the data distribution and the collaboration’s nature. In the horizontal architecture, data features are similar across devices, with variations primarily in data volume. On the other hand, the vertical architecture focuses on data with similar data correspondence (IDs) but differing feature spaces. Horizontal architecture, often called sample-based FL, is commonly employed in medical cases where different local devices share similar data structures. In contrast, vertical architecture, known as featurebased FL, is well-suited for aggregating user data from different servers. This architecture enables communication and collaboration between devices, requiring aggregating diverse features Aledhari et al. (2020). An example of horizontal architecture is the Google keyboard, where users from different regions update model parameters locally, which are then aggregated by the server. In contrast, the vertical architecture finds its relevance in scenarios where multiple institutions, such as banks, collaborate to generate personalized models by aggregating knowledge from different domains of the same person. FTL also leverages multiple data sources to train a model on the server. The goal is to identify commonalities across features, reducing the overall error. FTL supports both homogeneous (different samples) and heterogeneous (different features) training approaches, making it particularly valuable in healthcare applications Jing et al. (2019). Each FL architecture offers distinct advantages. The horizontal architecture ensures data independence, allowing devices to update model parameters without extensive communication. Vertical architecture enables collaboration between different servers, facilitating the aggregation of diverse features. FTL leverages different data sources to identify common patterns, ultimately improving model accuracy Aledhari et al. (2020); Yao et al. (2021). These architectural approaches find applications across various domains where privacy and data security are paramount. For example, in the healthcare domain, by leveraging FL, healthcare institutions can collaboratively train models while preserving patient privacy, facilitating personalized healthcare services, and advancing medical research. 2.2. Challenges Federated Learning encompasses various research areas Kairouz et al. (2021); Saeed et al. (2020), including optimization Li et al. (2020), communication efficiency Koneˇcn`y et al. (2016), personalization Wang et al. (2019), fault tolerance Bonawitz et al. (2019), privacy preservation Bonawitz et al. (2017), and computational efficiency Zhou et al. (2020). Likewise the case in many methodologies, FL also has pros and downsides. These challenges mainly arise due to device, data, and model heterogeneity (Figure 3). The use of different devices in data collection, such as urban sensors, smartphones, wearable devices, and autonomous vehicle sensors Jiang et al. (2020); Nguyen et al. (2021), introduces variations in sensing capabilities, memory, data processing, battery life, and communication capacity. Device heterogeneity: Consequently, challenges arise, including fault tolerance, communication efficiency, and computational efficiency, due to the diverse nature of IoT devices. Device heterogeneity leads to non-IID (non-identically and independently distributed) data, manifesting in different labels, feature distributions, and concept drift Xu et al. (2021). For example, when data is collected from multiple devices, each device may have only a subset of the activity classes in Human Activity Recognition (HAR) research. Moreover, different individuals may assign different labels to the same activity. Data heterogeneity: Furthermore, variations in feature distributions can occur due to individuals’ unique characteristics and behaviours. Addressing these challenges requires distributed optimization techniques and personalized approaches. While the standard model used in basic FL architecture facilitates aggregation, it may only be suitable for some scenarios. Model heterogeneity arises from the diverse needs of client devices. Model heterogeneity: For example, when constructing a model for predicting user activities using data from smartphones and smartwatches, adapting the model based on each device’s capacity may be necessary. Additionally, clients may have privacy concerns that discourage sharing certain model parts. This challenge underscores the need for further research on privacy preservation and client participation. 2.3. Federation Algorithms FL algorithms vary based on the model aggregation methods used in the literature. The initial algorithm proposed by Google, known as FedAvg McMahan et al. (2017), employs the average function for aggregation. Subsequently, several variations of averaging methods emerged, including weighted averaging Shlezinger et al. (2020), one model selection, best model averaging Yao et al. (2021), stochastic controlled average Karimireddy et al. (2020), and periodic averaging combined with quantization Reisizadeh et al. (2020). However, FedAvg has limitations when working with non-IID data and fails to capture fine-grained information specific to each device Wu et al. (2020). Other widely used algorithms include FedMA Wang et al. (2020), which constructs a global model layer-wise using matching and averaging, specifically designed for neural network architectures to address data heterogeneity. FedPer Sannara et al. (2021) tackles data heterogeneity caused by non-IID distributions by providing personalized models composed of base and personalized layers. The base layers are aggregated on the server, while clients focus on representation learning in the personalized layer using transfer learning. Moreover, the server must receive models from all participating devices to apply any aggregation method which causes latency problems. Slower clients are called stragglers, and the problem is called the straggler problem Zaharia et al. (2008). In other terms, we can define this problem as a synchronous and asynchronous learning problem. The literature proposes asynchronous Xie et al. (2019) and semi-synchronous Wu et al. (2020) methods to overcome the straggler problem. In addition to latency, when the server faces unbalanced and non-IID data, even these proposed methods converge slowly Duan et al. (2019). In Liu et al. (2021), the authors propose a partial aggregation strategy (FedPA) where the server updates its model using models from only a subset of clients determined by a reinforcement learning model. This parameter’s adaptive and dynamic calculation considers data distribution and device heterogeneity, preventing accuracy loss caused by selecting an inappropriate aggregation number. Similarly, in Gao et al. (2021), the authors propose the n-soft sync aggregation algorithm to address cloud idle time. This algorithm combines the benefits of synchronous and asynchronous aggregation by uploading only the models of n clients in each round, striking a balance between transmission overhead and training time. This approach ensures faster completion of training by corresponding clients. In Erg¨un et al. (2022), authors introduce FL with Secure Aggregation (FedSecAgg), an multi-party computation (MPC)-based aggregation protocol that ensures privacy and security in FL scenarios. It addresses the challenge of privacy leakage during the aggregation process by employing secure MPC techniques. This approach allows the aggregation to be performed while preserving the privacy of individual client updates. In He et al. (2020), Federated Meta-Learning with Model Agnostic MetaLearning (FedMeta-MAML) is proposed. FedMeta-MAML combines the principles of FL with model-agnostic meta-learning (MAML). Meta-learning enables rapid adaptation to new tasks and clients by learning a good initialization for client models. FedMeta-MAML aggregates client updates using a meta-update rule, allowing clients to learn how to learn from their local data. FL with Momentum SGD (FedMoSGD) extends federated learning by incorporating the momentum technique Xu et al. (2021), which is commonly used in stochastic gradient descent (SGD) optimization. The momentum technique accelerates convergence and improves the robustness of the optimization process by adding a fraction of the previous update direction to the current update direction. Another proposed method in literature is FedProx which adresses the challenges of heterogeneity Li et al. (2020). It is a generalization and reparametrization of the popular FedAvg method, which allows for local updating and low participation to tackle communication costs. The proposed framework introduces modifications that address both system and statistical heterogeneity, providing convergence guarantees in scenarios with non identically distributed data across devices. Empirically, FedProx demonstrates more robust convergence and improved accuracy compared to FedAvg on realistic federated datasets, especially in highly heterogeneous settings. The field of FL has witnessed advancements in aggregation methods to overcome challenges posed by non-IID data, latency, and privacy. These state-of-the-art algorithms provide valuable contributions toward enhancing FL processes’ efficiency, privacy, and convergence. 2.4. Metrics In order to evaluate and compare Federated Learning (FL) algorithms, various metrics are employed in the literature. These metrics provide insights into different aspects of FL performance and effectiveness. When assessing FL algorithms, it is crucial to consider metrics that capture model performance, communication cost, computational power, and fairness. In literature Semwal et al. (2020), accuracy and loss for a model performance measure, number of global rounds and the amount of transmitted data for communication cost, number of local rounds and convergence for computational power measure, fairness (similar performance over clients) are used as metrics. Model performance in FL is evaluated using accuracy and loss metrics, which measure the accuracy of predictions and the deviation between predicted and actual values. Higher accuracy and lower loss values indicate better model performance in terms of prediction quality and generalization. These metrics are also used to compare FL performance with traditional ML algorithms. Communication cost is crucial in FL systems, with metrics like global rounds and transmitted data assessing efficiency. Minimizing the number of rounds and reducing the transmitted data volume can optimize the overall communication cost in FL systems. Computational power measures evaluate algorithms’ efficiency in reaching stable solutions. The number of local rounds reflects the computational burden on individual client devices during training. Convergence, another important metric, evaluates the algorithm’s efficiency in reaching a stable and optimal solution. Faster convergence indicates a more computationally efficient approach. Fairness is an essential consideration in FL, aiming to ensure equitable performance and benefits for all participating clients. Fairness metrics assess the consistency of model performance across different clients, avoiding biases or imbalances during the learning process. This paper is under a CC 4.0 license. available on Arxiv

Collaborative and Privacy-Preserving Model Training: The Pros and Cons

Hello, fellow knowledge seekers! My name is and today I want to talk about a book that changed the way I think, so therefore it changed my life. Alex Cloudstar Are you ready to shake up your learning game and take on the world of ultralearning? Strap in for an exhilarating ride through Scott H. Young's mind-bending book on turbocharged learning. Word on the street is that our fellow learner here highly recommends this gem of a book. So, let's dive into these ultralearning principles while sprinkling in some quirky humor along the way. Picture this: a modern-day learning wizard whose experiments with intense learning projects have turned heads and flipped textbooks. That's Scott H. Young for you! He’s the guru behind "Ultralearning," and trust me, his ideas pack a punch! Get ready to harness the power of ultralearning and revolutionize how you tackle new skills. Principle 1: Metalearning Ever felt like 90% of your learning time goes into choosing what to learn? Scott's got you covered. His secret sauce? Devote  5-10% to research, saving the rest for diving deep into those instrumental (work-related) or intrinsic (nerd-worthy) learning projects. Principle 2: Focus We’ve all been there—trying to learn calculus while battling the distractions of a buzzing phone or that lurking YouTube playlist. Scott's advice? Identify your foes, those pesky distractions, and wrangle your wandering mind back on track. Your calculus nemesis won't stand a chance. Principle 3: Directness It's like method acting for learning! Scott’s tip? Immerse yourself in an environment mirroring the real-world application of your skill. Learning French? Move to Paris! Or at least pretend you did. Grab a beret and a baguette—bonjour, ultralearning! Principle 4: Drill Scott’s got more drills than a fitness instructor! Dive deep into your weaknesses, like a ninja training montage, with time slicing, component focusing, and the elusive Magnifying Glass Method. Channel your inner learning warrior. Principle 5: Retrieval - Test to Learn Scott's recipe for knowledge retention? It’s a feast of tests, flashcards, and free recall. Transforming notes into questions? It’s like turning bland veggies into spicy trivia nuggets! The more, the merrier! Principle 6: Feedback Feedback, ahoy! Scott urges you to seek out critics who’ll tell you what’s wrong without necessarily giving you a how-to-fix-it manual. It’s like getting roasted at a comedy club, but hey, it’s for your learning. Principle 7: Retention Picture this: spacing out learning sessions like spaced-out astronauts in a space station. That’s the secret to retention! Over-learning like a math wizard? Scott’s got you covered—conquer addition before you even think about division. Principle 8: Intuition Ah, the mystical world of intuition… Scott’s guide here? Don’t give up on those brain-bending problems too soon. Explain complex concepts like you're telling a bedtime story—simple, yet captivating. Principle 9: Experimentation Break free from the chains of conventional learning! Scott’s mantra? Experiment, experiment, experiment! Mix and match learning hours, courses, and tools like a mad scientist concocting the perfect potion for genius-level learning. Embracing Ultralearning: A Personal Project And now, for the pièce de résistance—a personal ultralearning project! Our valiant learner here is taking on algorithms. Yes, those perplexing puzzles that make even Sherlock Holmes scratch his head. But fear not! Armed with Scott’s quirky wisdom, our learner's set to crack these algorithmic codes like a secret agent on a mission. Lifelong learning just got a turbo boost. "Ultralearning" is the roadmap to your learning superhero transformation. So, whether you're diving into algorithms, languages, or becoming a master chef, buckle up and embrace these principles. It’s time to unleash your inner learning superhero and conquer the world, one skill at a time! Also published here.

How a Book Rewrote the Script of My Life

The Game Changer Deep Learning has become the main driver of many new applications and it’s time to really look at why this is the case. With so many other options that we’ve been using for so long, why Deep Learning? Deep Learning is popular right now because it’s easy and it works. OK you maybe thinking deep learning is easy and works, but what is it? I’m glad you asked let’s jump right to it. 😄 Deep learning is a subset of machine learning where artificial neural networks, algorithms inspired by the human brain, learn from large amounts of data. Similarly to how we learn from experience, the deep learning algorithm would perform a task repeatedly, each time tweaking it a little to improve the outcome. We refer to ‘deep learning’ because the neural networks have various (deep) layers that enable learning. Just about any problem that requires “thought” to figure out is a problem deep learning can learn to solve. Deep learning differs from traditional machine learning techniques in that they can automatically learn representations from data such as images, video or text, without introducing hand-coded rules or human domain knowledge. Their highly flexible architectures can learn directly from raw data and can increase their predictive accuracy when provided with more data. For example in facial recognition, how pixels in an image create lines and shapes, how those lines and shapes create facial features and how these facial features are arranged into a face. Deep learning changes how you think about representing the problem that you’re solving with analytics. It moves from telling the computer how to solve a problem to training the computer to solve the problem itself For example, a deep learning model known as a convolutional neural network can be trained using large numbers (as in millions) of images, such as those containing cats. This type of neural network typically learns from the pixels contained in the images it acquires. It can classify groups of pixels that are representative of a cat’s features, with groups of features such as claws, ears, and eyes indicating the presence of a cat in an image. Deep learning is fundamentally different from conventional machine learning. In this example, a domain expert would need to spend considerable time engineering a conventional machine learning system to detect the features that represent a cat. With deep learning, all that is needed is to supply the system with a very large number of cat images, and the system can autonomously learn the features that represent a cat. Credit For The Image Goes To: TechTarget The amount of data we generate every day is staggering — currently estimated at 2.6 quintillion bytes and it’s the resource that makes deep learning possible. Since deep-learning algorithms require a ton of data to learn from, this increase in data creation is one reason that deep learning capabilities have grown in recent years. In addition to more data creation, deep learning algorithms benefit from the stronger computing power that’s available today. Deep learning has been also instrumental in the discovery of exoplanets and novel drugs and the detection of diseases and subatomic particles. It is fundamentally augmenting our understanding of biology, including genomics, proteomics, metabolomics, the immunome, and more. What kind of problems does deep learning solve? As mentioned, deep neural networks excel at making predictions based on largely unstructured data. That means they deliver best in class performance in areas such as speech and image recognition, where they work with messy data such as recorded speech and photographs. Should you use always deep learning instead of machine learning? No, because deep learning can be very expensive from a computational point of view. For non-trivial tasks, training a deep-neural network will often require processing large amounts of data using clusters of high-end GPUs for many, many hours. Given top-of-the-range GPUs can cost thousands of dollars to buy, or up to $5 per hour to rent in the cloud, it’s unwise to jump straight to deep learning. If the problem can be solved using a simpler machine-learning algorithm such as Bayesian inference or linear regression, one that doesn’t require the system to grapple with a complex combination of hierarchical features in the data, then these far less computational demanding options will be the better choice. Deep learning may also not be the best choice for making a prediction based on data. For example, if the dataset is small then sometimes simple linear machine-learning models may yield more accurate results — although some machine-learning specialists argue a properly trained deep-learning neural network can still perform well with small amounts of data. What are the drawbacks of deep learning? One of the big drawbacks is the amount of data they require to train, with Facebook recently announcing it had used one billion images to achieve record-breaking performance by an image-recognition system. When the datasets are this large, training systems also require access to vast amounts of distributed computing power. This is another issue of deep learning, the cost of training. Due to the size of datasets and number of training cycles that have to be run, training often requires access to high-powered and expensive computer hardware, typically high-end GPUs or GPU arrays. Whether you’re building your own system or renting hardware from a cloud platform, neither option is likely to be cheap. Deep-neural networks are also difficult to train, due to what is called the vanishing gradient problem, which can worsen the more layers there are in a neural network. As more layers are added the vanishing gradient problem can result in it taking an unfeasibly long time to train a neural network to a good level of accuracy. Why are deep neural networks hard to train? As mentioned deep neural networks are hard to train because of the number of layers in the neural network. The number of layers and links between neurons in the network is such that it can become difficult to calculate the adjustments that need to be made at each step in the training process — a problem referred to as the vanishing gradient problem. Credit For The Image Goes To: makeameme.org Another big issue is the vast quantities of data that are necessary to train deep learning neural networks, with training corpuses often measuring petabytes in size. How much does it cost to invest in deep learning? It depends on your approach, but it will typically cost you hundreds of dollars upwards, depending on the complexity of the machine-learning task and your chosen method. What deep learning techniques exist? There are various types of deep neural network, with structures suited to different types of tasks. For example, Convolutional Neural Networks (CNNs) are typically used for computer vision tasks, while Recurrent Neural Networks (RNNs) are commonly used for processing language. Each has its own specializations, in CNNs the initial layers are specialized for extracting distinct features from the image, which are then fed into a more conventional neural network to allow the image to be classified. Meanwhile, RNNs differ from a traditional feed-forward neural network in that they don’t just feed data from one neural layer to the next but also have built-in feedback loops, where data output from one layer is passed back to the layer preceding it — lending the network a form of memory. There is a more specialized form of RNN that includes what is called a memory cell and that is tailored to processing data with lags between inputs. The most basic type of neural network is a multi-layer perceptron network, the type discussed above in the handwritten figures example, where data is fed forward between layers of neurons. Each neuron will typically transform the values they are fed using an activation function, which changes those values into a form that, at the end of the training cycle, will allow the network to calculate how far off it is from making an accurate prediction. There are a large number of different types of deep neural networks. No one network is inherently better than the other, they just are better suited to learning particular types of tasks. More recently, generative adversarial networks (GANS) are extending what is possible using neural networks. In this architecture two neural networks do battle, the generator network tries to create convincing “fake” data and the discriminator attempts to tell the difference between fake and real data. With each training cycle, the generator gets better at producing fake data and the discriminator gains a sharper eye for spotting those fakes. By pitting the two networks against each other during training, both can achieve better performance. GANs have been used to carry out some remarkable tasks, such as turning these dashcam videos from day to night or from winter to summer, as shown in the video below, and have applications ranging from turning low-resolution photos into high-resolution alternatives and generating images from written text. GANs have their own limitations, however, that can make them challenging to work with, although these are being tackled by developing more robust GAN variants. Which deep learning software frameworks are available? There are a wide range of deep learning software frameworks, which allow users to design, train and validate deep neural networks, using a range of different programming languages. A popular choice is Google’s TensorFlow software library, which allows users to write in Python, Java, C++, and Swift, and that can be used for a wide range of deep learning tasks such as image and speech recognition, and which executes on a wide range of CPUs, GPUs, and other processors. It is well-documented, and has many tutorials and implemented models that are available. Another popular choice, especially for beginners, is PyTorch, a framework that offers the imperative programming model familiar to developers and allows developers to use standard Python statements. It works with deep neural networks ranging from CNNs to RNNs and runs efficiently on GPUs. Among the wide range of other options are Microsoft’s Cognitive Toolkit, MATLAB, MXNet, Chainer, and Keras. Deep Learning Use Case Examples Robotics Many of the recent developments in robotics have been driven by advances in AI and deep learning. For example, AI enables robots to sense and respond to their environment. This capability increases the range of functions they can perform, from navigating their way around warehouse floors to sorting and handling objects that are uneven, fragile, or jumbled together. Something as simple as picking up a strawberry is an easy task for humans, but it has been remarkably difficult for robots to perform. As AI progresses, that progress will enhance the capabilities of robots. Developments in AI mean we can expect the robots of the future to increasingly be used as human assistants. They will not only be used to understand and answer questions, as some are used today. They will also be able to act on voice commands and gestures, even anticipate a worker’s next move. Today, collaborative robots already work alongside humans, with humans and robots each performing separate tasks that are best suited to their strengths. Agriculture AI has the potential to revolutionize farming. Today, deep learning enables farmers to deploy equipment that can see and differentiate between crop plants and weeds. This capability allows weeding machines to selectively spray herbicides on weeds and leave other plants untouched. Farming machines that use deep learning–enabled computer vision can even optimize individual plants in a field by selectively spraying herbicides, fertilizers, fungicides, insecticides, and biologicals. In addition to reducing herbicide use and improving farm output, deep learning can be further extended to other farming operations such as applying fertilizer, performing irrigation, and harvesting. Medical Imaging and Healthcare Deep learning has been particularly effective in medical imaging, due to the availability of high-quality data and the ability of convolutional neural networks to classify images. For example, deep learning can be as effective as a dermatologist in classifying skin cancers, if not more so. Several vendors have already received FDA approval for deep learning algorithms for diagnostic purposes, including image analysis for oncology and retina diseases. Deep learning is also making significant inroads into improving healthcare quality by predicting medical events from Virtual assistants Whether it’s Alexa or Siri or Cortana, the virtual assistants of online service providers use deep learning to help understand your speech and the language humans use when they interact with them. Translations In a similar way, deep learning algorithms can automatically translate between languages. This can be powerful for travelers, business people and those in government. By the way check out this funny fail by Google translate (For all you triggered Bieber fans out there right now it’s not me it’s Google’s fault OK?! :] ) PVision for driverless delivery trucks, drones and autonomous cars The way an autonomous vehicle understands the realities of the road and how to respond to them whether it’s a stop sign, a ball in the street or another vehicle is through deep learning algorithms. The more data the algorithms receive, the better they are able to act human-like in their information processing — knowing a stop sign covered with snow is still a stop sign. Chatbots and service bots Chatbots and service bots that provide customer service for a lot of companies are able to respond in an intelligent and helpful way to an increasing amount of auditory and text questions thanks to deep learning. Image colorization Transforming black-and-white images into color was formerly a task done meticulously by human hand. Today, deep learning algorithms are able to use the context and objects in the images to color them to basically recreate the black-and-white image in color. The results are impressive and accurate. Facial recognition Credit For The Image Goes To: Yann LeCun Deep learning is being used for facial recognition not only for security purposes but for tagging people on Facebook posts and we might be able to pay for items in a store just by using our faces in the near future. The challenges for deep-learning algorithms for facial recognition is knowing it’s the same person even when they have changed hairstyles, grown or shaved off a beard or if the image taken is poor due to bad lighting or an obstruction. The Future of Deep Learning Today, there are various neural network architectures optimized for certain types of inputs and tasks. Convolution neural networks are very good at classifying images. Another form of deep learning architecture uses recurrent neural networks to process sequential data. Both convolution and recurrent neural network models perform what is known as supervised learning, which means they need to be supplied with large amounts of data to learn. In the future, more sophisticated types of AI will use unsupervised learning. A significant amount of research is being devoted to unsupervised and semisupervised learning technology. Reinforcement learning is a slightly different paradigm to deep learning in which an agent learns by trial and error in a simulated environment solely from rewards and punishments. Deep learning extensions into this domain are referred to as deep reinforcement learning (DRL). There has been considerable progress in this field, as demonstrated by DRL programs beating humans in the ancient game of GO. Designing neural network architectures to solve problems is incredibly hard, made even more complex with many hyperparameters to tune and many loss functions to choose from to optimize. There has been a lot of research activity to learn good neural network architectures autonomously. Learning to learn, also known as metalearning or AutoML, is making steady progress. Current artificial neural networks were based on 1950s understanding of how human brains process information. Neuroscience has made considerable progress since then, and deep learning architectures have become so sophisticated that they seem to exhibit structures such as grid cells, which are present in biological neural brains used for navigation. Both neuroscience and deep learning can benefit each other from cross-pollination of ideas, and it’s highly likely that these fields will begin to merge at some point. Conclusion Now with deep learning and GPUs we can achieve higher accuracy at a practical speed! Deep learning is also much more accessible in terms of the learning curve. Much of the open source software is very easy to use and getting a simple language translator, chat bot, or image recognizer isn’t too challenging. Less complex math and coding, more making cool stuff! Thanks for reading!😄 🙌

Introduction to Deep Learning

Olaf Witkowski is the Chief Scientist at Cross Labs, which aims to bridge the divide between intelligence science and AI technology. A researcher of artificial life, Witkowski started in artificial intelligence by exploring the replication of human speech through machines. He founded Commentag in 2007, and in 2009 moved to Japan to continue research, where he first became interested in artificial life. In his own words, Witkowski says, “artificial intelligence means that you are trying to copy human intelligence as best as possible. and recreate it from the fundamental knowledge we have acquired. It’s more constructive. It’s a bit like the Richard Feynman quote: what I cannot create, I do not understand.” Artificial life says, okay, that’s good, but let’s try to understand human intelligence In this interview, we talk to Witkowski about his work in artificial life, how it will advance technology, and why he thinks deep learning is already dead. I know your focus is on artificial life, but can you tell us to what extent you work within the field of AI? I work with an AI company called Cross Compass as part of Cross Labs. I’ve talked with a lot of AI companies over the last three years in Japan. I wanted to do more research into artificial life, and Cross Compass founded a research center which I direct. At Cross Labs we cover three main areas: the neuroscience of intelligence, the theory of agency and learning, and collective AI. What trends have you noticed in AI and machine learning over the last few years? So the thing is, I’m very biased. I really like things that are open-ended. For example, things that create complexity, or things that model new parts of the mind. I really like attention-based algorithms; not like attention to humans, but attention mechanisms in neural nets. Motivation has been very interesting. The basics of Karl Friston’s free energy principle are really attractive. Valuing surprise and the idea that you want to be surprised. That idea existed already, but Friston translated it, he modeled the error and the difference between prediction and result. It’s called predictive coding, and it’s used in machine learning. I’m not in love with everything about it, but at the end of the day, it works. I’m attracted to taking little principles from living systems and translating them into a coded system. Have you noticed more in terms of complexity and open-ended approaches? Yeah. So the trends I’ve noticed, they’re not new. People have been working on them. But I think GANs are super nice. I could talk about GANs for a long time, because it’s an example of two models interacting with each other adversarially to create complexity and make new discoveries. We have one student who is using GANs for cellular automata, trying to see if you can discover new rules and create more complexity. It was partially inspired by . You see immediately the application of that, right? How can we create systems automatically? What’s the meta algorithm that can create more discovery for free? So it’s about algorithms that try to discover more things. this study I know discovery is a big part of your research into artificial life. What year did you make the move into the field? 2010, I think? Given my interest in the nature of language, it was natural to try to recreate it with AI. And to recreate it properly, one must understand what principles in the living systems are necessary to make this communication happen. This happens to be an important research question in Artificial Life. It’s a rather small field, but really creative and promising, with plenty of key researchers here in Japan. In 2018 we organized a big conference in Tokyo, and should have another one next September. The field of AI itself is super interesting, and it will keep growing, but deep learning – which is what people often mean when they say AI – is already dead. What I mean is, it’s kind of an old idea. It was invented in the 60s, although the first models weren’t doing any learning. The first attempts at learning came later, even before backpropagation existed. We recently made neural learning work extremely well with convolutions and other tricks, as we also have huge amounts of computation we didn’t have before. It’s pretty cool, and we’ve made this combination with backprop in those architectures, and now we’re applying this research to many application areas. But the research in deep learning is actually dead. When you say dead, do you mean there’s a ceiling and we’ve hit it? We are on a plateau. And we’ll be stuck on that plateau until we discover learning principles that are radically different. As long as we spend time on tuning the current paradigm, we are not focusing on searching for other, different AI algorithms. Within the field of AI, researchers are slowly starting to explore new ideas, but ALife (artificial life) was already doing that for a few decades. This is why I feel like ALife is the natural next step in the large field. It’s not the only one, of course, but it’s a very promising one. What areas of artificial life do you find interesting? ALife studies how to reproduce life from the bottom up. There is a subfield of ALife called open-ended evolution, an area I’m particularly interested in. It looks at mechanisms able to create novelty by itself, forever. So, think of the Earth. It’s isolated, but it keeps creating more diversity. We have different species, we create and recreate theorems as humans; so why do we have all this diversity, and all this richness? Think of the earth as a box; how come when you shake it, and wait for long enough, suddenly you get humans? And then from that you have technology, and then you have robots, and then you have robots that kill all the humans… Well, okay, that last one was a joke, but you get all the exciting stuff. In our research into open ended evolution, we try to understand phase transitions in the emergence of intelligence: how come if you shake a box of stuff, the stuff turns into more interesting stuff? How come atoms naturally evolve through time into intelligent machines? I apply those principles to machine learning, basically. You’re applying that idea to create algorithms that keep creating, and within that you have the invention of, or rather the emergence of, goals. Concepts like intrinsic motivation are a close example. The idea is to get intelligent behavior without hard-coding it, and without giving the machine any data. Instead, from internal goals, the agent will try to , new solutions. We want to avoid hard coding rules into a robot or just presenting it with loads of data. discover solutions We want them to be fully emergent instead, by using fundamental principles of intelligence, such as maximizing their “empowerment” over the whole system, or their acquisition of relevant information. We want the agent to discover by itself the goals that will drive it to better solutions, and keep improving them continuously. For example, we want robots to implement curiosity, and all skills that make babies excellent learners. We want to implant the mechanisms that create similar drives inside a robot. If it works, that’s a better way to get intrinsic motivation. I guess along the way to looking at what life is, you’ve got smaller goals like what is intelligence, what is curiosity, what is motivation. From your perspective, what kind of practical solutions or use cases do you see for this kind of research? So I think if you’re trying to solve problems now, that can drive you for a while, but it’s the same as the environment. What I mean is, when you try to fix a problem within a time window of one to four years, for example, yeah, you can fix or patch it, but it will reappear in a few years. So I think of ALife as long-term research. It’s not going to solve problems itself; it’s going to solve problems as a side effect of better understanding what nature is about. In terms of structuring your experiments, how do you define what you’re looking for? There is of course the big question: what is life? What is intelligence? So that’s my big question. What is the nature of intelligent systems? What is the difference between an intelligent system and, say, this cup of coffee? People say cups are not intelligent because they’re just objects, but actually we are objects too; we’re just more complex. We have other degrees of freedom too, but even a cup has affordances; ways to grab it, et cetera. It has intelligence in the sense it is made in a shape that has been selected over many generations in terms of cultural evolution. This is the same for all objects, and it’s a sort of intelligence. So there are things that we can measure here, but we don’t have yet the theory to measure it. So trying to understand that is the purpose of the field, and the part of the field I’m interested in. Where do you see AI and ALife going next? So we know that backprop is really quick, but we need to either tweak AI or find another paradigm. Another paradigm is going to take time, so for now we can add tweaks to make it more interesting. Examples for AI are attention mechanisms, making them adversarial, or making them communicate, which hasn’t been talked about so much. That’s actually a lot of my research; collective AI. So GANs is about conflict, right? But I believe in nature you also have parasitism and cooperation mutualisms and that’s actually very easy to translate into math; it’s basically networks helping networks, with both gaining from the interaction. We did it with two networks, and the basic system works by having a teacher and a learner, and you try to transfer knowledge. It’s very tricky if they have different tasks, but maybe they still have knowledge they can transfer to each other. The transfer of knowledge is one thing, but there’s also collaboration; maybe you have systems that can discover new solutions together they couldn’t discover alone. We wrote about this, and it’s an exciting research direction. It works on the principle that maybe there is an intrinsic value to misinterpreting information, leading to the type of learning that would not have happened without social relations among the learning agents. Communication-based AI, basically. a paper Also published on: https://lionbridge.ai/articles/deep-learning-is-dead-towards-artificial-life-with-olaf-witkowski/

Deep Learning is Already Dead: Towards Artificial Life with Olaf Witkowski

Introduction Model training is arguably the most time consuming, and computationally demanding part of the Machine Learning pipeline. Depending on the complexity of your model, or search space, it can take weeks or even months to find an adequate set of parameters that allow your model to fit the data. is a state-of-the-art approach for speeding up model training and hyperparameter optimization. Our benchmarking studies have shown that Predictive Early Stopping can speed up model training of the underlying infrastructure. Predictive Early Stopping by up to 30% independent We build on insights gathered from projects such as Learning Curve Extrapolation, Hyperband, and Median Stopping, in order to . create a predictive model that can estimate the convergence value of a loss curve Comet is able to leverage model data, such as hyperparameters and loss curves, from in the section of its platform to create a model whose predictions generalize across hyperparameters and model architectures. over two million models public In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs. In addition to predicting the convergence value, our Predictive Early Stopping product provides an estimate of the probability that the current model will outperform the best model result seen in the current training sweep. In some cases we are able to provide an estimate of convergence, hundreds of epochs before it actually occurs . These predictions allow us to terminate the training of underperforming models, so that the search process is spent evaluating only the most promising candidates. Benchmarking: We tested our Predictive Early Stopping method in three different settings: 1. A hyperparameter search that optimizes the parameters of a function that acts as a surrogate for a neural network 2. A hyperparameter search to optimize a 6 layer CNN on CIFAR10 using the optimizer, with and without predictive early stopping. SMAC 3. A hyperparameter search to optimize the same 6 layer CNN using . Random Search with Hyperband vs Random Search with Predictive Early Stopping Results for the Surrogate Function: In our first test, we setup an exponentially decaying function as our surrogate for a Neural Network. We ran this surrogate model for 20 steps and determined the optimal values for the parameters of this function using Comet’s Bayesian Optimizer and Predictive Early Stopping.During the hyperparameter search we observed that suboptimal models were to train the full 20 steps. , illustrates our stopping mechanism. not allowed Figure 1 Figure 1. Surrogate Model Loss Curves Results for the CNN model with SMAC: Our benchmarking test for the CNN model was setup in the following way: We used the SMAC optimizer to estimate the following hyperparameters in a 6-Layer CNN model. The model hyperparameters, and architecture were based on AlexNet. { :{ : , :[ , ]
    }, :{ : , :[ , ]
    }, : { : , : [ , ]
    }
} "learning_rate" "type" "loguniform" "value" 0.0000001 0.01 "learning_rate_decay" "type" "uniform" "value" 0.000001 0.001 "weight_decay" "type" "loguniform" "value" 0.0000005 0.005 We ran 8 trials of the optimizer, with and without Predictive Early Stopping. Each optimizer trial is given a 6 hour budget to evaluate as many configurations as possible. Each hyperparameter configuration was allowed to train for a maximum of , and the validation set was evaluated at the end of every epoch. The validation loss was used by our Predictive Early Stopping model to determine whether or not to terminate a hyperparameter configuration. 100 epochs At the end of all the trials, we determined the of the test loss across all trials as a function of the total number of epochs in the hyperparameter sweep that are required to achieve that loss value. mean value We can see in , that using Predictive Early Stopping allows SMAC to get to a comparable loss value, almost 300 epochs faster. This is a in the amount of time spent doing hyperparameter optimization. Figure 2 25% decrease Figure 2. Mean Test Loss across trials vs Total Number of Epochs We also divided the hyperparameter configurations into quantiles, based on the final validation loss. We then calculated the average amount of epochs that the Optimizer spent in each quantile across all trials. In , we see that both SMAC and Predictive-Early-Stopping-SMAC spend roughly the same amount of time evaluating the top 25% and top 50% of configurations. However, Predictive Early Stopping spends training models in the bottom 50% of results, and epochs in bottom 25%. Figure 3 30 fewer epochs 20 fewer In , we see sample loss curves from a hyperparameter sweep with Predictive Early Stopping. Suboptimal configurations are stopped well before the total number of allowed training steps. Figure 4 Figure 4. Sample validation loss curves from a hyperparameter sweep. Results for the CNN model with Hyperband: We setup the test for Hyperband in a similar manner to SMAC. We specifically used the implemented in . We can think of this as Hyperband with a single bracket. We selected 120 hyperparameter configurations at random. Each configuration was allotted a minimum resource of and allowed to train for a maximum of This leads to a maximum training budget of Asynchronous Successive Halving Pruner Optuna 10 epochs, 100 epochs. 12000 epochs. At every evaluation point, the number of configurations are reduced by a factor of based on the worst performing validation losses In our experiments we evaluated Hyperband with N . N values of 2, 4 and 8. For the Predictive Early Stopping, we tested each configuration with different values of an parameter. Configurations were evaluated every 10, 15, and 20 epochs. At every evaluation we estimated the probability that the current configuration will be better than the best configuration seen so far. If this probability is less than a we terminate the current configuration. , and are both configurable hyperparameters for Predictive Early Stopping. interval threshold, in our case 90%, Threshold Interval We then determine the best validation loss achieved in the hyperparameter sweep, and the number of epochs remaining in the budget after the sweep. In however, Predictive Early Stopping is able to evaluate the configurations using just of the total budget, compared to Hyperband which uses Figure 5a, we see that all approaches find the same best value for the validation loss, 15% 25%. This is a 10% improvement in speed. Figure 5. Predictive Early Stopping vs Hyperband a) Best Value of Validation Loss b) Number of Epochs left in the Training Budget Conclusion Predictive Early Stopping has very obvious time, energy, and cost saving benefits. Wasted compute cycles aren’t good for the environment, or a researcher’s budget. The recently published a on the rising computational costs of training machine learning models, and how these increasingly large energy requirements adversely affect the environment. The paper states that current advancements in state of the art AI research have largely focussed on metrics such as accuracy, or error, at the cost of being environmentally unfriendly. They call this paradigm, Red AI. To counter the prevalence of Red AI research, they propose a shift towards AI research that places an emphasis on computational efficiency: Green AI. Allen Institute for AI report The paper recommends tracking the efficiency of an AI algorithm based on the total number of floating point operations (FPO’s) required to generate a result. The total number of FPO’s directly correlates with the number of hyperparameter configurations that are evaluated during tuning, as well as the number of training iterations spent on each configuration. We hope our efforts with Predictive Early Stopping contribute towards increasing the computational efficiency of the hyperparameter search process. We see the this tool as a way to lower some of the monetary barriers associated with AI research, and as a step towards adopting Green AI practices. In our next post we will describe applying predictive early stopping in individual runs that do not belong to a larger parameter search. *Predictive Early Stopping is available as an add-on with the purchase of Comet Teams or Comet Enterprise. Patent protection is being sought on Predictive Early Stopping via one or more pending patent applications. Learn more and sign-up here .

Predictive Early Stopping - A Meta Learning Approach 

Welcome back to our 'Behind the Startup' series where we dive into the journeys of innovative companies reshaping industries. Today, we're revisiting Travala.com through a conversation with Juan Otero, who last joined us in May 2020. At that time, Juan forecasted a recovery in the travel sector which largely rang true. As we catch up with him today, we'll learn how Travala.com has navigated the evolving landscape of the travel industry, especially in light of the ongoing shifts towards digital and decentralized technologies. Join us as Juan shares insights on the company's growth, the integration of web3 technologies, and how these strategies are positioning Travala.com for future challenges and opportunities in the travel sector. Ishan Pandey: Welcome back, Juan; it's a pleasure to have you in our 'Behind the Startup' series. Last time we spoke in May 2020, you correctly predicted about a recovery starting in Q3, particularly for domestic travel post Covid-19, with a full recovery including international travel by Q4. As we continue our 'Behind the Startup' series, could you update us on how Travala.com has evolved since then, especially considering the ongoing shifts in the travel industry? Juan Otero: Thanks Ishan, great to be speaking with you again. So much has happened at Travala.com in the last four years as the travel industry rebounded and the web3 industry continued to swell. In that time, we’ve significantly expanded our offering through partnerships with some of the biggest names in travel, including Expedia and KAYAK. Our accommodation selection multiplied to over 2.2 million properties, and we introduced flights with 600+ airlines as well as 410,000+ activities, with a total of over 3 million travel products now available on Travala.com. Our team has also more than doubled in size to accommodate our growth, with hires across all departments from the management team to security, developers, marketing, and customer support. All of these developments have culminated in tremendous growth in customers, bookings, and revenue, with our gross revenue growing from less than $1 million in Q1 2020 to over $22 million in Q1 2024. Ishan Pandey: During our last discussion, you highlighted some pivotal adjustments made at Travala.com in response to the pandemic. Looking at the current times, how has your business model or strategy changed to align with the new dynamics of the travel industry? Juan Otero: From the rise of domestic travel to the influx of digital nomads across the globe, the pandemic has prompted the emergence of many new trends that look likely to continue well into the future. Where we once might’ve promoted a weekend away, we’re now encouraging longer, more immersive experiences through marketing, exclusive mobile-only deals on stays, and by offering an average of 16% savings on hotels after booking a flight. One of the often overlooked developments in recent years has been the rapid emergence of metasearch engines. With shifting behaviours in how savvy travellers search for, compare, and book travel, we recognised the importance of these metasearch engines—which have quickly gained market share when it comes to the discovery phase of travel—and became the first crypto-friendly online travel agency to be listed on leading metasearch engines KAYAK and Google Hotels, alongside incumbents in the travel industry. Ishan Pandey: Cryptocurrencies have seen substantial growth in value and popularity since we last talked. How is Travala.com planning to capitalise on this trend? Are there new strategies or technologies you're implementing? Juan Otero: We’ve honed our web3 strategy by partnering with projects to promote cross-community growth and leverage the power of highly engaged web3 audiences. As a result, we now have over 100 cryptocurrency payment options and frequently conduct campaigns with partners to promote the ability to travel with crypto on Travala.com. From leading web3 events like TOKEN2049 and Paris Blockchain Week to traditional brands interested in expanding their audience—such as Marriott International and the Catalan Tourist Board—we’re leveraging our position as the bridge between the world of traditional travel and the web3 space to continue underscoring our reputation as the leading crypto-native travel booking service. We’ve also introduced Bitcoin rewards for Platinum and Diamond members of the AVA Smart Program, the travel rewards program available on Travala.com in partnership with the AVA Foundation. These two membership groups can receive up to 10% back in Bitcoin on bookings made via Travala.com, which can be put towards future bookings or transferred out of the platform. Ishan Pandey: With rising competition and shifting consumer behaviors, how does Travala.com intend to differentiate itself in the travel industry? Juan Otero: As consumer behaviours continue evolving and products become increasingly tailored to the millennial and Gen Z audience that now make up the majority of the travel market, the key advantage that we have over more established players in the travel industry is our agility as a team, coupled with our innovation-first approach. We predict that crypto payments will become more commonplace as the web3 industry continues to mature, to the point where it will no longer be a point of differentiation. Other web3 elements (and how successfully they can be incorporated into the travel experience) will therefore be the main area of differentiation for Travala.com. From areas like web3-oriented loyalty programs to digital collectibles, we’re continuously exploring new ways that enhance the experience and make it more rewarding, pushing the boundaries of what travel is in today’s age. Ishan Pandey: Previously, you mentioned the impact of Bitcoin halving on your operations. Can you expand on how recent cryptocurrency events have influenced Travala.com's market dynamics and user behavior? Juan Otero: The positive market action driven by the Bitcoin halving saw the cryptocurrency reach an all-time high in mid-March earlier this year and resulted in our bookings soaring, leading to the strongest quarter in Travala.com’s history. As approximately 75% of our bookings are purchased with crypto, we experience a strong correlation between a crypto bull market and high booking volume on our platform, given the additional value available for travellers to derive from their crypto assets in dollar-value terms.
With the fourth Bitcoin halving now behind us, we expect the crypto market to continue performing well in the next 12–18 months, based on historic trends following previous Bitcoin halving events. Ishan Pandey: Finally, considering all the challenges and opportunities that have emerged, what is your long-term vision for Travala.com in this new era of travel and technology? Juan Otero: We see Travala.com as becoming a hub for crypto travellers to create genuine connections with the world, themselves, and each other. Web3 is a community movement, and our community is constantly brimming with discussions about travel tips, destinations to explore, the broader web3 space, and suggestions for Travala.com. We want Travala.com to be more than just a platform where travellers go to book their travel; in the long term, we envision Travala.com solidifying its position as the world's leading web3 travel ecosystem, where knowledge, creativity, and of course the act of travel itself is rewarded. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Welcome back to our 'Behind the Startup' series where we dive into the journeys of innovative companies reshaping industries. Today, we're revisiting Travala.com through a conversation with Juan Otero, who last joined us in May 2020. At that time, Juan forecasted a recovery in the travel sector which largely rang true. As we catch up with him today, we'll learn how Travala.com has navigated the evolving landscape of the travel industry, especially in light of the ongoing shifts towards digital and decentralized technologies. Join us as Juan shares insights on the company's growth, the integration of web3 technologies, and how these strategies are positioning Travala.com for future challenges and opportunities in the travel sector. Ishan Pandey: Welcome back, Juan; it's a pleasure to have you in our 'Behind the Startup' series. Last time we spoke in May 2020, you correctly predicted about a recovery starting in Q3, particularly for domestic travel post Covid-19, with a full recovery including international travel by Q4. As we continue our 'Behind the Startup' series, could you update us on how Travala.com has evolved since then, especially considering the ongoing shifts in the travel industry? Ishan Pandey: Welcome back, Juan; it's a pleasure to have you in our 'Behind the Startup' series. Last time we spoke in May 2020, you correctly predicted about a recovery starting in Q3, particularly for domestic travel post Covid-19, with a full recovery including international travel by Q4. As we continue our 'Behind the Startup' series, could you update us on how Travala.com has evolved since then, especially considering the ongoing shifts in the travel industry? Ishan Pandey: Welcome back, Juan; it's a pleasure to have you in our 'Behind the Startup' series. Last time we spoke in May 2020, you correctly predicted about a recovery starting in Q3, particularly for domestic travel post Covid-19, with a full recovery including international travel by Q4. As we continue our 'Behind the Startup' series, could you update us on how Travala.com has evolved since then, especially considering the ongoing shifts in the travel industry? Travala.com Juan Otero: Thanks Ishan, great to be speaking with you again. So much has happened at Travala.com in the last four years as the travel industry rebounded and the web3 industry continued to swell. In that time, we’ve significantly expanded our offering through partnerships with some of the biggest names in travel, including Expedia and KAYAK. Juan Otero: Travala.com Our accommodation selection multiplied to over 2.2 million properties, and we introduced flights with 600+ airlines as well as 410,000+ activities, with a total of over 3 million travel products now available on Travala.com . Travala.com Our team has also more than doubled in size to accommodate our growth, with hires across all departments from the management team to security, developers, marketing, and customer support. All of these developments have culminated in tremendous growth in customers, bookings, and revenue, with our gross revenue growing from less than $1 million in Q1 2020 to over $22 million in Q1 2024. Ishan Pandey: During our last discussion, you highlighted some pivotal adjustments made at Travala.com in response to the pandemic. Looking at the current times, how has your business model or strategy changed to align with the new dynamics of the travel industry? Ishan Pandey: During our last discussion, you highlighted some pivotal adjustments made at Travala.com in response to the pandemic. Looking at the current times, how has your business model or strategy changed to align with the new dynamics of the travel industry? Ishan Pandey: During our last discussion, you highlighted some pivotal adjustments made at Travala.com in response to the pandemic. Looking at the current times, how has your business model or strategy changed to align with the new dynamics of the travel industry? Travala.com Juan Otero: From the rise of domestic travel to the influx of digital nomads across the globe, the pandemic has prompted the emergence of many new trends that look likely to continue well into the future. Where we once might’ve promoted a weekend away, we’re now encouraging longer, more immersive experiences through marketing, exclusive mobile-only deals on stays, and by offering an average of 16% savings on hotels after booking a flight. Juan Otero: One of the often overlooked developments in recent years has been the rapid emergence of metasearch engines. With shifting behaviours in how savvy travellers search for, compare, and book travel, we recognised the importance of these metasearch engines—which have quickly gained market share when it comes to the discovery phase of travel—and became the first crypto-friendly online travel agency to be listed on leading metasearch engines KAYAK and Google Hotels, alongside incumbents in the travel industry. Ishan Pandey: Cryptocurrencies have seen substantial growth in value and popularity since we last talked. How is Travala.com planning to capitalise on this trend? Are there new strategies or technologies you're implementing? Ishan Pandey: Cryptocurrencies have seen substantial growth in value and popularity since we last talked. How is Travala.com planning to capitalise on this trend? Are there new strategies or technologies you're implementing? Ishan Pandey: Cryptocurrencies have seen substantial growth in value and popularity since we last talked. How is Travala.com planning to capitalise on this trend? Are there new strategies or technologies you're implementing? Travala.com Juan Otero: We’ve honed our web3 strategy by partnering with projects to promote cross-community growth and leverage the power of highly engaged web3 audiences. Juan Otero: As a result, we now have over 100 cryptocurrency payment options and frequently conduct campaigns with partners to promote the ability to travel with crypto on Travala.com . From leading web3 events like TOKEN2049 and Paris Blockchain Week to traditional brands interested in expanding their audience—such as Marriott International and the Catalan Tourist Board—we’re leveraging our position as the bridge between the world of traditional travel and the web3 space to continue underscoring our reputation as the leading crypto-native travel booking service. Travala.com We’ve also introduced Bitcoin rewards for Platinum and Diamond members of the AVA Smart Program, the travel rewards program available on Travala.com in partnership with the AVA Foundation. These two membership groups can receive up to 10% back in Bitcoin on bookings made via Travala.com , which can be put towards future bookings or transferred out of the platform. Travala.com Travala.com Ishan Pandey: With rising competition and shifting consumer behaviors, how does Travala.com intend to differentiate itself in the travel industry? Ishan Pandey: With rising competition and shifting consumer behaviors, how does Travala.com intend to differentiate itself in the travel industry? Ishan Pandey: With rising competition and shifting consumer behaviors, how does Travala.com intend to differentiate itself in the travel industry? Travala.com Juan Otero: As consumer behaviours continue evolving and products become increasingly tailored to the millennial and Gen Z audience that now make up the majority of the travel market, the key advantage that we have over more established players in the travel industry is our agility as a team, coupled with our innovation-first approach. Juan Otero: We predict that crypto payments will become more commonplace as the web3 industry continues to mature, to the point where it will no longer be a point of differentiation. Other web3 elements (and how successfully they can be incorporated into the travel experience) will therefore be the main area of differentiation for Travala.com . From areas like web3-oriented loyalty programs to digital collectibles, we’re continuously exploring new ways that enhance the experience and make it more rewarding, pushing the boundaries of what travel is in today’s age. Travala.com Ishan Pandey: Previously, you mentioned the impact of Bitcoin halving on your operations. Can you expand on how recent cryptocurrency events have influenced Travala.com's market dynamics and user behavior? Ishan Pandey: Previously, you mentioned the impact of Bitcoin halving on your operations. Can you expand on how recent cryptocurrency events have influenced Travala.com's market dynamics and user behavior? Ishan Pandey: Previously, you mentioned the impact of Bitcoin halving on your operations. Can you expand on how recent cryptocurrency events have influenced Travala.com 's market dynamics and user behavior? Travala.com Juan Otero: The positive market action driven by the Bitcoin halving saw the cryptocurrency reach an all-time high in mid-March earlier this year and resulted in our bookings soaring, leading to the strongest quarter in Travala.com ’s history. Juan Otero: Travala.com As approximately 75% of our bookings are purchased with crypto, we experience a strong correlation between a crypto bull market and high booking volume on our platform, given the additional value available for travellers to derive from their crypto assets in dollar-value terms.
With the fourth Bitcoin halving now behind us, we expect the crypto market to continue performing well in the next 12–18 months, based on historic trends following previous Bitcoin halving events. Ishan Pandey: Finally, considering all the challenges and opportunities that have emerged, what is your long-term vision for Travala.com in this new era of travel and technology? Ishan Pandey: Finally, considering all the challenges and opportunities that have emerged, what is your long-term vision for Travala.com in this new era of travel and technology? Ishan Pandey: Finally, considering all the challenges and opportunities that have emerged, what is your long-term vision for Travala.com in this new era of travel and technology? Travala.com Juan Otero: We see Travala.com as becoming a hub for crypto travellers to create genuine connections with the world, themselves, and each other. Web3 is a community movement, and our community is constantly brimming with discussions about travel tips, destinations to explore, the broader web3 space, and suggestions for Travala.com . Juan Otero: Travala.com Travala.com We want Travala.com to be more than just a platform where travellers go to book their travel; in the long term, we envision Travala.com solidifying its position as the world's leading web3 travel ecosystem, where knowledge, creativity, and of course the act of travel itself is rewarded. Travala.com Travala.com Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program . HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR. Vested Interest Disclosure: business blogging program business blogging program

MetalBear

metalbear.co

-- employees

Since n.d.

COMPANY RANKING

METALBEAR

EVERGREEN INDEX #6628

MetalBear WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

MetalBear

metalbear.co

-- employees

Since n.d.

COMPANY RANKING

METALBEAR

EVERGREEN INDEX #6628

RELATED COMPANIESRANK

STORIES ABOUT METALBEAR

MetalBear WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!