Featureform

featureform

Hello everyone! My name is Vlad Vinogradov, I am the head of the computer vision department at EORA.ai. We have been engaged in deep learning for over three years and during that time we have implemented many projects for both Russian and international clients, which included the research and training of models. Recently, we have focused on solving the problems of finding similar images and at the moment have created systems for finding logos, drawings, furniture, clothing, and other goods. This post is for Machine Learning Engineers and is based on my talk Finding Similar Images - . A Guide from A to Z, which was published by the Open Data Science community at Data Fest Online 2020 This article provides background information on the recommended methods used in the Image Recovery task. After reading the article, you will be able to create a search engine for similar images for your objective from scratch (not counting the process of developing a production solution). About the project A Similar Image Retrieval (aka Content-Based Image Retrieval or CBIR) is any search that involves images. The picture above from the article will tell you about the problem most easily. Recent Advance in Content-based Image Retrieval: A Literature Survey (2017) Nowadays, the "Search by photo" approach is being used actively, in particular, in e-commerce services (AliExpress, Wildberries, etc.). "Keyword search" (with an understanding of the content of images) has long settled in the search engines Google, Yandex, etc., but has not yet reached marketplaces and other private search engines. I think that since the appearance of the sensational in computer vision circles, the globalization of this approach will accelerate. CLIP: Connecting Text and Images Since our team specializes in neural networks in computer vision, in this article I will focus only on the Search by Photo approach. Basic service components The model can be made on the classic CV or on the basis of a neural network. Model input - image, output - D-dimensional descriptor/embedding. In the case of the classics, it can be a combination of SIFT-descriptor + Bag of Visual Words. In the case of a neural network, a standard backbone like ResNet, EfficientNet, etc. + intricate pooling layers + clever learning techniques, which we will talk about later. I can say that if you have enough data or good practice, neural networks will almost always benefit greatly (we checked), so we will focus on them. Step 1. Train the model. Indexing is a running of the trained model on all images and writing embeddings to a special index for quick search. Step 2. Indexing the base of images. Using the image uploaded by the user, the model is run, the embedding is obtained and this embedding is compared with the rest in the database. The search result is the search results sorted by relevance. Step 3. Search. Neural Networks and Metric Learning The neural network in the task of finding similarities is used as a feature extractor (backbone). The choice of backbone depends on the volume and complexity of the data - you can consider everything from to . ResNet18 Visual Transformer The first feature of the models in Image Retrieval is the magic in the head of the neural network. On the__ __, they are fighting for building the best descriptors - there are Combined Global Descriptors with parallel pools and Batch Drop Block for a more even distribution of activation over the output feature map. Image Retrieval leaderboard The second main feature is the loss functions. There are a lot of them. In alone, there are over a dozen recommended pair losses. There are also the same numbers of classification ones. The main point of all these losses is to train the neural network to transform the image into a vector of linearly separable space, so that further these vectors can be compared by cosine or Euclidean distance: similar images will have close embeddings, dissimilar images will have distant ones. Let's take a closer look. Deep Image Retrieval: A Survey Loss Functions Contrastive Loss This is a double loss, i.e. objects are compared by the distance between each other. The neural network is penalized for the distance from each other of the embeddings of images p and q if these images are actually similar. Similarly, there is a penalty for the proximity of embeddings, the images of which are actually different from each other. In this case, in the latter case, we set the boundary m (for example, 0.5), overcoming which, we believe that the neural network has coped with the task of "separating" dissimilar images. Triplet Loss Here we aim to minimize the distance from the anchor to the positive and maximize the distance from the anchor to the negative. Triplet Loss was first introduced in Google's on facial recognition and has long been a state-of-the-art solution. FaceNet article N-tupled Loss N-tupled Loss- a development of Triplet Loss, which also takes anchor and positive, but uses multiple negatives instead of one negative. Angular Additive Margin (ArcFace) The problem of paired losses is in choosing combinations of positives, negatives, and anchors - if they are just taken uniformly random from the dataset, then the problem of "light pairs" arises. These are such simple pairs of images for which the loss will be 0. It turns out that the network converges quickly enough to a state in which most of the elements in the batch will be "easy" for it, and the loss for them will turn out to be zero - the network will stop learning. To avoid this problem, they began to come up with sophisticated mining techniques for pairs - hard negative and hard positive mining. You can read more about the problem in . There is also a , which implements many mining methods, and in general, the library contains a lot of useful information on the Metric Learning task on PyTorch. this article PML library Another solution to the problem is classification losses. Consider one popular bug feature that led to state-of-the-art facial recognition three years ago - . ArcFace The main idea is to add an indent m to the usual cross-entropy, which distributes the embeddings of images of one class in the region of the centroid of this class so that they are all separated from the clusters of embeddings of other classes by at least an angle m. This seems to be the perfect loss feature, especially when you look at the . But you need to keep in mind that it will work only if there is a classification markup. If you don't have one, you will have to work with paired losses. MegaFace benchmark Here I visually show you which loss functions are best used when you have single-class and multi-class markup (you can derive paired markup from the latter by counting the percentage of intersection between multilabel vectors of examples). Pooling Let's go back to the neural network architecture and consider a couple of pooling layers used in Image Retrieval tasks. R-MAC is a pooling layer that accepts the output map of the neural network (before the global pooling or classification layers) and returns a descriptor vector calculated as the sum of activations in different windows of the output map. Here, the activation of the window is to take the maximum for this window for each channel independently. Regional Maximum Activation of Convolutions (R-MAC) The resulting descriptor takes into account the local features of the image at different scales, thereby creating a rich feature description. This descriptor itself can be an embedding, so it can be sent directly to the loss function. Generalized Mean Generalized Mean (GeM) is a simple pooling that can improve the quality of the output descriptor. The bottom line is that the classical average pooling can be generalized to the lambda norm. By increasing the lambda, we make the network focus on significant parts of the image, which can be important in certain tasks. Ranging Index The key to a high-quality search for similar images is ranking, i.e. displays the most relevant examples for a given query. It is characterized by the speed of building the descriptor index, the speed of the search, and the memory consumed. The simplest thing is to keep the embeddings "head-on" and do a brute-force search on them, for example, using the cosine distance. Problems appear when there are a lot of embeddings - millions, tens of millions, or even more. The search speed is significantly reduced, the amount of heap occupied increases. One positive thing remains - the search quality is perfect with the existing embeddings. These problems can be solved at the expense of quality - to store embeddings, not in their original form, but compressed (quantized). And also to change the search strategy - not to search for brute-force, but to try for the minimum number of comparisons to find the required number of those closest to the given query. There are a large number of efficient frameworks for approximate search for the nearest ones. A special has been created for them, where you can see how each library behaves on different datasets. benchmark Most Popular: , , , . Also, if you want to take indexing with the REST API out of the box, you can consider the . NMSLIB Spotify Annoy Facebook Faiss Google Scann Jina application Re-ranking Researchers in the Information Retrieval field have long understood that ordered search results can be improved by some way of reordering items after the original search results have been received. One such method is Query Expansion. The idea is to use the top-k of the closest elements to generate a new embedding. In the simplest case, you can take the averaged vector, as shown in the picture above. You can also weight embeddings, for example, by distance in the issue or cosine distance from the request. Such improvements are described in a single framework in the article . Optionally, you can apply Query Expansion recursively. Attention-Based Query Expansion Learning k-reciprocal k-reciprocal - a set of elements from top-k, including the k nearest of which is the request itself. On the basis of this set, the process of re-ranking the results is built, one of which is described in the article . By definition, k-reciprocal is closer to the query than k-nearest neighbors. Accordingly, one can roughly consider the elements that are in the k-reciprocal set as knowingly positive and change the weighting rule, for example, for Query Expansion. In this article, a mechanism has been developed for recalculating distances using k-reciprocal sets of the elements themselves in top-k. The article contains a lot of calculations, this is beyond the scope of this post, so I suggest the readers familiarize themselves with it. Re-ranking Person Re-identification with k-reciprocal Encoding Validation We come to the part of checking the quality of the search for similar ones. There are many subtleties in this task that may not be noticed by beginners when they first start working on an Image Retrieval project. Metrics Let's consider such popular metrics in the Image Retrieval task: precision@k, recall@k, R-precision, mAP и nDCG. precision@R Pros Shows the percentage of top-k responses that are relevant. Cons very sensitive to the number of relevant examples for a given query, which does not allow an objective assessment of the quality of search, where there are different numbers of relevant examples for different queries reaching the value 1 is possible only if the number of relevant >= k for all queries R-precision Same as precision@k, where k is set equal to the number of relevant queries. Pros sensitivity to the number k in precision@k disappears, the metric becomes stable Cons you have to know the total number of relevant ones to the request (it can be a problem if not all relevant ones are marked up) Recall@k Shows what proportion of relevant items were found in top-k Pros answers the question of whether found relevant in principle among the top-k stable and well averaged over requests mAP (mean Average Precision) Shows how densely we fill the top of the search results with relevant examples. You can look at this as the amount of information received by the user of the search engine, which has read the least number of pages. Accordingly, the greater the amount of information to the number of pages read, the higher the metric. Pros objective stable assessment of search quality is a one-digit representation of the precision-recall curve, which itself is rich in information for analysis Cons you have to know the total number of relevant ones to the request (it can be a problem if not all relevant ones are marked up) You can read more about metrics in Information Retrieval, including the mAP output, here nDCG (Normalized Discounted Gain) This metric shows how correctly the elements in top-k are ordered among themselves. We will not consider the pros and cons of this metric, since this is the only metric in our list that takes into account the order of elements. However, there are studies showing that this metric is fairly stable when it is necessary to take into account the order and can be suitable in most cases. Estimation a metric is calculated for each request, averaging over all requests macro: pros: there are no significant fluctuations with a different number of relevant to this query cons: all queries are considered equal, even if some are more relevant than others the number of tagged relevant and separately successfully found relevant is summed over all queries, then participates in the fraction of the corresponding metric micro: pros: queries are evaluated taking into account the number of tagged relevant for each of them cons: the metric can become very low / very high if there were a lot of tagged relevant ones for some request and the system unsuccessfully / successfully brought them to the top Validation system proposing to consider two options for validation Validation on a set of queries and selected relevant ones Input: image requests and images relevant to them. There is markup in the form of a list relevant to this query. To calculate metrics, you can calculate the relevance matrix each with each and, based on the binary information about the relevance of the elements, calculate the metrics. Full-base validation Input: image requests, and images relevant to them. There should also be a validation database of images, in which, ideally, all relevant queries are marked. Also, it should not contain query images, otherwise, they will have to be cleaned at the search stage so that they do not clog the top-1. The validation base is involved as a base of negatives - our task is to draw out those that are relevant to it. To calculate metrics, you can go through all the requests, calculate the distances to all elements, including relevant ones, and send them to the metrics calculation function. Completed project example Some companies argue with other companies that the latter should not use the pictorial elements of the former's brand. In such cases, a weaker manufacturer tries to parasitize on the success of a large brand by presenting its products and services under similar symbols. Users also suffer from this - you can mistakenly buy cheese from the wrong manufacturer, which you already trust, but take a fake product without carefully reading the label. An example from a recent one: Apple Is Attempting to Block a Pear Logo Trademark There are special companies, governmental and private, to fight illegal immigration. They have a register of registered trademarks by which they can compare new incoming trademarks and approve / reject an application for the registration of a trademark. Above is an example of the interface of the system. In such systems, a search for similar images will be a good helper - the expert will be able to find analogs faster. WIPO Examples of how our system works The size of the indexed database of images: several million trademarks. Here the first image is a query, on the next line is a list of expected relevant ones, the rest of the lines are what the search engine gives out in order of decreasing relevance. Conclusion That's all. It was overview material. Hopefully, those who are building or about to build similar image search engines have gained some benefit. And vice versa, if you think that I am wrong somewhere, tell me in the comments, I will be glad to receive feedback.

How to Build an Image Search Engine to Find Similar Images

Embeddings have pervaded the data scientist’s toolkit, and dramatically changed how NLP, computer vision, and recommender systems work. However, many data scientists find them archaic and confusing. Many more use them blindly without understanding what they are. In this article, we’ll deep dive into what embeddings are, how they work, and how they are often operationalized in real-world systems. What’s an embedding? To understand embeddings, we must first understand the basic requirements of a machine learning model. Specifically, most machine learning algorithms can only take low-dimensional numerical data as inputs. In the neural network below each of the input features must be numeric. That means that in domains such as recommender systems, we must transform non-numeric variables (ex. items and users) into numbers and vectors. We could try to represent items by a product ID; however, neural networks treat numerical inputs as continuous variables. That means higher numbers are “greater than” lower numbers. It also sees numbers that are similar as being similar items. This makes perfect sense for a field like “age” but is nonsensical when the numbers represent a categorical variable. Prior to embeddings, one of the most common methods used was one-hot encoding. One-Hot Encoding One-hot encoding was a common method for representing categorical variables. This unsupervised technique maps a single category to a vector and generates a binary representation. The actual process is simple. We create a vector with a size equal to the number of categories, with all the values set to 0. We then set the row or rows associated with the given ID or IDs to 1. An illustration of One Hot Encoding This technically works in turning a category into a set of continuous variables, but we literally end up with a huge vector of 0s with a single or a handful of 1’s. This simplicity comes with drawbacks. For variables with many unique categories, it creates an unmanageable number of dimensions. Since each item is technically equidistant in vector space, it omits context around similarity. In vector space, categories with little variance are not any closer together than those with high variance. This means that the terms “Hotdog” and “Hamburger” are no closer together than “Hotdog” and “Pepsi”. As a result, we have no way of evaluating the relationship between two entities. We could generate more one-to-one mappings, or attempt to group them and look for similarities. This requires extensive work and manual labeling that’s typically infeasible. Intuitively, we want to be able to create a denser representation of the categories and maintain some of the implicit relationship information between items. We need a way to reduce the number of categorical variables so we can place items of similar categories closer together. That’s exactly what an embedding is. Embeddings solve the encoding problem Embeddings are dense numerical representations of real-world objects and relationships, expressed as a vector. The vector space quantifies the semantic similarity between categories. Embedding vectors that are close to each other are considered similar. Sometimes, they are used directly for the “Similar items to this” section in an e-commerce store. Other times, embeddings are passed to other models. In those cases, the model can share learnings across similar items rather than treating them as two completely unique categories, as is the case with one-hot encodings. For this reason, embeddings can be used to accurately represent sparse data like clickstreams, text, and e-commerce purchases as features to downstream models. On the other hand, embeddings are much more expensive to compute than one-hot encodings and are far less interpretable. How are Embeddings created? A common way to create an embedding requires us to first set up a supervised machine learning problem. As a side-effect, training that model encodes categories into embedding vectors. For example, we can set up a model that predicts the next movie a user will watch based on what they are watching now. An embedding model will factorize the input into a vector and that vector will be used to predict the next movie. This means that similar vectors are movies that are commonly watched after similar movies. This makes for a great representation to be used for personalization. So even though we are solving a supervised problem, often called the surrogate problem, the actual creation of embeddings is an unsupervised process. Defining a surrogate problem is an art, and dramatically affects the behavior of the embeddings. For example, YouTube’s recommender team realized that using the “predict the next video a user is going to click on” resulted in clickbait becoming rampantly recommended. They moved to “predict the next video and how long they are going to watch it” as a surrogate problem and achieved far better results. Common Embedding Models Below are some common embedding models to learn more about. Principal Component Analysis (PCA) One method for generating embeddings is called Principal Component Analysis (PCA). PCA reduces the dimensionality of an entity by compressing variables into a smaller subset. This allows the model to behave more effectively but makes variables more difficult to interpret, and generally leads to a loss of information. A popular implementation of PCA is a technique called SVD. SVD Singular Value Decomposition, also known as SVD, is a dimensionality reduction technique. SVD reduces the quantity of data set features from N-dimensions to K-dimensions via matrix factorization. For example, let’s represent a user’s video ratings as a matrix of size (Number of users) x (Number of Items) where the value of each cell is the rating that a user gave that item. We first pick a number, k, which is our embedding vector size, and use SVD to turn it into two matrices. One will be (Number of users) x k and the other will be k x (Number of items). In the resulting matrices, if we multiply a user vector by an item vector, we should get our predicted user rating. If we were to multiply both matrices, we’d end up with the original matrix, but densely filled with all of our predicted ratings. It follows that two items that have similar vectors would result in a similar rating from the same user. In this way, we end up creating user and item embeddings. Word2Vec Word2vec generates embeddings from words. Words are encoded into one-hot vectors and fed into a hidden layer that generates hidden weights. Those hidden weights are then used to predict other nearby words. Although these hidden weights are used for training, word2vec will not use them for the task it was trained on. Instead, the hidden weights are returned as embeddings and the model is tossed out. Words that are found in similar contexts will have similar embeddings. Beyond that, embeddings can be used to form analogies. For example, the vector from king to man is very similar to the one from queen to woman. One problem with Word2Vec is that single words have one vector mapping. This means that all semantic uses for a word are combined into one representation. For example, the word “play” in “I’m going to see a play” and “I want to play” will have the same embedding, without the ability to distinguish context. BERT Bidirectional Encoder Representations of Transformers, also known as BERT, is a pre-trained model that solves Word2Vec’s context problems. BERT is trained in two steps. First, it is trained across a huge corpus of data like Wikipedia to generate similar embeddings as Word2Vec. The end-user performs the second training step. They train on a corpus of data that fits their context well, for example, medical literature. BERT will be fine-tuned for that specific use case. Also, to create a word embedding, BERT takes into account the context of the word. That means that the word “play” in “I’m going to see a play” and “I want to play” will correctly have different embeddings. BERT has become the go-to transformer model for generating text embeddings. Embeddings in the Real World Embedding usage started in research labs and quickly became state of the art. Since then, embeddings have found themselves in production machine learning systems across a variety of different fields including NLP, recommender systems, and computer vision. Recommender Systems A recommender system predicts the preferences and ratings of users for a variety of entities/products. The two most common approaches are collaborative filtering and content-based. Collaborative filtering uses actions to train and form recommendations. Modern collaborative filtering systems almost all use embeddings. As an example, we can use the SVD method defined above to build a recommender system. In that system, multiplying a user embedding by an item embedding generates a rating prediction. This provides a clear relationship between users and products. Similar items beget similar ratings from similar users. This attribute can also be used in downstream models. For example, Youtube’s recommender uses embeddings as inputs to a neural network that predicts watch time. Semantic Search Users expect search bars to be smarter than a regex. Whether it’s a customer support page, a blog, or Google, a search bar should understand the intent and context of a query, not just look at words. Search engines used to be built around TF-IDF, which also creates an embedding from text. This kind of semantic search worked by finding a document embedding that’s closest to the query embedding using the nearest neighbor. Today, semantic search utilizes more sophisticated embeddings like BERT and may use them in downstream models. In fact, even Google uses BERT on a large percentage of their queries: . https://searchengineland.com/google-bert-used-on-almost-every-english-query-342193/ Computer Vision In computer vision, embeddings are often used as a way to translate between different contexts. For example, if training a self-driving car, we can transform the image from the car into an embedding and then decide what to do based on that embedded context. By doing so, we can perform transfer learning. We can take a generated image from a game like Grand Theft Auto, turn it into an embedding in the same vector space, and train the driving model without having to feed it tons of expensive, real-world images. Tesla is doing this in practice today. Another interesting example is the AI Art Machine: . It will generate an image based on user input text. For example, if we type in “Nostalgia”, we get the following image. https://colab.research.google.com/drive/1n_xrgKDlGQcCF6O-eL3NOd_x4NSqAUjK#scrollTo=TnMw4FrN6JeB It works by transforming the user’s text and an image into an embedding in the same latent space. It’s composed of four transformers: Image -> Embedding, Text -> Embedding, Embedding -> Text, Image -> Text. With all these, transformations we can translate text to image and visa-versa using an embedding as an intermediate representation. Embedding Operations In the above examples, we see that there are a few common operations applied to embeddings. Any production system that uses embeddings should be able to implement some or all of the below. Averaging Using something like word2vec, we can end up with an embedding for each word, but we often need an embedding for a full sentence. Similarly, in recommender systems, we may know the items a user clicked on recently, but their user embedding may not have been retrained in days. In these situations, we can average embeddings to create higher-level embeddings. In the sentence example, we can create a sentence embedding by averaging each of the word embeddings. In the recommender system, we can create a user embedding by averaging the last N items they clicked. Subtraction/Addition We mentioned earlier how word embeddings also encode analogies via vector differences. Adding and subtracting vectors can be used for a variety of tasks. For example, we can find the average difference between a coat from a cheap brand and a luxury brand. We can store that delta and use it whenever we want to recommend a luxury item that’s similar to the current item that a user is looking at. We can find the difference between a coke and a diet coke and apply it to other drinks, even those that don’t have diet equivalents, to find the closest thing to a diet version. Nearest Neighbor Nearest neighbor (NN) is often the most useful embedding operation. It finds things that are similar to the current embedding. In recommender systems, we can create a user embedding and find items that are most relevant to them. In a search engine, we can find a document that’s most similar to a search query. Nearest neighbor is a computationally expensive operation however. Performed naively it is O(N*K), where N is the number of items and K is the size of each embedding. However, in most cases when we need nearest neighbors, an approximation would suffice. If we recommend five items to a user, and one is technically the sixth closest item, the user probably won’t care. Approximate nearest neighbor (ANN) algorithms typically drop the complexity of a lookup to O(log(n)). Implementations of ANN There are many different algorithms to efficiently find approximate nearest neighbors, and many implementations of each of them. We’ll list a few of the most common algorithms and implementations below and break down how they work at a high level. Spotify’s Annoy In Spotify’s ANN implementation (Annoy), the embeddings are turned into a forest of trees. Each tree is built using . At every intermediate node in the tree, a random hyperplane is chosen, which divides the space into two subspaces. This hyperplane is chosen by sampling two points from the subset and taking the hyperplane equidistant from them. This is performed k times to generate a forest. Lookups are done via in-order traversal of the nearest tree. Annoy’s approach allows the index to be split into multiple static files, the index can be mapped in memory, and the number of trees can be tweaked to change speed and accuracy. random projections Here’s how Annoy breaks down the embeddings into an index of multiple trees via random projections. Locality Sensitive Hashing (LSH) Locality Sensitive Hashing (LSH) is another common approach. LSH employs a hash table and stores nearby points in their respective buckets. To create the index, LSH runs many hashing functions which place similar points in the same bucket. In doing so, LSH keeps points with large distances in separate buckets. To retrieve the nearest neighbor, the point in question is hashed, a lookup is done and the closest query point is returned. Some pros include a sub-linear run time, zero reliance on data distribution, and the ability to fine-tune accuracy with the existing data structure. Facebook’s FAISS and Hierarchical Navigable Small World Graphs (HNSW) Facebook’s implementation, FAISS, uses Hierarchical Navigable Small World Graphs (HNSW). HNSW typically performs well in accuracy and recall. It utilizes a hierarchical graph to create an average path towards certain areas. This graph has a hierarchical, transitive structure with a small average distance between nodes. HNSW traverses the graph and finds the closest adjacent node in each iteration and keeps track of the “best” neighbors seen so far. HNSW has a polylogarithmic time complexity (O(logN)). HNSW works by creating a hierarchical index to allow for faster nearest neighbor lookup. How embeddings are operationalized today Moving embeddings out of labs and into real-world systems, has surfaced real gaps in the current data infrastructure capabilities. For example, traditional databases and caches don’t support operations like nearest neighbor lookups. Specialized approximate nearest neighbor indices lack durable storage and other features required for full production use. MLOps systems lack dedicated methods to manage versioning, access, and training for embeddings. Modern ML systems need an embedding store: a database built from the ground up around the machine learning workflow with embeddings. Getting embeddings into production isn’t easy. The most common ways we’ve seen embeddings operationalized today are via Redis, Postgres, and S3 + Annoy/FAISS. We’ll cover how each of those implementations typically looks and the challenges with each. Redis Redis is a super-fast in-memory object-store. It makes storing and getting embeddings by key very fast. However, it doesn’t have any native embedding operations. It can’t do nearest-neighbor lookups and it can’t add or average vectors. All of this must be done on the model service. It also doesn’t fit cleanly in a typical MLOps workflow. It doesn’t support versioning, rolling back, or maintaining immutability. When training, the Redis client doesn’t automatically cache embeddings and this can cause unnecessary pressure and cost. It also doesn’t support partitioning embeddings and creating sub-indices. Postgres Postgres is far more versatile but far slower than Redis. Via plugins, it can be made to perform some of the vector operations manually. However, it does not have a performant nearest neighbor index. Also, having a Postgres lookup in the critical path of a model may add too much latency. Finally, Postgres doesn’t have a great way to cache embeddings on the client when training, causing very, very slow training. S3 files + Annoy/FAISS Annoy or FAISS are often used to operationalize embeddings when nearest-neighbor lookups are required. Both of these systems build indices of embeddings for approximate nearest neighbor operations, but they do not handle durable storage or other vector operations. Alone, they only solve one problem. Typically, companies will store their embeddings in S3 or a similar object storage service to fill in the rest of the gaps. They will load the embeddings and create the ANN index directly on the model when required. Updating embeddings becomes tough, and the system typically ends up with a lot of ad-hoc rules to manage versioning and rollbacks. FAISS and Annoy are great, but they need a full embedding store built around them. The Embedding Hub Machine learning systems that use an embedding need a type of data infrastructure that: - Stores their embeddings durably and with high availability - Allow for approximate nearest neighbor operations - Enable other operations like partitioning, sub-indices, and averaging - Manage versioning, access control, and rollbacks painlessly Embedding Hub sits atop your existing infrastructure. There are a few examples of companies like Pinterest who have built their own in-house, but the gap is still yet to be filled. Some startups have raised a few millions to build proprietary systems, and some database companies have tried to add nearest neighbors as a feature on top. There is a clear gap for an open-source embedding store, and we’ve decided to try to fill it. We’ve made our initial release of Embedding Hub on . Feel free to try it out, leave your comments and issues, and hand us a star! Github Conclusion Embeddings are a critical part of the data science toolkit, and continue to gain in popularity. Embeddings have allowed teams to break the state of the art in multiple disciplines from NLP to recommender systems. As they grow in popularity, a lot more focus will go into operationalizing them in real-world systems. We think that an embedding store will become a critical piece of machine learning infrastructure and that’s why we’ve open-sourced ours. GIve it a try, and if this sounds like an awesome project to work on full-time, we’re hiring! This article was first published on the featureform blog

Embeddings in Machine Learning: Everything You Need to Know

Hello and welcome! I’m a software developer. For the past 7 years, I have been working in a growing industry that evolves really fast and needs to adapt quickly to its environment. For this reason, we had to take shortcuts and deliver software evolutions rapidly, more often than not at the expense of its long-term quality. I’ve grown more and more mature throughout the years of experience I went through in this professional world. When I started working, the company was a startup and all I wanted to do was to write some nice code. Now — a few years later in this company, which grew really well — I still want to write some nice code. The key difference is that I have now a better understanding of I’m doing it, and my work affects the present and future of the company. why how To understand that, we need to take a step back and look at the bigger picture: . It does so by bringing a product (or a set of products) that is useful in some way to a part of the population, in exchange for something (generally money). the goal of a company is to create some form of wealth Companies in the tech industry face problems everyday, be it e.g. features or improvements requested by users and clients, or bugs that need to be fixed. . The most important role of a developer is to solve these problems Developers don’t write code just for the sake of writing code. Sure, it’s fun, but the main objective — the “why” —  is to find solutions that . The better the solution to a given problem, the more value is brought by the developer to the company. The “best” solution is not always the fastest one to implement though, but we’ll talk about this later in this article. bring business value Developers should aim at bringing as much value as they can, by solving problems solve problems. A good developer is one that brings value, a better developer is one that helps others grow by bringing more value as well. and helping others How can one solve the problems of an organisation? Well, there are pretty much infinite ways of doing that — the “how” — ranging from “quick and dirty” to “long and perfect” solutions. Both of these extremes are bad, and I’ll try to explain why in the following sections. Additionally, I’ll introduce another way that I think is a good tradeoff. The “quick and dirty” way Solving problems this way brings value quicker to the market. I’m not saying it’s the “best value” we can get, but it’s something. “Compromise” by monkeyuser.com . If you ask business stakeholders, they will tell you that the best developer is the one that solves their problems quickly. If you ask technical stakeholders though, they will probably tell you the opposite, since “quick” is often paired with “dirty”, and this “dirt” is cleaned up only by tech people, they are granted the time to clean that mess up, which is not that common. if Therefore, in theory, the lower the (or TTM), the better, because a solution to the problem is found in a small amount of time. In practice, sacrifices are made to buy some time, and the increasing cost of these sacrifices can jeopardise the entire velocity of the team, the more time passes. These “sacrifices” have actually a name that you might have already heard of: technical debt. time to market There are hundreds of definitions regarding technical debt. For me, it’s the fact that we . By saving time now, we lose time later. The more technical debt is piled up, the longer it takes to solve problems. borrow time from the future to solve problems faster now Introducing “quick and dirty” solutions has some consequences that are best to avoid: it makes the code more , thus understanding, bringing changes, and debugging it gets harder and harder. Misunderstanding what a code base does is the perfect environment to introduce new bugs, adding frustration, and losing productivity. complex At some point, it takes so much time to solve a new problem that it’s better to either rewrite entirely the software, or engage in a very long refactoring phase of the existing project. In any case, this undoubtedly  slows down the company. Obviously, if a critical bug is found and needs a quick fix, then “quick and dirty” is the way to go. Nevertheless, it’s very important to come back on that “hotfix” (after its release to production) to make it a “fix”, by refactoring what is needed in order to pay off the technical debt introduced in the first place. More on that later. The “long and perfect” way The more time we take to design a solution, the better it should be, right? Well I don’t really agree on this one. Sure, it’s really tempting to spend weeks thinking about the perfect architecture for a project or a feature, but in the end, , and if it is, it never stays “perfect” for long. In the meantime, the business loses patience and trust. perfection can never be achieved Furthermore, what is considered the “perfect” implementation to someone could be considered an “awful” one to someone else. Writing code, i.e. finding a solution to a problem, is a process. Even a“perfect” piece of code could be considered “awful” by the authors themselves, given enough time. subjective I remember writing a module in TypeScript using classes and a 1-level hierarchy between these classes. Back then, I considered that implementation to be pretty great (I’m not saying perfect, but I thought it was really nice). Six months and an intensive learning about functional programming later, going back to that module didn’t feel so good. I knew I was the one that wrote it, but I wasn’t satisfied anymore. I wanted to refactor it with what I considered to be “perfect” at that time, but what value would have I brought? There was no feature planned for this part of the code base, and it worked fine without any bugs. It just wasn’t anymore. beautiful What is “beautiful code” you might ask? There are as many definitions as there are developers. I can give you my current definition, knowing that it will probably change in a few years anyway: First and foremost, it solves the problems It’s ( ) DRY Don’t Repeat Yourself It’s SOLID It’s consistent (it follows some conventions and guidelines) It fits well with the existing code (no part that feels weird) It’s easily readable and understandable by peers It’s self-documented (meaningful naming, static types, purity) It’s easily testable with unit tests (purity) It mutate state only when necessary (immutability all the way, except for cache, centralised state management…) I consider all these principles to be great, leading to a clean, beautiful code base. However, these principles are not universal. As I said, if you ask other developers what is “beautiful code” to them, they will answer differently. Furthermore, I takes some time and practice to master all these principles, and it’s quite hard to apply them in an existing code base that is not built on top of these foundations. Finally, some softwares may have a strong focus on performance, which is and writing beautiful code. not always suitable for applying best practices Anyway, there’s no point in spending weeks to write the perfect solution to a problem. As soon as a is needed in that part of the code — which happens often since a product changes over time — the “perfect” solution falls apart. As soon as developers — which happens quite frequently — the “perfect” solution falls apart. new requirement learn new things So what should we do then? Our role as developers is to find the best middle-ground between these 2 ends of the spectrum, in order to . I like to call this , or “being efficient at solving problems”. bring value as fast as possible, without increasing the technical debt pragmatism The “pragmatic” way What I’m introducing here is not a universal way of solving problems. This is what I consider to be ideal based on my professional experience over the past years. It doesn’t mean I have always been using this method. I wish I did, but sometimes there are just too many problems to solve in a short period of time, and sacrifices are made because the cost (technical debt) is considered lower than the revenue (value added to the market). I’m pretty sure we’ve all been in this situation! Whenever we need to solve a new problem, we should go through the following steps: what could be a solution to the problem, but don’t implement it (yet). Imagine great Find and implement a solution that requires as fewer changes as possible, while following the guidelines of the project, e.g. don’t mutate object properties, use module X to make HTTP requests, etc. Write high-level tests to make sure this solution answers the initial problem. Furthermore, you can create some dashboards for monitoring the production state, to witness the impact of your solution. Ask for a review from your peers, then ship it to production as soon as you can. This will bring value to the market. immediately , refactor the part of the code that includes the solution from step 2 in order to implement the initial solution you thought about. High-level tests and monitoring should ensure that your refactoring doesn’t break anything. Right after great Imagine the great solution First of all, what is a solution? It’s a solution where the , by using some properties of beautiful code. great business problem is well understood and modelled with the correct abstractions Writing beautiful code leads to writing perfect solutions. But here, we’re not looking for writing the perfect solution, for all the reasons I mentioned earlier. We want to get closer to perfection without achieving it, because it would require too much resources (time and energy). According to the , we can say that 80% of the effort is spent in the last 20% of the solution implementation. For example, if a solution takes 20 days to be implemented, Pareto tells us that 80% of the solution is done in the first 4 days, and 20% are done in remaining 16 days. This may not apply every time of course, but you get the rough idea. Pareto principle As a rule of thumb, I’d say . This means we can’t have all the sweetness available in the “beautiful code” description. We have to pick some elements by finding the best . a great solution would account for 80% of the perfect solution tradeoff How can we do that? In my opinion, a good practice would be to assign a note for each principle of “beautiful code” on the code parts that need to change, in order to solve the problem. Then, one can make decisions based on these notes. For example: is the “easily testable” note close to 0? Then your solution should emphasise the effort on making the new code more testable. Then another iteration could be planned to increase the note of another principle, and so on. great While thinking about the solution, we should look for the answers to the following questions: great How much time will it take to implement this solution? It’s very important to be . In the majority of cases, several weeks is not reasonable: by working on this solution, you won’t be able to bring immediate value for several weeks. “Freezing” the value you could bring for that long is hardly acceptable for the business stakeholders. However, keep in mind that freezing your value now — by reducing and/or preventing the technical debt — will allow the future solutions to be implemented quicker, i.e. faster time to market. As always in our job, : how much (reasonable) time should we spend now to keep our velocity in the long run? reasonable it’s a matter of compromise Can the solution be split into smaller parts, in order to implement it ? If the solution is too big and requires too much work, it’s best to break it down in smaller parts, instead of completely ditching it because it doesn’t fit the “reasonable” time frame. Furthermore, it should make the peer reviews easier. iteratively Does the agree with this solution? (do I want to use a new shiny library I learned recently? Does my team know this library? Does it need some kind of training if we decide to use that library, or that coding paradigm? Does it know something that I don’t which could break the project with that implementation?) team How much does the existing code base need to be for this solution? (is it a standalone, isolated module/service? Is a “core” part of the code base affected by it? Do we need a whole new state management system to make it work?) adapted Can we easily write to avoid further regressions and make sure this solution is the answer to the problem we’re having? tests Does it need some form of , or is it self-documented (e.g. with features from statically typed languages)? documentation How are changes made in this part of the code? If it’s rarely changed then maybe it’s not worth spending too much time on it. If it changes a lot, then adapting the code could be interesting to improve its understanding and making changes faster in the future. often This is not an exhaustive list, but it provides some direction. Notice that at this point, we are not implementing the solution yet. We’re only thinking about it, which shouldn’t take time. It helps us understand the existing code, and how it should be adapted so the new code fits well and doesn’t feel like an “alien” in the middle of the rest. great too much Implement an acceptable solution “Tech Debt” by monkeyuser.com . Once a solution has been found, it is time to find a “less great” solution, while still avoiding the “quick and dirty” danger zone. Here, we are going to make as fewer changes as possible, while keeping some consistency with the existing code, and making sure that the new code is and works well. This part really depends on the project and its guidelines. Generally, this is where we add a new “if” branch, a new parameter to some existing function, a new method to an existing class, etc. great tested If the code was initially made to evolve in that direction, e.g. extending it by adding the support of a new data type, then chances are we are not going to need that solution after all, because the new code already fits well in the existing code. However from my experience, this is not often the case, because we are more inclined to solve problems in the “quick and dirty” way, which doesn’t make the code easily extendable. great About the tests, I would advice writing high-level tests such as integration, functional or end-to-end tests. They are slower than unit tests, but the solution we found earlier will more likely break and/or remove APIs/interfaces/modules from the existing code, which will break some unit tests as well. Here, it’s best not to invest in unit tests for the “acceptable solution”, because we know they will disappear anyway with the upcoming refactoring from a better solution. great Once we are done implementing this solution and the tests are written and passing, our peers should review the changes we made in the code, and once accepted, the changes should be shipped to the production state. We can confirm that everything works fine if we previously set up some monitoring dashboards, which I highly recommend. Implement the great solution Usually at this point, we found a solution to the problem, and another one waves at us, waiting for us to pick it up and find a fitting solution for it. However, in this method, there’s a step before going after the next problem: implementing the solution we thought about at the beginning. The only objective of this step is to introduced in the solution we implemented just before. It can also be used to remove previous debt introduced by other solutions implemented in this area of the code base. It really depends on how much time we want to spend on it. great pay off the technical debt reasonable This step should take more time than the previous one because we need to adapt the code we just changed in order to fit the new, greater solution. The high-level tests we wrote should help us avoid unwanted regressions. We should make sure to write unit tests for the new solution, since they are fast and very flexible regarding the combination of inputs and data involved. If we got favorable answers to all the questions mentioned earlier, then we shouldn’t have any issue with the stakeholders regarding this step. Keep in mind that this solution could become a one in the future, depending on the requirements (i.e. new problems) that could emerge, as well as the learnings from the author and/or team along the way. When this happens, remember to stick with this method, and you should be able to bring value fast, while keeping the technical debt at a low level. great poor Summary This chart is not based on real data, it’s just a way to represent the problem-solving methods we’ve been talking about in this article. With the “quick and dirty” method (the blue ever-growing line), we bring value fast at the beginning, but the more time passes, the longer it takes to solve problems. It’s pretty easy to convince people to use it because business will be happy for a while, until the technical debt — which is hard to measure — becomes overwhelming. With the “long and perfect” method (the red steady line), we bring value at a regular pace, but not really fast. It’s hard to convince people because business wants the solution to its problem as soon as possible (and who can blame it), it doesn’t care how “beautiful” the solution is. Furthermore, the definition of “beautiful” can change over time. With the “pragmatic” method I introduced (the orange oscillating line), we bring value rather fast, and . It should be easy to convince people regarding the first part. The second one (i.e. refactoring) is a bit challenging because it’s difficult to measure the consequences of technical debt, until it’s too late. It is our duty as software developers to explain why this debt shouldn’t be ignored, and act on it. we make sure we keep this pace for the future Thank you for reading! I hope I convinced you to test this “pragmatic” way for solving the problems of your organisation. Sometimes the lack of resources in a team and the volume of problems to solve is too important to apply this method. In such cases, you should always remember that you are borrowing time from the future to solve problems faster now. Reducing the technical debt by refactoring early means investing time early to avoid spending too much time later. Special thanks to Tristan Sallé and Jérémy Grosjean for their reviews and suggestions.

Speed Up Your Updates Delivery: This Method Really Works

Let’s attempt to coin a term here: "Static Form Provider." You bring your HTML <form>, but don’t worry about the back-end processing that makes it work. There are a lot of these services out there! Static Form Providers do all tasks like validating, storing, sending notifications, and integrating with other APIs. It’s lovely when you can delegate big responsibilities like this. The cost? Typically a monthly or annual subscription, except for a few providers and limited plans. The cost is usually less than fancier " " that help you build the form itself process it. form builders and In this article, we are going to review some of the most popular : static form providers Kwes Basin FieldGoal Formcarry FormKeep Formspree FormSubmit formX Getform Netlify Forms Before moving forward, just a note that the information for these comparisons came from visiting the site for each product and learning about the included features. Once I got all the information, I sent an email to each provider to confirm the list of features. Some of them confirmed, some didn't. Special thanks to Kwes, FormKeep, Formspree, FormSubmit, formX, and Netlify Forms teams for confirming. Form building components and validation Components for building a form are HTML input elements, like text inputs, textareas, checkboxes, and radio buttons. When using a static form, most providers require adding custom HTML attributes. By providing the custom URL in the form action attribute, the form gets submitted on the provider’s side where it gets stored. If you are looking for a form builder, FormKeep has a form designer feature. That means you embed custom HTML and JavaScript files in the page, and you get a styled form. Otherwise, you have to style the form by yourself. If you need custom components, like a date picker or card inputs, Kwes and FormKeep are the only providers with this feature. If you want to validate input fields in the browser, you might use third-party libraries or writing your code which means adding additional JavaScript code to the site. Kwes is the only provider that supports front-end validation based on the rules you set in each input component. To enable this feature, you should include additional JavaScript file, which you might do nevertheless. Other static form providers rely only on HTML5 validation. Kwes is the only provider with back-end validation, too. The rules you set in the front end are passed to the back end side. In case when front-end validation fails, you are safe, the backend validation would work. Other providers don't have this feature; they rely only on spam protection. Spam protection Kwes advertises a 99.6% spam block success rate with no setup required. Once your form is ready for submissions, you might find it hard to deal with spam. That’s why spam protection is essential, especially if you want to keep your sanity and serenity. All providers provide spam protection in this way or another. Some rely on or , some on Honeypot techniques, and some use artificial intelligence to get the job done. It is worth noting that adding an additional step to your form, like adding reCAPTCHA might affect the conversion rates on form submissions. Google reCAPTCHA Akismet Email notifications Email confirmations are essential if you want to provide a fast response to your users. With a contact form, for example, you want to get an email for every new submission. That way, you're able to respond to the submission quickly and efficiently. All providers, except FieldGoal, have confirmation, notification, and email routing logic features. You could set up an email form element which would be used to send an email automatically to the user with confirmation about the submission. Some providers have other sorts of notifications besides email, like push notifications or Slack messages, which might be handy. White labeling When communicating via email notifications with your clients, you might want to use your brand and style. It creates better awareness and that way you familiarize your clients with your product. All providers offer this feature, with the exception of FieldGoal, which I was unable to confirm (although it might be under paid plans). Custom redirects Once you have captured a response from your user, you may want to let the user continue using your site. Also, you might want to communicate to the user that the submission was received. This feature is called "custom redirect," and every provider has this feature (with another exception for FieldGoal because I was unable to confirm). Note that this feature might not be available in a free plan and require a paid or upgraded account. Upload storage Not every static form provider provides file storage. For example, formX doesn't provide it at all. In most cases, this feature is available under paid plans. You might want to invest additional time to find out which provider offers the best service for you. Be sure to look specifically at single file size form submission size limitations. and Data export Data export is important feature if you want to use it for analysis or for import to third-party software. Most providers offer CSV and JSON exports, which are the most commonly used ones. API access If you want to control your data submissions by building a custom application or script, you might benefit from having API access. Most providers have this feature, except Getform. formX offers it, but only on demand. Webhooks/Zapier When building a custom application or a script is out of budget, you might want to use webhooks to integrate data submissions with third-party software. Zapier is one of the most commonly used services for this, and only FormSubmit doesn't support it (though it does support webhooks). Analytics Analytics for static forms is a neat feature that could offer beneficial insight into how your form is performing. It may help you understand how your users interact with it, and you may spot ways to improve the form submission experience as a result. This feature is the least supported of all other features. Only Basin, FormKeep, and formX provide it. Plan comparison Static form providers have different plans, from entirely free plans and trials, to enterprise plans for every business need. Depending on a plan, you might have different features enabled. For example, FormSubmit is the only provider that offers all of its features for free, though it doesn't support every feature we've covered here. You will want to invest some time to learn about which features are most important for you and your product or business. Then go ahead and decide on which provider is most suitable for your needs. Wrapping up Having a form of any kind is a must-have for a large number of sites. When you use a static site generator, you might discover that static form providers make adding forms to a site almost trivial. By following a few rules for enabling static forms, you could benefit from essential features like spam protection and email notifications. I have been using Kwes for a while now and I can honestly tell you it is a great product that fulfills all of my needs. It has smart spam protection, an easy-to-use dashboard, and impressive validation, both on the front end and back end. Before choosing your static form providers, be sure to put down all requirements to the paper, and then find your perfect provider. This article was originally published on CSS-Tricks .

Compare Static Form Providers Side-by-Side

Knowing how to find bottlenecks in your application is a valuable skill when trying to scale to thousands or even millions of users. Some useful techniques for achieving this goal are and profiling. load testing In this article, we will learn how to use JMeter to load test an e-commerce application and MongoDB Profiler to find slow queries. Additionally, we will apply the Extended Reference Pattern to optimize it. These techniques have been proven effective in one of the companies I worked for, reducing the latency of endpoints by 90% and decreasing MongoDB CPU usage from 20% to 3%. The results were instrumental in enabling an e-commerce platform to handle thousands of users during a Black Friday-like event. The e-commerce application MongoStore is a simple e-commerce platform that operates in two regions of the world. It currently offers approximately 100,000 products in each region and has plans to expand to other locations. The application is developed using , TypeScript, Serverless Framework, AWS, and . Node.js MongoDB Let’s examine its current implementation and architecture: You can view the complete code along with deployment instructions. here on GitHub The Product schema contains the , , and fields. sku name description const ProductSchema = new Schema<IProduct>(
  {
    sku: { type: String, required: true },
    name: { type: String, required: true },
    description: { type: String, required: true },
  },
  { timestamps: true },
); Each region can have multiple products, and the prices are specified per region. In this case, the schema references the and fields of the product. sku price const RegionSchema = new Schema<IRegion>(
  {
    name: { type: String, required: true, index: true, unique: true },
    products: [
      {
        _id: { id: false },
        sku: { type: String, required: true },
        price: { type: Number, required: true },
      },
    ],
  },
  { timestamps: true },
); For its architecture, the application uses a MongoDB Cluster hosted on 3 EC2 instances. The servers employ AWS Lambda and are behind an API Gateway. With this configuration, MongoStore can scale the backend servers using AWS Lambda. However, the database can still present scalability challenges since it does not scale automatically. Note: Auto-scaling has intentionally not been set up to identify database bottlenecks more effectively. Profiling with MongoDB Users have reported that the website is down, and CloudWatch data confirms that the database CPU usage has reached 100%. Consequently, it is likely that a database query is causing the issue. To investigate further, you can utilize the . The profiler collects data from MongoDB queries that exceed a specific threshold and stores it in the collection. MongoDB Database Profiler system.profile Connect to the primary MongoDB instance and use to execute the following command, which sets the data collection threshold to 100 ms: Mongo Shell db.setProfilingLevel(1, 100)
{
        "was" : 0,
        "slowms" : 100,
        "sampleRate" : 1,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1683765280, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1683765280, 1)
} Now, retrieve the queries that have been executed and took longer than the threshold. db.system.profile.find().pretty()
{
        "op" : "command",
        "ns" : "database.regions",
        "command" : {
                "aggregate" : "regions",
                "pipeline" : [
                        {
                                "$match" : {
                                        "_id" : ObjectId("645c38d52f5c6ee098f9a392")
                                }
                        },
                        {
                                "$unwind" : "$products"
                        },
                        {
                                "$lookup" : {
                                        "from" : "products",
                                        "localField" : "products.sku",
                                        "foreignField" : "sku",
                                        "as" : "products.product"
                                }
                        },
                        {
                                "$unwind" : "$products.product"
                        },
                        {
                                "$limit" : 100
                        },
                        {
                                "$project" : {
                                        "_id" : "$products.product._id",
                                        "sku" : "$products.product.sku",
                                        "name" : "$products.product.name",
                                        "description" : "$products.product.description",
                                        "price" : "$products.price"
                                }
                        }
                ],
                "cursor" : {

                },
                "lsid" : {
                        "id" : UUID("d3e2a4d9-4970-4d80-917f-384be47acbd8")
                },
                "$clusterTime" : {
                        "clusterTime" : Timestamp(1684015798, 1),
                        "signature" : {
                                "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                                "keyId" : NumberLong(0)
                        }
                },
                "$db" : "database"
        },
        "keysExamined" : 1,
        "docsExamined" : 20000001,
        "cursorExhausted" : true,
        "numYield" : 20000,
        "nreturned" : 100,
        "queryHash" : "5B7E4F14",
        "planCacheKey" : "8928FB1D",
        "locks" : {
                "FeatureCompatibilityVersion" : {
                        "acquireCount" : {
                                "r" : NumberLong(20203)
                        }
                },
                "ReplicationStateTransition" : {
                        "acquireCount" : {
                                "w" : NumberLong(1)
                        }
                },
                "Global" : {
                        "acquireCount" : {
                                "r" : NumberLong(20203)
                        }
                },
                "Mutex" : {
                        "acquireCount" : {
                                "r" : NumberLong(202)
                        }
                }
        },
        "flowControl" : {

        },
        "readConcern" : {
                "level" : "local",
                "provenance" : "implicitDefault"
        },
        "writeConcern" : {
                "w" : "majority",
                "wtimeout" : 0,
                "provenance" : "implicitDefault"
        },
        "storage" : {

        },
        "responseLength" : 21893,
        "protocol" : "op_msg",
        "millis" : 12095,
        "planSummary" : "IDHACK"
} The whole operation took 12 seconds and by the variables and we can see that it only checked one index key and went through millions of documents. keysExamined docsExamined After some investigation, we noticed that we forgot to add the index to our collection. Let’s do it. sku products const ProductSchema = new Schema<IProduct>(
  {
    sku: { type: String, required: true, index: true, unique: true },
    name: { type: String, required: true },
    description: { type: String, required: true },
  },
  { timestamps: true },
); Now, let’s set the threshold to a lower time to get the results. But, first, clear the database by disabling the profiler and enabling again. db.system.profile db.setProfilingLevel(0) # disable profile
db.system.profile.drop() # drop profile database
db.setProfilingLevel(1, 10) # set threshold to 10ms db.system.profile.find().pretty()
{
        "op" : "command",
        "ns" : "database.regions",
        "command" : {
                "aggregate" : "regions",
                "pipeline" : [
                        {
                                "$match" : {
                                        "_id" : ObjectId("645c38d52f5c6ee098f9a392")
                                }
                        },
                        {
                                "$unwind" : "$products"
                        },
                        {
                                "$lookup" : {
                                        "from" : "products",
                                        "localField" : "products.sku",
                                        "foreignField" : "sku",
                                        "as" : "products.product"
                                }
                        },
                        {
                                "$unwind" : "$products.product"
                        },
                        {
                                "$limit" : 100
                        },
                        {
                                "$project" : {
                                        "_id" : "$products.product._id",
                                        "sku" : "$products.product.sku",
                                        "name" : "$products.product.name",
                                        "description" : "$products.product.description",
                                        "price" : "$products.price"
                                }
                        }
                ],
                "cursor" : {

                },
                "lsid" : {
                        "id" : UUID("cbbc0d5c-c35b-4428-8af6-d8b4a8ac0bd0")
                },
                "$clusterTime" : {
                        "clusterTime" : Timestamp(1684015558, 1),
                        "signature" : {
                                "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                                "keyId" : NumberLong(0)
                        }
                },
                "$db" : "database"
        },
        "keysExamined" : 101,
        "docsExamined" : 101,
        "cursorExhausted" : true,
        "numYield" : 0,
        "nreturned" : 100,
        "queryHash" : "5B7E4F14",
        "planCacheKey" : "2FBA11A3",
        "locks" : {
                "FeatureCompatibilityVersion" : {
                        "acquireCount" : {
                                "r" : NumberLong(203)
                        }
                },
                "ReplicationStateTransition" : {
                        "acquireCount" : {
                                "w" : NumberLong(1)
                        }
                },
                "Global" : {
                        "acquireCount" : {
                                "r" : NumberLong(203)
                        }
                },
                "Mutex" : {
                        "acquireCount" : {
                                "r" : NumberLong(202)
                        }
                }
        },
        "flowControl" : {

        },
        "readConcern" : {
                "level" : "local",
                "provenance" : "implicitDefault"
        },
        "writeConcern" : {
                "w" : "majority",
                "wtimeout" : 0,
                "provenance" : "implicitDefault"
        },
        "storage" : {

        },
        "responseLength" : 21893,
        "protocol" : "op_msg",
        "millis" : 29,
        "planSummary" : "IDHACK
} After adding the necessary index, the execution time was dramatically reduced to only 29 milliseconds. Additionally, the number of keys and documents examined also decreased significantly. That is a lot better! And finally, our users have stopped seeing crashes in the app. Load testing with jMeter Now that we fixed the bottleneck we had, we are confident that our system can handle thousands of users on a Black Friday, right? … WRONG! Improving just one endpoint and a database query does not guarantee we can handle the load we are expecting. One effective tool for load testing is , which can simulate multiple users accessing and making requests to the system simultaneously. Apache JMeter In this scenario, let’s assume we are expecting 10,000 users are expected, with each user making approximately 1 request every 10 seconds to the endpoint responsible for fetching products. This translates to a requirement of 1,000 requests per second. To validate the system’s ability to handle this expected throughput, follow these steps: Retrieve the ID of one of the regions using . cURL curl https://3mspdgu8cb.execute-api.us-east-1.amazonaws.com/staging/regions
{"regions":[{"_id":"645c38d52f5c6ee098f9a392","name":"Alaska"},{"_id":"645c39102f5c6ee098fcb0e9","name":"Rio"}]} Use the provided endpoint to fetch 100 products from the Alaska region. https://3mspdgu8cb.execute-api.us-east-1.amazonaws.com/staging/regions/645c38d52f5c6ee098f9a392/products\?limit\=100 Create a Test Plan in JMeter using the . Configure it to simulate 100 threads with a 30-second startup time, holding the load for 60 seconds. Ultimate Thread Group plugin Configure the HTTP Request sampler in JMeter to target the appropriate endpoint. Additionaly, add the View Results Tree and Aggregate Report. Run the load tests. Upon executing the load tests, it is observed that the system only reaches approximately 20 requests per second before encountering 502 (Bad Gateway) errors. What does this mean? If we SSH into the machine and run we can see that the service crashed, meaning it couldn’t handle the load. sudo service mongod status mongod [ec2-user@ip-10-0-0-100 ~]$ sudo service mongod status
Redirecting to /bin/systemctl status mongod.service
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2023-05-16 23:28:20 UTC; 474ms ago The question is, but our query is already using the correct indexes, it is not doing a Collection Scan, and it is executing in 29ms. Can we do better than this? Yes! The Extended Reference Pattern When working with NoSQL databases, it is essential to understand that JOIN operations tend to be slow. In the context of this application, the step performed a join, negatively impacting the performance. $lookup To mitigate this, it is valuable to be aware of the . This pattern involves copying only the necessary data into the collection, eliminating the need for joins and allowing querying a single collection for the required information. Extended Reference Pattern It is also important to know that our first implementation had one drawback: having an array of products does not scale, because there is a limit of 16MB per document. Imagine if we had millions of products in each region. That array would grow quickly. To address these concerns, the Extended Reference Pattern can be applied by embedding the regions and prices directly into the product collection. This design change involves introducing a new array, , which contains the of the region and the corresponding product’s price in that region. regions _id const ProductSchema = new Schema<IProduct>(
  {
    sku: { type: String, required: true, index: true, unique: true },
    name: { type: String, required: true },
    description: { type: String, required: true },
    regions: [
      {
        _id: { type: Schema.Types.ObjectId, required: true, index: true },
        price: { type: Number, required: true },
      },
    ],
  },
  { timestamps: true },
); The application’s endpoint has been modified to utilize a new aggregate query that only queries the collection, eliminating the need for additional lookups. products export async function getProductsOptimized(
  regionId: string,
  limit: string | number,
): Promise<IProduct[]> {
  const mongoRegionId = new mongoose.Types.ObjectId(regionId);
  return Product.aggregate([
    { $match: { regions: { $elemMatch: { _id: mongoRegionId } } } },
    { $limit: Number(limit) },
    {
      $unwind: '$regions',
    },
    {
      $match: { 'regions._id': mongoRegionId },
    },
    {
      $project: {
        _id: 1,
        sku: 1,
        name: 1,
        description: 1,
        price: '$regions.price',
      },
    },
  ]);
} Upon retesting the endpoint with the optimized query using the query string parameter, the system demonstrates a significant improvement. The aggregate report shows that the throughput has increased to 80 requests per second, representing a 400% improvement. optimized=true While this improvement is substantial, it is still short of the required throughput of 1,000 requests per second. To handle such loads, additional strategies can be employed, such as vertically scaling the EC2 instances by adding more CPU and RAM, distributing the load across multiple replicas, or even sharding the database. Conclusion By utilizing MongoDB’s profiling capabilities for query optimization, load testing with tools like Apache JMeter, and applying effective design patterns like the Extended Reference Pattern, it is possible to significantly improve the performance and scalability of the application. Additionally, implementing observability practices, including monitoring, logging, and alerting, allows for proactive identification and resolution of performance bottlenecks. Also published here.

Finding Bottlenecks in MongoDB: JMeter Profiling and Load Testing Strategies

In this tutorial, we will be using advanced JavaScript syntax in our app to maintain code as per latest standards and use Node.js require to import npm modules in our front-end project. Why do you need to upgrade? When I started writing JavaScript before 4 years, things were totally different. We use jQuery a lot, use script and link tags to include JS and CSS files in our application, and replace our file every time when we need to update it. Consider this line, <script type="text/javascript" src="jQuery.js"></script> You need to update this file if you have been using it locally. Else, you need to change versions manually when you need to. Consider this below, import jQuery from “jQuery”; Can you see the transformation? You don’t need to update manually at any cost. Just npm update will update all your modules in your application. No separate commands or works for anything. You can bring in amazing tools and utilities from the broader JavaScript ecosystem in a modular way. So, when things are changed, why don’t your front end application? Yes, it’s time to upgrade yourself. While in Rome, be a ____? Yes, you should when you use JavaScript in today’s world. Prerequisites (NPM comes with Node.js, you don’t need to install it separately) Node.js & NPM (Let’s install it later, don’t do it now) Browserify (Let’s install it later, don’t do it now) Watchify Let’s Code Install all your prerequisites. Create a directory, navigate and initialize your project using NPM by issuing the below command. $ mkdir es6-app$ cd es6-app$ npm init Fill in your details. Refer the below image for more info. Package.json configuration. You can leave fields empty and hit enter when you don’t have values to fill. This is just a reference and let’s concentrate on our application right now. Now in your root directory, you can see a file called created with the configurations provided by you and a create an file in your project or whatever you have given in the entry field during initialisation. es6-app package.json index.js Let’s install the npm module called to calculate the reading time of the content like the one you see on Medium and our other prerequisites. reading-time In your root directory, issue the following commands. $ npm i browserify watchify --save$ npm i reading-time --save Now you can see folder created in your root directory and the dependency entry in the package.json file. node_modules Install the dev dependencies required for your file conversions. $ npm i babel-core babel-preset-es2015 babelify --save-dev You have installed all the dependencies needed. Now, in your file, add the following code. index.js import readingTime from "reading-time"; window.calcRT = ev => {var stats = readingTime(ev.value).text; document.getElementById("readingTime").innerText = stats;}; Here, you have imported the reading-time npm module which you have installed and getting a sample text from the user to calculate the reading time. Note: You are using **import** keyword in your JavaScript file. Browsers are yet to upgrade themselves to use these syntaxes and when browsers do this in the near future, we don’t know what will be the ES version. We need to transpile file so that we can use the converted versions in our browser. index.js Add the following browserify presets (dependencies) in your package.json file. (Refer the when you need help) project "browserify": {"transform": [["babelify",{"presets": ["es2015"]}]]} along with the other configs and replace the scripts config with the below one. "scripts": {"build": "browserify index.js -o dist/bundle.js","watch": "watchify index.js -o dist/bundle.js -v"} It’s time to transpile. Issue the following command to transpile the file. $ npm run build Now your file is transpiled and created in a folder called dist. Now let’s add this converted file in our HTML file. Create a html file called index.html in your root project and add the following contents to it. <!DOCTYPE html><html> <head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>ES6 - App</title><link rel="stylesheet" href=" " /><script defer src=" "></script><script type="text/javascript" src="dist/bundle.js"></script></head> https://cdnjs.cloudflare.com/ajax/libs/bulma/0.7.1/css/bulma.min.css https://use.fontawesome.com/releases/v5.1.0/js/all.js <body><section class="section"><div class="container"><div class="level"><h1 class="title is-4 level-item has-text-centered">Welcome to ES6 - APP</h1></div><div class="level"><h1 class="subtitle is-5 level-item has-text-centered">Learn to use  <a href=" ">ES6 syntaxes</a>  in your front-end application</h1></div><br><div class="level"><p class="level-left level-item">Type your text to calculate the reading time</p></div><div class="level"><textarea class="textarea" onkeyup="calcRT(this)" placeholder="Add your text ..."></textarea></div><div class="level"><div class="level-left"><p>Reading time: </p><p class="has-text-info" id="readingTime">0 min read</p></div></div><div class="level"><div class="level-left"><p>Medium Tutorial: </p><a target="_blank" href=" ">Use ES6 JavaScript syntax (require, import etc.) in your Front end project.</a></div></div><div class="level"><div class="level-left"><p>Source code: </p><a target="_blank" href=" ">GitHub - es6-app</a></div></div></div></section></body> http://es6-features.org/ https://medium.com/full-stack-web-development-from-scratch-to/use-es6-javascript-syntax-require-import-etc-in-your-front-end-project-5eefcef745c2 https://github.com/balasubramanim/es6-app </html> That’s it. You have created your front end project using advanced JavaScript syntax. I hope this will definitely act as a reference for your upcoming build. Now, you need to automate your build process. Run the following command instead of to watch and update the file as soon as it gets updated. npm run build $ npm run watch You have automated your process successfully. If you have any queries, let me know in the comments section. Show your appreciation by holding the claps icon as much as you love. Final project can be found . here Play with the es6-app @ https://balsu.me/es6-app/ Thank you.

Use ES6 JavaScript syntax (require, import etc.) in your Front end project.

05: PushInterview The journey of how a Solo, Non-Technical Founder learned to code and built EDM Hunters In the 21st century that we live in, whenever we talk about a tech , the first thing that comes to mind is often “coding”. startup So is it even possible for a non-technical founder to build a successful tech startup? Don’t be disheartened if you are a non-technical founder. History says, Yes of course! Credits: iPrice (Founder of AirBnB), (Founder of ) and Sean Rad (Founder of ) all being non-technical founders built some of the most successful tech startups. Brian Chesky Ryan Hoover Product Hunt Tinder But how did they succeed? Read through this interview to know: “How a Solo, Non-Technical Founder, , learned to code and built a successful Tech Startup” Jaskaran This is the fifth interview of the series “ powered by . PushInterview : Interviews that helps you Pushstart ” Pushstart Check out our previous interview on “ ” if you happen to miss it. How I built one of the most active startup communities of India _My journey of building a startup community from zero to one_hackernoon.com How I built one of the most active startup community of India 🕴Who are you? I am the one wearing the black t-shirt Hi ! I’m , a 25-year-old enthusiast, and a non-techie turned coder from India. I graduated from Manipal Institute of Technology in the year 2014. Pushstarters Jaskaran Rana music I built , an electronic music discovery startup back in my final year at college all by myself. EDM Hunters Post college, I worked with and for around 3 years before finally quitting in February 2017 and working full time on . BuyingIQ CollegeDekho EDM Hunters Working with these startups immensely helped me hone my technical skills and gain critical insights into what goes behind building a tech startup from scratch. 🤔 What’s the deal with EDM Hunters ? was a passion project born out of my love for electronic music. EDM Hunters The current version of EDM Hunters was built for enthusiasts like me to get the daily fix of electronic music. is currently developed and run solely by me. EDM Hunters dance music EDM Hunters Being extremely passionate about , I started building back in college in the year 2014. Electronic Dance Music EDM Hunters It was started with the aim of helping both new and seasoned listeners of the genre discover the best and latest songs of electronic music. I believe that community wins over algorithms and experts when it comes to supporting new and undiscovered talent. That’s why on it is the community which decides the top songs. EDM Hunters You can browse through various music charts to discover top songs by artists, genre, DJs, month and year, all compiled based on likes, play count, and downloads. 💯 What motivated you to start ? EDM Hunters My motivation for building was to consolidate all my learnings of “How to Code” into a real-life project, while at the same time build a product that I would personally love to use. EDM Hunters Back in college, I started listening to electronic music and got instantly hooked. had just started gaining popularity and entering the mainstream music scene. As an ardent electronic music fan, I was disgruntled by the lack of mediums on which one could discover electronic music of his/her choice. EDM There wasn’t a one-stop solution to help me discover this kind of music at one place. Thus, was born to solve this problem. EDM Hunters 🛠 What all went into building the initial product? To give you a bit of a background, I wasn’t a bright student academically in my college. Though my major was Computer Science, I didn’t know how to code until my final year. I had a 5.0 GPA and was barely able to pass my exams. The main reason for this was the way the course was taught. The education system here in India focuses more on Rote learning than Practical learning. After a point, It becomes difficult to pay attention. To learn to code, I decided to give online courses a shot. helped me immensely at the initial stage. Codecademy I learned , , , and in a span of 6 months. Following that, I decided to learn , a web framework based on . Python JavaScript jQuery HTML/CSS Django Python I started building in my final year at college. I used to work on it after my classes and on the weekends. EDM Hunters I was done with the basic prototype in 3 months and launched the initial version in January 2014. This is how EDM Hunters looked at the initial stage: The initial version of EDM Hunters Post-launch I kept on iterating with the design and features of the website. It didn’t take a lot of money to launch the site. Primarily, the only cost associated was with hosting it. Since I decided to use a , that turned out to be pretty cheap. VPS How do you source your songs? I get this question a lot: I use 3 different sources for the songs. My primary source is , which is a music store for electronic music. I use their to get all the songs of a specific artist. They also provide me with a 2-minute preview of each song to stream. Beatport API The other source is , where artists upload their own songs. I use their to stream the songs. I use the to stream the videos. SoundCloud API YouTube API Do remember that I learned to code because I was really passionate about solving the problem. 🤝 How did you get your initial users? The growth in traffic has been pretty phenomenal. To be honest, when I first launched , I didn’t expect it to grow as big as it is right now. EDM Hunters For the first 6 months, the traffic was pretty much non-existent with less than 1k visitors/month. However, after I got my first job, I started learning new things every day. Whenever I learned something new which was relevant and could be integrated into my own site, I did so the same night. I also started doing a lot of research on . SEO Counter to what many people believe, there isn’t a secret recipe for SEO. It is just about building a website with good user experience while making it easily indexable and searchable for the bots. Some SEO tricks that helped me: Try to get a backlink from other relevant and high-quality websites. Ensure a faster load time. Make sure that your website is mobile friendly and test this exhaustively. Keep content unique and relevant to your website. Give each page a relevant and unique title. Add in a meta description to each page describing the purpose. Mark up your site using , if applicable, to display rich snippets. schema.org Build a sitemap for your website and submit it using the . Google Webmaster Tools Use HTML correctly . ( tags, titles, and alt attributes on images ) Try to use relevant keywords in the URL. Growth in traffic from 2014 to 2016 Currently, the website has over 500k+ sessions every month, with 85% of the traffic being organic and it is increasing at the rate of 20% month-on-month. SEO has been the major driving force behind the growth of . For me, it’s been more about identifying the right keywords with sufficient traffic and building products around them. EDM Hunters If I had to attribute another reason for growth, it would be good media coverage. I tried to get the word out for by attempting to get it covered by various blogs. EDM Hunters I was lucky enough to get featured in some of the prominent blogs like . Recently, it also got featured in a promotional video for . YourStory , IndieHackers , Homegrown , and MyVenture Codecademy Growth has all been organic till now with SEO contributing to the majority of it. 💰 What is your business model? I didn’t start EDM Hunters with a “business model” in mind. I started it because of my passion and love for electronic music. Currently, the major source of income is Advertisement. I started off with one of the most famous Ad network. Google AdSense , After using it for some time, I realised that it didn’t have potential to offer substantial revenue, as the Ads were not really relevant to my users. I decided to integrate a few niche Ad networks that primarily focussed on music. I am currently using and on my website. These offer better value to the users and are now a major chunk of my Ad revenue. Wavo Boost Ad network Feature.fm The current average revenue from these Ad networks: Around Rs. 80k — Rs. 100k per month The revenue is directly proportional to the traffic. So an increase in traffic leads to an increase in revenue. I have also started promoting upcoming artists and labels using sponsored listings on my website. There are various plans available to choose from, ranging from $10 to $300. The money made from these ranges anywhere between: Rs. 15k to Rs. 50k per month I’ll be working towards building a more stable and concrete sponsored listing network in the time to come. 💪 Being a Solo Founder, how do you keep yourself motivated? Being a Solo Founder is definitely not the easiest thing out there. It’s easy to lose motivation and keep on procrastinating. But I guess what really motivated me was the cause. Since I was passionate about the product I was building, I wanted to build it in the best possible way. Taken in the right sense, Obsession can be a good motivator. There are a bunch of other things Solo Founders can try though: Try surrounding yourself with other entrepreneurs, programmers, or designers. You could also try working in a co-working space where you can find like-minded people, especially those with whom you can bounce off ideas. 🏁 What are your future goals and how do you plan to achieve them? I just launched a revamped UI for . Creating a mobile app for has been on my mind for quite some time now. I have been learning and would be using it to build the app for both and . EDM Hunters EDM Hunters React Native iOS Android I am planning to build a concrete sponsored listing network which would help budding artists gain visibility alongside the mainstream artists. Also, I will be focussing on building a strong social presence alongside steady organic growth. I have a bunch of other projects I’m planning to work on. I will be building a product which is aimed at music producers and creators. I also have plans to launch a project around live gigs in India. Some exciting times ahead I guess, keep a look out guys! 🌋 The biggest challenges you have faced till now? Like any other non-technical founder, the initial phase when I started to learn to code was the toughest for me. The decision to build an entire website on my own, from the frontend to the backend and being able to visualise how all the things connect and work together as a whole was quite challenging at the start. I had help from a couple of coder friends which was a big plus. For anything else, I always had the internet at my disposal. All you need from your end is total dedication. If you are ready to put in the efforts, they will definitely bear fruits in the long run. 🗒 What is your advice to starting out? Pushstarters Things might seem complex at the beginning, just don’t be intimidated. Everyone started off at the same spot. Start with a small project and try to get your fundamentals clear. Especially for non-technical founders: It gets intimidating at times but the key is not to give up. There is nothing more satisfying than watching something you wanted to build come to life. Try to find an internship/job at a startup to start with. Startups are always on the lookout for enthusiastic people with relevant skills. This is the best possible way to gain some on-field experience. Be prepared to fail. Not all Startup Ideas turn into the Next Big Thing. In order to succeed in a business, you must be prepared to fail. It is all about how passionate you are to solve the problem and not about knowing “How to Code”. ✉️ How can we get in touch with you? You can reach out to me on Linkedin , Facebook or info@edmhunters.com . Thanks a lot for reading! If you liked it, please support by clapping 👏🏻 and sharing the post. Feel free to leave a comment 💬 below. Want to become part of India’s most active startup community? Send a request on Pushstart and become part of our ever-growing family. Hey! Neeraj here, the founder of Pushstart , India’s most active entrepreneur community. I release interviews with successful entrepreneurs every week. Feel free to reach out to me on Facebook | Linkedin | neeraj.joshi@pushstart.in | Twitter .

How a Solo, Non-Technical Founder learned to code and built a successful  Tech Startup

article This originally appeared on dormoshe.io In the last five years, the web development zone overflowed with new frameworks, libraries, tools, languages and power factors. The web development field is gaining momentum and its evolution is being recorded in the history books. After the first half of the year, we can say that 2017 continues this path and is being written down as an important year. In this article, we will arrange our thoughts and focus on the main interesting, important and valuable hallway conversations of the web development in 2017. 1. Yarn — The Killer of NPM? The goal of Yarn is the same as of npm— to be a package manager for JavaScript. Yarn has been developed as a part of a collaboration between , Exponent, and Tilde. With Yarn, we still have access to the npm registry. In addition, we can install packages more quickly and manage dependencies consistently across machines or in secure offline environments. Yarn enables us to move faster and with confidence when it is being used to share code. Probably, the improvements of Yarn will incorporate into npm in the future, so both users will benefit from the improvements of others. Facebook, Google _Everything you need to know about Yarn package manager_shift.infinite.red NPM vs Yarn Cheat Sheet 2. Angular 1, 2, 4 Angular is a widely-used framework in the JavaScript area. Google released Angular version 2 in September 2016, as a rewrite of Angular 1. Google choose the method for Angular. Six months after version 2, version 4 of the framework has released. Version 3 was skipped as part of because of the version of the router. Angular 1 is called and Angular 2+ is called just . It seems that Angular has successfully of the community and that a massive number of developers made their path to the new Angular. semantic versioning technical skipping, AngularJS Angular passed the test _Angular version 4 is out!_hackernoon.com Top 8 Resources to Explore Angular 4 3. React Fiber — The Next Generation of React React Fiber is an ongoing of React’s core algorithm. It is the culmination of over a two years research that made by the React team. The main concern of the new React is . Scheduling is the option to pause, resume and cancel the update process in order to get high priority to other tasks — it’s also called . Animations are one of the tasks that will gain from this feature. It’s necessary because the human eyes recognize ‘lag’s in the animations if the update process is long. The plan is to put this rewrite into the hands of the developers once will be launched later this year. Unlike Angular, when Fiber will be released, the old React applications will as before. reimplementation scheduling Incremental Rendering React 16.0 continue to work — A Cartoon Introduction to Fiber @ React Conf 2017 Lin Clark 4. Vue.js — A Real Competitor of React and Angular A lot of words has written about the state of the front-end frameworks. You can see many new frameworks in the zone. Most of the developers use Angular or React. Vue.js is a that focuses on building user interfaces. It offers a lot of functionality for the . The popularity of Vue.js and the attention of the community set Vue.js as something different. It’s not just another framework. Vue.js holds an from Angular and React. We need to keep eyes and open ears to what will be happened with this cool framework. The first official Vue.js conference will take place in June 2017. progressive JavaScript framework view layer integration of advantages Talk about Modern Frontend with Vue.js @ Laracon EU 2016 Evan You 5. The power of Webpack Webpack is a JavaScript module bundler tool. Its main purpose is to bundle JavaScript files for usage in a browser, yet it is also capable of transforming, bundling, or packaging just about any resource or asset. The Webpack Plugin system is . In order to use a plugin, you just need to it and add it to the plugins array. Webpack makes our experience more easy and convenient. Webpack’s core idea of a dependency graph is what makes it so powerful and useful. Webpack 2 & are now available. Angular-CLI and many products already use Webpack. It makes its path to be a consensus. extremely powerful and customizable require() Webpack-CLI _This article was peer reviewed by Scott Molinari, Joan Yin and Joyce Echessa. Thanks to all of SitePoint's peer…_www.sitepoint.com A Beginner's Guide to Webpack 2 and Module Bundling - SitePoint 6. The Features of ES6 ES6 or ES2015 is the latest version of the ECMAScript standard. The previous update to the standard was in 2009. The Goals for ES6 provides better support for large applications, library creation and for using of ECMAScript as a compilation target for other languages. ES6 brings a and improvements like promises, lexical block scoping, iterators, generators, modules, class declarations and destructuring patterns. The implementation of these features in the major JavaScript engines is . massive number of features now underway _Support for constants (also known as "immutable variables"), i.e., variables which cannot be re-assigned new content.…_es6-features.org ECMAScript 6: New Features: Overview and Comparison 7. Prepack — A Run-Time Optimizer Prepack is a tool that optimizes JavaScript source code. Computations that can be done at , instead of run-time, get eliminated. Prepack is a partial evaluator for JavaScript. Prepack , resulting in JavaScript code that executes . It is under active development and still in very . Prepack can be a “ ”. It is backed by a stable company — . They want to integrate this tool in and they have been worked on that for a while. There is a lot to do. The vision of Facebook is to . compile-time rewrites a JavaScript bundle more efficiently early stage life changer Facebook React leveraging Prepack as a platform _In the last days, the social networks are stormy about Prepack. Probably you still haven’t heard about it. It’s…_hackernoon.com Facebook’s Prepack — The Next Killer In The JavaScript Zone 8. Firebase For the Serverless Applications Serverless architectures refer to applications that depend on third-party services. Firebase is a mobile and web application development . Firebase is built on Google infrastructure and scales with you automatically, so you don’t have to worry about meeting user demand. Firebase lets you just code and it and infrastructure. Cloud Functions, Real-time Database, Crash Reporting, Authentication, Cloud Storage, Hosting, Firebase Notifications and Android Test Lab are just a that go to makes your life happier. You also need to with Firebase like lack of local development option, limited JavaScript SDK and relations with NoSQL DBs. can be achieved with a lot of libraries like , and . significantly platform taking care of your backend few of the features pay attention to the difficulties Easy integration with Firebase angularfire2 reactfire , vuefire _Firebase is an astounding tool. Hundreds of millions of people use Firebase -powered apps. The company started as…_hackernoon.com How to Build a Product Loved by Millions and Get Acquired by Google: The Firebase Story 9. Redux — Unidirectional Data Flow The web pages and content being wealthy, massive, long, and detailed. The traditional static HTML pages gave away for the dashboards and stateful dynamic pages. Now, it’s not sufficient to save the data just in an object. There is context between the page parts and this is a . Redux is a predictable state container for JavaScript applications. Redux architecture revolves around a strict . It was and functional programming language Elm. Redux is one of the in front-end development these days. Redux got popular very quickly because of its simplicity. It’s familiar as the for React. It can be used by any JavaScript framework. Angular still has libraries that implemented these concepts. recipe for a mass unidirectional data flow inspired by Facebook’s Flux hottest libraries BFF of React, but it’s suitable not only Talk about Redux @ 2016 Dan Abramov react-europe 10. Microservices for Your Flexibility Microservices is an that structures an application as a collection of . Each of this services can focus on sub-domain of the application. This architecture enables the continuous delivery of large and complex applications. It also enables organizations to because each service is an and have a defined API for communication. Microservices improve fault isolation, eliminate long-term commitment to a single technology stack and make it easier for a new developer to understand the application functionality. The concepts usage expands and big companies like eBay, Amazon, and Netflix already use this architecture. architectural style loosely coupled services evolve its technology stack autonomy _How we’re rebuilding our infrastructure from scratch._blog.poki.com From Monolith to Microservices Conclusion The permutations in the web development are outward. We can’t get away from this changes. We need to embrace them and keep learning. Now you have a lot of contents for hallway conversations with your colleagues. You can dive deeply and explore these topics. The second half of 2017 will be wealthy and interesting — that’s for sure. So keep following… You can follow me on dormoshe.io or Twitter to read more about Angular, JavaScript and web development.

10 Hallway Conversations in 2017 about Web Development

It's already challenging enough to create a website that is blazing fast, adheres to all the latest best practices and offers optimal experience for various network conditions and device types. As the website and its codebase evolves over time, it can become even harder to maintain the desired user experience over time. To prevent UX deterioration, it’s often a good idea to place guardrails which ensure that crucial website elements, such as images, hold to a certain criteria. In this article, we are going to look at a robust mechanism which helps to steer your website in the right direction - an HTTP header called . Feature Policies Note: Feature Policies are soon to change name to " Permission Policies ". In the following, we'll continue to talk about Feature Policies as this is what is still implemented in the wild. Feature Policy Overview Feature policies comprise a set of rules you can selectively define in relation to various browser features and APIs to enforce the desired best practices for your website. For instance, you can ensure that under no conditions a user on a device with a small viewport will receive an oversized, unoptimized image - this effectively prevents layout thrashing and avoids wasting user’s bandwidth. Feature Policy might remind you of Content Security Policy - you control which origins can use which features, both in the top-level browsing context and in any embedded frames. It’s worth noting that preventing the use of the sub-optimal functionality requires explicitly specifying a policy that disables the features as everything is allowed by default to maintain backwards compatibility. Here’s an example of a policy which covers all bases when it comes to optimal image delivery: HTTP
Feature-Policy:
  legacy-image-formats ‘none’;
  oversized-images ‘none’;
  unoptimized-images ‘none’;
  unsized-media ; 'none' legacy-image-formats Prevents browser from displaying any images in legacy formats: oversized-images Disables the use of images whose sizes that are much bigger than the containers. unoptimized-images Detects and blocks any images with large byte-per-pixel ratio. unsized-media Allows developers to enforce that image/video elements have explicit dimensions. If any image matches any of the first three rules, a lightweight placeholder will be rendered in place of it. As to image and video elements without explicitly defined dimensions, the browser will set the default size of 300x150. You are also allowed to split one statement into several headers, like so: HTTP
Feature-Policy: legacy-image-formats ‘none’;
Feature-Policy: oversized-images ‘none’;
Feature-Policy: unoptimized-images ‘none’;
Feature-Policy: unsized-media ‘none’; For each rule, you can define a list of allowed origins. ` ` in the example above is a keyword denoting that the rule is enforced across all browsing contexts. Adding your website could be done via another keyword called ` `, while other origins must be spelled out: none self Feature-Policy: legacy-image-formats ‘self’ https: //third-party-service.com As you can imagine, we don’t want to block it from accessing the images, hence you need to include CDNs origin into the list of allowed origins: Let’s take a look at a practical example of picking and adding an origin to the allowlist. When serving your image assets through an I , you have a single high-resolution image included as a source in your markup. Image CDN picks it up and applies optimizations on-the-fly, returning the most suitable variation to the end user. To ensure this mechanism continues to function smoothly after introducing the aforementioned policies, you’d have to add your Image CDN’s origin to the allowlist: mage CDN Feature-Policy: legacy-image-formats https: Feature-Policy: oversized-images https: Feature-Policy: unoptimized-images https: Feature-Policy: unsized-media https: //img.cdn.imgeng.in; //img.cdn.imgeng.in; //img.cdn.imgeng.in; //img.cdn.imgeng.in; Conclusion Feature Policy is a powerful tool that allows you to control the use of a variety of browser features and APIs, both on your own website and within iframes. Policies fall into two broad categories: Enforcing best practices for good user experiences and avoid using outdated APIs Providing granular control over sensitive features with security and privacy implications Deciding which policies to apply and to what extent depends on your circumstances. For new content, you can start developing with all the features turned off. This approach ensures that none of the undesired functionality sneaks into your codebase. When applying a policy to the existing content, you need to test and verify that it continues to work as expected. This is especially important for embedded content: it’s a good idea to first observe effects on your own content before extending the rules to the third-party content that you do not control. As for image-related policies, using an Image CDN saves a considerable amount of time, freeing you from all the heavy-lifting associated with efficiently delivering optimized, responsive images which comply with your image policies. Everything is handled for you behind the scenes automatically, without cutting corners: choosing the most appropriate format, resizing and reducing file size by applying smart refinements. The only thing you have to do is to add Image CDN’s origin to the list of allowed hosts - who wouldn’t want to instantly take a leap above the competition in just a few minutes? Alos published at https://dev.to/jonarnes/comply-with-permission-policies-feature-policies-with-an-image-cdn-3m71

Stories, News & Mentions about FEATUREFORM

Featureform’s Stories on HackerNoon