We’re building Verse™: a visual programming tool made for developers by developers

Factorize

Efficient Neural Network Approaches for Conditional Optimal Transport: Numerical Experiments

In today's edition: What does research say about how far we are from AGI? News from the usual subjects: Anthropic, Cohere, OpenAI, etc. The freshest AI&ML research papers from the week of Mar 4 — Mar 10 Though the question, “ ” predates the term “artificial intelligence” itself, it saw a significant resurgence on Twitter last week, prompted by the Musk vs. OpenAI lawsuit (Musk accuses OpenAI of abandoning open-source principles and prioritizing profit over safety, hindering the safe development of AGI.) . Today, we will discuss “How Far Are We from Intelligent Visual Deductive Reasoning?” “How Far Are We From Automating Front-End Engineering?” and Stephen Wolfram’s article “Can AI Solve Science?” These papers offer fascinating explorations of the differences between human and artificial intelligence. How far are we from achieving human-level intelligence in machines (or AGI, or ASI)? But far more interesting were the papers and an article that came out last week tackling this question Intelligent Visual Deductive Reasoning In “ ?”, researchers from Apple explore Vision-Language Models (VLMs), like GPT-4V, in visual-based deductive reasoning, a complex yet less studied area, using Raven’s Progressive Matrices (RPMs)*. How Far Are We from Intelligent Visual Deductive Reasoning *Raven’s Progressive Matrices is a nonverbal intelligence test measuring abstract reasoning, using patterns to assess cognitive functioning without language. What caught my attention was the finding that . The paper notes, “ ” This inability to deal with abstract concepts marks a fundamental difference between computational processing and human cognitive abilities. AI systems like VLMs struggle with tasks requiring abstract pattern recognition and deduction VLMs struggle to solve these tasks mainly because they are unable to perceive and comprehend multiple, confounding abstract patterns in RPM examples. Being a sophisticated pattern recognizer doesn’t equate to sentience. The observation that “ ” highlights , suggesting a nuanced aspect of intelligence that current AI lacks. Another intriguing point was the models’ overconfidence. all the tested models never express any level of uncertainty the importance of doubt and uncertainty in human cognition Automating Front-End Engineers In “ ”, researchers from Stanford University, Georgia Tech, Microsoft, and Google DeepMind have developed a benchmark for Design2Code, aiming to evaluate how well multimodal LLMs convert visual designs into code. Here, the replacement of humans came closer. Despite some limitations, there were considerable advancements in using generative AI to convert designs into front-end code. It’s remarkable that “ .” This finding challenges traditional notions of artistic and creative value, Design2Code: How Far Are We From Automating Front-End Engineering? annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and in 64% of cases, GPT-4V generated webpages are considered better questioning whether creativity is uniquely human or can be algorithmically reproduced — or even surpassed. However, . VLMs struggle with “ ” posing questions about understanding and interpretation. significant limitations persist recalling visual elements from the input webpages and generating correct layout designs. So, the important question is actually not how far we are from AGI (whatever it is), but how we embrace human-AI collaboration most effectively. AI Solving Science In that sense, Stephen Wolfram’s blog post “ ” serves as an excellent example. In the very beginning, he plainly states that AI cannot solve all scientific questions. However, there is significant value in AI assisting scientific progress. He discusses how LLMs can serve as a new kind of linguistic interface to computational capabilities, providing high-level “autocomplete” for scientific work. As he usually does, he emphasizes the transformative potential of representing the world computationally and suggests that pockets of computational reducibility* can be found by AI as well. Can AI Solve Science? *A pocket of computational reducibility — a fascinating concept introduced by Wolfram — is a situation or problem within a complex system where, despite the system’s overall unpredictability, predictable patterns or simplified behaviors emerge, allowing for easier understanding or calculation. Wolfram argues that AI can significantly aid scientific discovery by providing new tools for analysis and exploration, but its ability to completely “solve” science is limited by fundamental principles such as computational irreducibility. The future of AI in science lies in its integration with human creativity and understanding, leveraging its strengths to uncover new knowledge within the constraints of what is computationally possible. We might be able to survive without front-end developers (no offense intended), but scientists remain indispensable! To summarize: https://x.com/pmddomingos/status/1766945083314827455?s=20&embedable=true News from The Usual Suspects © Cohere and its commitment to the research community https://x.com/cohere/status/1767275128813928611?s=20&embedable=true Hugging Face Starting an ambitious open robotics project and hiring , a former Tesla scientist, to expand beyond software into robotics, including work on humanoid robots like Tesla’s Optimus. Remi Cadene Offers an amazing tool: https://x.com/maximelabonne/status/1767124527551549860?s=20&embedable=true Russia’s talent is invisible According to “The Global AI Talent Tracker 2.0”, Russia has no chance in the AI race: this analysis Inflection enhances its Pi Inflection , enhancing its personal AI, Pi, with IQ capabilities alongside its empathetic EQ. They claim to compete with GPT-4, achieving significant efficiency and performance improvements, especially in STEM areas, with less computational resource usage. launches Inflection-2.5 Chips TSMC (based in Taiwan), the world’s largest contract chipmaker, is for an Arizona chip plant. This funding, part of the CHIPS and Science Act of 2022, aims to boost domestic semiconductor production. TSMC’s $40 billion investment in the plant marks one of the largest foreign investments in U.S. history. expected to receive over $5 billion in U.S. federal grants OpenAI: new members on the board Sam Altman is reinstated as a member of the OpenAI board. , with a rich history as the CEO of the Bill and Melinda Gates Foundation, brings extensive experience in healthcare and philanthropy. , a seasoned executive with roles at Sony Entertainment and as a lawyer, offers legal and entertainment industry insights. , leading Instacart and with a background at Meta Platforms, including as head of Facebook, contributes expertise in technology and e-commerce. This diverse trio enhances OpenAI’s governance with their wide-ranging expertise and perspectives. He is joined by three women: Sue Desmond-Hellmann Nicole Seligman Fidji Simo Elon’s Grok https://x.com/elonmusk/status/1767108624038449405?s=20&embedable=true Anthropic Offers a great collection of prompts in its Prompt library Enjoyed This Story? I write a weekly analysis of the AI world in the newsletter. We aim to equip you with comprehensive knowledge and historical insights, so you can make informed decisions about AI and ML. Turing Post 🎁 Bonus: The freshest research papers, categorized for your convenience Enhancements in Language Models and Multimodal Understanding : Showcases an LLM’s intrinsic ability to understand and generate music, expanding LLMs’ applications beyond text. ChatMusician Read the paper : Demonstrates advanced multimodal understanding by processing extensive contexts of text, video, and audio. Gemini 1.5 Read the paper : Enhances TTS synthesis with a factorized diffusion model for zero-shot natural speech generation. NaturalSpeech 3 Read the paper : Utilizes fine-tuned MLMs for superior image-text data filtering, improving dataset quality for MLM training. Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters Read the paper : Introduces the first LLM designed for the legal domain, demonstrating proficiency in legal text understanding and generation. SaulLM-7B Read the paper Novel Training and Evaluation Techniques : Proposes a method for scaling instruction tuning in mathematical reasoning, significantly enhancing LLMs’ problem-solving abilities. MathScale Read the paper : Reveals redundancy in LLM layers and proposes a pruning strategy, maintaining performance while reducing model size. ShortGPT Read the paper : Enhances memory efficiency in LLM training without sacrificing performance by applying gradient low-rank projection. GaLore Read the paper : Develops a method for collaborative decoding among multiple LLMs, improving performance across various tasks. Learning to Decode Collaboratively with Multiple Language Models Read the paper : Advocates for training RL value functions through classification instead of regression, boosting performance and scalability. Stop Regressing Read the paper Advances in Generative Models and Data Synthesis : Introduces a framework for generating synthetic multimodal datasets, overcoming limitations in data privacy and diversity. MAGID Read the paper : Trains a generative model to create interactive virtual worlds from text, images, or sketches, advancing interactive environment simulation. Read the paper Genie Scalability and Efficiency in AI Systems : Details a system for training LLMs on over 10,000 GPUs, tackling challenges in training efficiency and stability at large scales. MegaScale Read the paper : Improves State Space Models with dense hidden connections for efficient large language models, enhancing performance with minimal parameter increase. DenseMamba Read the paper Exploring New Frontiers in AI and Machine Learning : Illustrates that planning and prediction in time series can be simplified through learned contrastive representations. Inference via Interpolation Read the paper : Employs simulated trial and error for tool learning, significantly enhancing LLMs’ practical application capabilities. LLMs in the Imaginarium Read the paper : Aims to improve context length generalization in LLMs by refining interpolation techniques for out-of-distribution token positions. Resonance RoPE Read the paper : Enhances the localization of behaviors in LLMs to specific components, improving diagnostic capabilities and model understanding. AtP Read the paper : Investigates using world models for enhancing visual representation learning, beyond reinforcement learning applications. Learning and Leveraging World Models in Visual Representation Learning Read the paper Platforms and Tools for Model Evaluation and Interaction : Provides a platform for evaluating LLMs based on human preferences, offering insights into model rankings through pairwise comparisons. Chatbot Arena Read the paper : Explores enhancing LLMs’ reasoning capabilities using RLHF, comparing different algorithms and reward strategies. Teaching Large Language Models to Reason with Reinforcement Learning Read the paper Also published . here

How Far Are We From Human-level Intelligence in AI?

A program that maintain its correctness in a multi-threaded environment is said to be a program. Thread-Safe Fundamentally, thread safety revolves around proper management of the shared state. In the absence of such proper management, the program could potentially spiral into an invalid state when executed in a multi-threaded environment. Defining Application’s State An application’s state is formed by the collective states of the individual objects that constitute it. An object’s state encompasses the data stored in its static or instance fields, which can influence its externally visible behavior. Furthermore, an object’s state may encompass fields from other dependent objects, creating a complex web of interconnected states within the application. Whether an object’s state need to be managed for concurrent access depends on its usage in your program. If an object is only ever accessed from a single thread, state management might not be needed! If multiple threads access the same mutable state of an object without proper synchronization, your application is technically broken. There are three ways to fix it — Don’t share the object across threads Make the object’s state immutable Implement proper synchronization for accessing the object’s shared state Using good Object Oriented Principles can help us in designing thread-safe classes. Thoughtful implementation of encapsulation, immutability, and precise specification of invariants can significantly reduce the effort required to create thread-safe classes, ultimately streamlining the development process. An important thing to note is that a program consisting of only thread-safe classes may not be thread-safe. Additionally, a program may be thread-safe even when some of its classes are not thread-safe. It all depends on how your objects interact with each other in your program! Defining Thread Safety A thread-safe class behaves correctly even when accessed from multiple threads, regardless of the scheduling or interleaving of the execution. The calling code needs no synchronization on its part when working with a thread-safe class. Thread-safe classes encapsulate the necessary synchronisation, relieving the client from the burden of providing their own. The need for synchronization arises when multiple threads access the same mutable state. However, stateless objects, by virtue of having no state to share, inherently possess thread safety. As a result, . stateless objects are always thread-safe Example of a thread-unsafe class Consider the following class that is used to find a certain number of primes, Java public class PrimeFinder {
    // Variable to count the number of invocations
    private int hitCount = 0;
    public int getHitCount() {
        return hitCount;
    }
    public List<Integer> findPrimes(int count) {
        hitCount++;
        // Find 'count' number of primes
        final List<Integer> primes = new ArrayList<>();
        for(int i = 2; i < Integer.MAX_VALUE; i++) {
            if(isPrime(i)) {
                primes.add(i);
                if(primes.size() == count) {
                    break;
                }
            }
        }
        return primes;
    }

    private boolean isPrime(int num) {
        for(int i = 2; i < num; i++) {
            if(num % i == 0) {
                return false;
            }
        }
        return true;
    }
} Note that the instance variable hitCount constitutes the state of any PrimeFinder object. Unfortunately, is not thread-safe, even though it would work fine in a single-threaded environment. This class is vulnerable to , as the operation to increase the , although seemingly atomic, is actually broken down into three distinct steps. Here is an example of an unlucky sequence of events causing to enter an invalid state. PrimeFinder lost updates hitCount hitCount Unlucky interleaving of execution causes hitCount to be 10 when it should be 11 Computation of incorrect results in case of some unlucky timings in a concurrent program is called a . The particular type of race condition that plagues our is called or . To eliminate this race condition, we must increment the hitCount in a single atomic operation. race condition PrimeFinder check-then-act read-modify-write There are several ways to fix the class. One such way is to use a thread-safe class inside . Here is the updated code, PrimeFinder PrimeFinder public class PrimeFinder {
    private final AtomicInteger hitCount = new AtomicInteger(0);
    public int getHitCount() {
        return hitCount.get();
    }
    public List<Integer> findPrimes(int count) {
        hitCount.getAndIncrement();
        // Find 'count' number of primes
        final List<Integer> primes = new ArrayList<>();
        for(int i = 2; i < Integer.MAX_VALUE; i++) {
            if(isPrime(i)) {
                primes.add(i);
                if(primes.size() == count) {
                    break;
                }
            }
        }
        return primes;
    }

    private boolean isPrime(int num) {
        for(int i = 2; i < num; i++) {
            if(num % i == 0) {
                return false;
            }
        }
        return true;
    }
} Using the atomic variable, we can ensure atomic operations on numbers and object references. Since the state of consists only of , which is thread-safe now, the class is thread-safe now! PrimeFinder hitCount PrimeFinder When a single element of state is added to a stateless class, the resulting class will be thread-safe if the state is entirely managed by a thread-safe object. Is it possible to add more state variables to PrimeFinder in a similar manner and still maintain its thread safety? Let us see… Consider the following (updated) class, PrimeFinder public class PrimeFinder {
    private final AtomicInteger numCache = new AtomicInteger(1);
    private final AtomicReference<List<Integer>> primeFactorCache = new AtomicReference<>(new ArrayList<>());
    
    // ...

    public List<Integer> findPrimeFactors(int num) {
        if(numCache.get() == num) {
            return new ArrayList<>(primeFactorCache.get());
        }
        // Find prime factors of 'num'
        final List<Integer> primeFactors = new ArrayList<>();
        for(int i = 2; i < num; i++) {
            if(isPrime(i) && num % i == 0) {
                primeFactors.add(i);
            }
        }
        numCache.set(num);
        primeFactorCache.set(new ArrayList<>(primeFactors));
        return primeFactors;
    }

    // ...

} The method is supposed to return prime factors of a number. It also stores a cache of the last number factorized and its factors. We have used atomic classes to store both the and the . findPrimeFactors numCache primeFactorCache Still, our class is not thread-safe! Can you spot the race condition here? Hint: There is a check-then-act race condition here! One invariant of our class is that the list will hold all prime factors of . Since two different variables participate in a single variant, these variables are not independent! In such cases, all the dependent variables must be updated in the same atomic operation! PrimeFinder primeFactorCache numCache To maintain state consistency, ensure that dependent state variables are updated in a single atomic operation. Intrinsic Locking in Java Every Java object can implicitly act as a lock for the purpose of synchronization. These built-in locks are what we call or . intrinsic locks monitor locks There is a built-in mechanism in Java to provide synchronization called a synchronized block. A synchronized block is a set of statements guarded by an intrinsic lock (object reference). Here is its syntax — synchronized (lock) {
  // Statements working with shared mutable state
} The lock is automatically acquired by an executing thread when it enters a synchronized block and is released when the block is exited. Since these intrinsic locks can be held by at most one executing thread at once, they allow exclusive access to the guarded code. A synchronized method is a synchronized block with the entire method as body and object as its lock. this Let us fix our class using intrinsic locks. PrimeFinder public class PrimeFinder {
    private final AtomicInteger numCache = new AtomicInteger(1);
    private final AtomicReference<List<Integer>> primeFactorCache = new AtomicReference<>(new ArrayList<>());
    
    // ...

    public List<Integer> findPrimeFactors(int num) {
        synchronized (this) {
            if (numCache.get() == num) {
                return new ArrayList<>(primeFactorCache.get());
            }
        }
        // Find prime factors of 'num'
        final List<Integer> primeFactors = new ArrayList<>();
        for(int i = 2; i < num; i++) {
            if(isPrime(i) && num % i == 0) {
                primeFactors.add(i);
            }
        }
        synchronized (this) {
            numCache.set(num);
            primeFactorCache.set(new ArrayList<>(primeFactors));
        }
        return primeFactors;
    }

    // ...

} With this, our class is thread-safe again! PrimeFinder With this, we reach the end of this blog. We discussed the fundamentals of thread safety in terms of program state and explored some basic synchronization techniques in Java. If you enjoy reading this blog, consider throwing in a like and subscribe to not miss future articles on Java concurrency! Also published . here

Safeguarding Concurrent Programs in Java: Thread Safety from an Object Oriented Perspective

Introduction Would you like to see Mona Lisa smile like a witch? Or would you like the girl with the pearl earring to wink and smile? Google has just dropped a video generation model named Lumiere [1] that's capable of doing all of it for you. Though it's primarily a text-to-video model, it's able to do much more than that. Given a reference image with a prompt, it can stylize your videos by copying the style of the reference image to your videos. You can even edit your videos with just a single prompt. The model is Lumiaire. It's even able to animate objects within a user-specified region in an image, a technique called Cinemagraphs. When it comes to inpainting, Lumiere is even able to reason about a completely missing object such as a cake in this example. It all boils down to a diffusion model with a novel space-time U-Net architecture [3]. It's a customization of the U-Net architecture to solve the problem of temporal consistency which is quite prevalent in video generation models. Visual Explanation A visual explanation of the Lumiere paper, the model architecture, and the results are available. https://youtu.be/QcYa6xxsMTU?si=IBpUqjPtj8lJDHo0&embedable=true So, What Is Temporal Consistency? We all know that videos are a sequence of images. So, let's take a sequence of images shown in the top row in the above figure. If we narrow down just one row in the image, which is indicated by the green line going from left to right, we need to see a smooth transition in the pixel values between images in the sequence. If the transition is smooth, then we won’t see a jumping effect while we are watching the video. For example, if we take Stable Video Diffusion [2], and see the video of an astronaut walking on the moon (above), we can see that his hands just disappear between frames. In other words, the temporal consistency between frames is lacking. This kind of temporal inconsistency between time and the intensity in the x direction can be plotted as the X-T Slice as highlighted in the figure above. And if there is temporal inconsistency, it is highlighted in the plot of the X-T slice. Lumiere addresses this problem by introducing a space-time diffusion model and a modified U-Net architecture present in the diffusion model. Pipeline of a Text to Video Before looking into the details, let's start with the typical pipeline of a Text-to-video generation model. These pipelines sample every 5th frame as the keyframes from the input video sequence and train a base model that can generate these keyframes at a resolution as low as 128 by 128 at just 3 frames per second. Then temporal Super Resolution is used to increase the frame rate by predicting the intermediate frames. So the frame rate now becomes 16 frames per second. The spatial resolution of these frames is then increased to say 1024 by 1024 by a spatial superresolution network (SSR) which finally leads to our generated video. The base model used in this framework is usually a diffusion model which in turn has a U-Net inside it. The Proposed Pipeline of Lumiere The proposed pipeline of Lumiere on the other hand processes all the frames at once without dropping frames. To cope with the computational cost of processing all the frames, the architecture of the base diffusion model is modified to come up with the space-time UNet architecture or STUNet. As STUNet deals with all the input frames, the need for Temporal Super Resolution or TSR is eliminated. So, the pipeline still has the spatial super-resolution or SSR. But the novelty is the introduction of MultiDiffusion. From U-Net to STUNet Let's quickly review the U-Net before looking into the space-time U-Net. The input to the U-Net is a 3 3-dimensional image with Width W, Height H, and channels RGB. After every double convolution stage of the U-Net, we apply max pooling to downsample or reduce the spatial dimension of the features. This spatial dimension reduction step is indicated by the red arrows. Similarly, during the decoder stage, there are up convolutions to increase or upsample the resolution back to the size of the input. When it comes to videos, we have an additional dimension in the input which is time. so space-time U-Net proposes to downsample and up-sample the video not only in the spatial dimension but also in the dimension of time T. This is the main idea of temporal resizing and is the main contribution of this Lumiere paper. For resizing, they use 3D pooling instead of 2D pooling as the input now has an additional dimension. Like me, you may be surprised by the simplicity of the idea. The authors themselves have mentioned in the paper: Surprisingly, this design choice has been overlooked by previous T2V models, which follows the convention to include only spatial down and up-sampling operations in the architecture, and maintain a fixed temporal resolution across the network. Implementation Let's get to some of the nuances of the implementation. They use factorized convolution introduced in this paper called Video Diffusion Models. The idea is to change each 2D convolution into a space-only 3D convolution, for instance, by changing each 3x3 convolution into a 1x3x3 convolution. For the attention, after each spatial attention block, we insert a temporal attention block that performs attention over the first axis and treats the spatial axes as batch axes. With those two changes, the factorized convolution blocks are added to the pre-trained model, and only the additional layers are trained with the pre-trained layer weights fixed. The second novelty of the paper is the MultiDiffusion introduced during the spatial superresolution. If you take the video generation models before lumiere, the spatial superresolution model takes a sequence of frames. However, the sequences were not overlapping. For example, the first 8 frames and the next 8 frames taken as input by the SSR module are separate without any overlap. But when it comes to Lumiere, the first 8 frames and the second 8 frames have an overlap of two frames. By doing this, the spatial super-resolution model seems to achieve smooth transitions between the temporal segments. This is what is referred to as multiDiffusion in the paper. Applications Coupling the two proposed techniques along with the absence of a cascaded diffusion model architecture which is prevalent in prior architectures such as imagen video, leads to quite a few varied applications. For example: The model can convert text to videos with prompts such as “an astronaut walking on planet Mars making a detour around his base” or “a dog driving a car wearing funny sunglasses.” It can convert images to videos along with a text prompt such as “A girl winking and smiling.” It can stylize generation with a reference image and a text prompt such as “a bear dancing.” Talking of Cinemagraphs, it can animate regions selected by the user such as fire or steam. It can even edit dresses that people are wearing with just a single prompt. Evaluation To evaluate the model quantitatively, the model was run through a user study where the users compared the results of the proposed model against some state-of-the-art models such as Pika, ZeroScope, or stable video diffusion. The results indicate that the users preferred the Lumiere model both in the case of text to video and image to video. Conclusion So, to conclude, other than all the promotional video stunts, such as the smiling Monalisa, the contribution of this paper is fairly simple. Put in one line, the paper introduces a downsampling of the temporal channel. This coupled with MultiDiffusion, which is nothing but having overlapped frames fed into the superresolution model, generates high-fidelity videos that are temporally consistent. What I would have liked to see is some ablation studies in the paper showing the results with and without the multiDiffusion process. That brings us to the end of this article. The next time someone talks to you about Lumiere, you know what to say in one line. I hope that provided some insight into the Lumiere model. I will see you in my next, until then, take care… References [1] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, Inbar Mosseri, (2024), arXiv preprint. A Space-Time Diffusion Model for Video Generation [2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, Robin Rombach, (2023), arXiv preprint. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets [3] Olaf Ronneberger, Philipp Fischer and Thomas Brox, (2015), International Conference on Medical Image Computing and Computer-Assisted Intervention. U-Net: Convolutional Networks for Biomedical Image Segmentation Also published here

Google Unveils Its Most Promising Text-to-Video Model Yet: Lumiere

গুগল এখনও পর্যন্ত তার সবচেয়ে প্রতিশ্রুতিশীল পাঠ্য-টু-ভিডিও মডেল উন্মোচন করেছে: লুমিয়ের

Google stellt sein bisher vielversprechendstes Text-to-Video-Modell vor: Lumiere

Google presenta su modelo de texto a video más prometedor hasta el momento: Lumiere

Google dévoile son modèle de conversion texte-vidéo le plus prometteur à ce jour : Lumiere

Google ने अपना अब तक का सबसे आशाजनक टेक्स्ट-टू-वीडियो मॉडल लॉन्च किया: लुमिएरे

Google、これまでで最も有望なテキストからビデオへのモデル「Lumiere」を発表

Google은 지금까지 가장 유망한 텍스트-비디오 모델인 Lumiere를 공개했습니다.

Google revela seu modelo de conversão de texto em vídeo mais promissor: Lumiere

Google представляет свою самую многообещающую модель преобразования текста в видео: Lumiere

Google Şimdiye Kadarki En Umut Verici Metinden Videoya Modelini Açıkladı: Lumiere

Google công bố mô hình chuyển văn bản thành video hứa hẹn nhất: Lumiere

谷歌推出迄今为止最有前途的文本转视频模型：Lumiere

In the dawn of time, long before the world unfurled its mysteries, the realm of mathematics concealed a plethora of treasures and secrets. Ancient, primordial laws seemed to govern the realm of numbers since time immemorial, revealing its most enchanting wonders through the dance of prime numbers. You are likely acquainted with the concept of primes—numbers divisible only by 1 and themselves. Among them, we encounter such distinguished figures as 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, and 31. (It's worth noting that 1 is not considered prime, and one might argue that 2 and 3 deserve unique categorization). Among the intriguing categories of prime numbers, we find the enigmatic "twin primes"—pairs of primes differing by a mere 2, such as 5-7, 11-13, 17-19, and so forth. For now, let's keep these notions in mind: prime numbers and their various categories, each showcasing specific similarities. Some may appear disorderly until their underlying patterns are unveiled. But until then... What is a prime? The concept of prime numbers serves as a method to delineate the number line without leaving any voids. Primes can be regarded as the fundamental building blocks of all numbers. Take, for instance, the number 2. Multiplying 2 by itself yields 4, then 6, 8, 10, 12, and so forth. Consequently, no even number can be prime, as it inevitably results from combining 2 with another number. We now understand that prime numbers, in their most foundational form, adopt the structure of odd numbers (2n+1). I opt for the 2n+1 format because, if we were to choose 2n-1, we would encounter -1 when considering 0. This may not necessarily violate any rules, as in the case of 2n-1, with n as 1, we obtain the result of 1, which is charming, but uncertainties persist. Now, we have illustrated that the essence of primes, in its most general and universal form, is 2n+1 (or -). Concerning twin primes, if we locate the midpoint between them, we discover that (2n+1)+(2n+3) equals 4n+4. (4n+4)/2 is 2n+2. This midpoint, as gleaned from numerous pattern-seeking endeavors, is consistently divisible by 6 and 9. Further division by 2, yielding n+1, reveals that n+1 is always divisible by 3, 6, and 9. The proportions among their divisors are remarkably balanced, with the difference between divisors of 3 being approximately ±5% more or less than 6 and 9, and the same for 6 and 9 in relation to others, respectively. But then, what do we find when seeking patterns? What exactly do we want to achieve? The patterns in our interactions I won't delve into the etymology of the word "pattern,” but I am willing to explain what it simply means to me. Patterns are self-repetitive structures or elements that define a whole structure. The emergence of patterns can be observed by an external observer focusing on the object or behavior itself—a means to predict outcomes. But why do we seek to predict prime numbers? What is so special about them? Why are they extensively used? Here, we step into the realm of cryptography and the way primes function as an unbreakable cipher meant to resist decryption. Their resilience is anchored in the difficulty of factoring these primes into smaller primes. This is a fundamental principle in the field of cryptography. The security of certain cryptographic algorithms, such as RSA (Rivest–Shamir–Adleman), relies on the difficulty of factoring the product of two large prime numbers back into their original primes. This process is known as integer factorization. The larger the prime numbers involved, the more computationally challenging it becomes to factorize their product, making it difficult for unauthorized parties to decipher encrypted messages. This property forms the basis for the security of many cryptographic systems that use prime numbers in their algorithms. Now, of course, a way of finding and playing with every single prime would indeed be an interesting avenue. But what could it mean in its very essence? What if we choose the envision this play differently? Exploring Primes as an Unbreakable Form of Numbering: Imagine thinking of prime numbers in a similar way to squares. For instance, take the number 7. You can represent it as a sum of 7 smaller squares. Now, the interesting thing is that these small squares cannot be divided into any smaller ones. Unlike other numbers, there's no way to break them down further. The analogy with Squares: Consider the idea of "folding" a number. Take the number 8 as an example. You can fold it into 2 x 4 squares, or even 4 x 2 squares, effectively changing its shape. This folding and reshaping are possible for certain numbers. The Concept of Folding: When it comes to prime numbers, this folding or reshaping is impossible. The shape of a prime number is always static, and there's no known way to alter it. This unique property suggests that primes could be used to create a new numbering system. Unalterable Nature of Primes: Now, you might wonder, what benefits could we derive from such a system based solely on primes? One interesting application could be in communication. For example, if we send signals in the form of primes from one point to another, the receiving end would immediately recognize that the message is based on a prime number. It could then intelligently select the most similar prime to interpret the message. This process could lead to more efficient noise reduction, filtering out irrelevant information. Potential Benefits: By considering primes as unalterable shapes in a numbering system, we open up possibilities for unique applications. The analogy with squares helps illustrate the distinct nature of prime numbers. The idea of using primes in communication suggests a potential avenue for creating a system where the unbreakable aspects of primes contribute to efficient information transmission and noise filtering. In Summary: I like how clearly ChatGPT explains some ideas. However, who is the one who arrives at those ideas? Well, you. The one who simply sits, reads, and learns something new every day. Not for the mere purpose of learning, but to use what you learn to learn something completely new. The end? As we delve into the profound realm of prime numbers, a tantalizing question arises: Can we devise a form of information that harnesses the intrinsic properties of primes, effectively mitigating the impact of noise interference? Such a revolutionary system might demand larger primes with wider intervals, considering the heightened differences between them. Remarkably, even for global communication, primes up to 10,000 could prove sufficient based on our existing knowledge. However, as complexity beckons, the necessity for larger intervals becomes apparent. Yet, this challenge invites us to redefine not only letters but entire concepts. Picture a scenario where, regardless of distance, we exchange ideas through a series of carefully crafted formulas—each recipient deciphering the concepts within their knowledge domain. The world of primes unveils not just mathematical wonders but an intriguing prospect for reshaping the very nature of information transfer.

Prime Numbers: The Unbreakable Building Blocks of Mathematics

Tranformer-based vision models are gradually evolving and are reported to be as good as Convolutional models on Classification, Segmentations and Action Recognition tasks. We have a whole array of Convolutional models for vision tasks and are more popular than Transformer-based ones. This blog delves into the SWin Transformer Vision model that was presented in the International Conference for Computer Vision (ICCV) 2021 by the Microsoft Research Team. It  benchmarks its performance against several SOTA Convolution-based models on the Dog Breed Image Classification task. Will transformer-based models become the next big thing in Computer Vision? With transformers being a successful solution for language tasks, will it unify the various AI subfields and present powerful solutions to more complex problems? So rolling up my sleeves to evaluate how good they are on the classification task to begin with. Chosen Data and its Problem Statement: The myriad of dog breeds with subtle changes in their physical appearances have been a challenge to veterinarians, dog owners, animal shelter staff and potential dog owners in identifying their right breed. They need to identify the breed in order to provide appropriate training, treatment and meet their nutritional needs.The data is sourced from the that contains ~20K images of 120 breeds of dogs across the world. This data has been split almost equally into train and test set for a Kaggle Competition . Stanford Dog Dataset Dog Breed Identification The objective of the solution is to build a dog breed identification system capable of correctly identifying dog breeds with minimal data and rightly identifying even similar looking dog breeds. This is  a multi-class classification task and for every image, the model has to predict probability for all the 120 breeds. The one with the highest probability is the most probable breed of the dog in the image. Exploratory Data Analysis For a breed, the training data has a maximum of 120+ images and a minimum of 66 images. Each class, on an average is expected to have 85 images. Looking at the bar plot, it is concluded that there are no class imbalances. Though there is no class imbalance, data may be insufficient to train the neural network. Image Augmentation using random image perturbations and pre-trained models will be able to circumvent this problem. Top 5 breeds with the most images are scottish_deerhound, maltese_dog, afghan_hound, entlebucher and bernese_mountain_dog. Bottom 5 breeds with the least images are golden_retriever, brabancon_griffon, komondor, eskimo_dog and briard. A quick analysis on spatial dimensions of the training images is done to understand the distribution of image height, width and their aspect ratios. Images with very low (<0.5) and very high (>2.3) aspect ratios are considered to be anomaly images. 1.5 is considered to be a good aspect ratio. In analysing various dog breeds, below pairs of breeds were generally found to look alike. 'boston_bull', 'french_bulldog' 'beagle', 'english_foxhound' 'bernese_mountain_dog', 'greater_swiss_mountain_dog' 'malamute', 'siberian_husky' 'basenji', 'ibizan_hound' 'vizsla', 'rhodesian_ridgeback' 'lhasa', 'shih-tzu' 'whippet', 'italian_greyhound' 'brittany_spaniel', 'welsh_springer_spaniel', 'blenheim_spaniel' 'malinois', 'german_shepherd' 'border_collie', 'collie' 'norfolk_terrier', 'norwich_terrier’ Evaluation Metrics LogLoss, which stringently evaluates the confidence of the model predictions for all 120 classes by comparing the prediction probability to the ground truth probability, is the key metric. N*N confusion matrix is used to study the outcome of the predictions to analyse if there are any pair of classes confused by the model and incorrectly predicted. SWIN - Transformer Based Model This architecture is based on developed by Microsoft Research Team. This paper discusses an improved ViT architecture that produces a hierarchical representation of feature maps reducing the computation complexity of self-attention mechanism from quadratic to linear. It is proven to give identical results as SOTA Convolution Networks like EfficientNet on Imagenet classification problem. “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows” The building blocks of this architecture is explained in the below notes: Image Patching: In NLP, the tokens which are the processing elements of a model, are the words in a sentence where the size of 1 token is 1 (just 1 word). ViT ( sion ransformers) treat “image patches” as tokens where each patch is a partition of an image consisting of a group of neighbouring pixels. Each image patch is a token. The size of 1 token in any ViT is patch_height * patch_width * number of channels. Based on the patch dimensions, we get a number of patches or tokens for a single image. If the image size (H*W*3) is 224 * 224 * 3 pixels and the patch size is 4 * 4, then we get 224/4 * 224/4 = 56 * 56 patches or 3136 tokens from the image. Each token or patch will be of size 4*4*3 = 48 dimensions or 48 pixels of data. So the input to the architecture for this image consists of 3136 tokens each of size 48 dimensions. Vi T The underlying mechanism of the SWIN transformer is analogous to any CNN based architecture where the spatial dimensions of the image is decreased and the number of channels is increased. SWIN transformer at every stage in the hierarchical architecture, also reduces the number of image patches or the number of tokens while increasing the token dimensions. With this mechanism in mind, it is easier to understand the SWIN architecture easily. At every stage of the architecture, we can see the number of tokens decreasing while the token size is increasing. The SWIN-T architecture, apart from the “ is also made up of 3 other building blocks - Linear Embedding, Swin Transformer Block, Patch Merging. These building blocks are repeated and it process feature maps in a hierarchical fashion. Patch Partitioner”, Linear Embedding: The 3136 tokens each of 48 dimension from the “Patch Partitioner” are fed to a feed forward layer to embed the token of 48 feature vector into a feature vector of size ‘C’. ‘C’ here acts as the capacity of the transformer and the SWIN architecture has 4 variants based on it. Tiny (Swin-T)    - C is ‘96’ Small (Swin-S) - C is ‘96’ Base(Swin-B)   -  C is ‘128’ Large(Swin-L)  - C is ‘192’ Image patching and linear embedding are jointly implemented in a single convolution  whose kernel-size and stride-length is same as the patch-size. Number of channels in the convolutional will be ‘C’. SWin Transformer Block: The SWin Transformer Block is different from the standard transformer block in the ViT architecture. In SWin Transformers, the layer is replaced by either the or the . Multi-head Self Attention (MSA) Window MSA (W-MSA) module Shifted Window MSA (SW-MSA) module Stage1 consists of 2 SWIN-T Transformer Blocks (refer image) where the first Transformer Block has Window MSA (W-MSA) and the second Transformer Block has Shifted Window MSA (SW-MSA) module. In the SWin Transformer Block, the inputs and the outputs of the W-MSA and SW-MSA layers are passed via Normalization Layers. It is then subjected to a 2 layered Feed Forward Network with Gaussian Error Linear Units (GELU) activation. There are residual connections within each block and between these 2 blocks. Window MSA (W-MSA) and Shifted Window MSA (SW-MSA) modules Why is the standard attention layer in ViT replaced with the Windowed MSA layer? The standard attention layer in ViT was a global one calculating the attention of a patch with all other patches in the image thus leading to a quadratic complexity proportional to the image dimensions. This doesn't scale very well for high resolution images. The self-attention mechanism in the W-MSA or SW-MSA module is a local one that calculates self-attention only between patches within the same window of the image and not outside the windows. Windows are like larger partitions of the image where each window comprises of M*M patches. Replacing global self-attention with local self-attention reduced the computational complexity from quadratic to linear. Why both W-MSA and SW-MSA for local self-attention? What is the  difference between them? The key difference between W-MSA and SW-MSA attention modules is in the way how the windows for the image are configured. In W-MSA module, a regular window partitioning strategy is followed. The image is evenly partitioned into non-overlapping windows starting from the top-left pixel of the image, and each window contains M*M or M2 patches . In SW-MSA module, the window configuration is shifted from that of the W-MSA layer, by displacing the windows by (M/2, M/2) patches from the regular partitioning strategy. Why a shifted window partitioning strategy in SW-MSA? Since the attention is restricted locally within a window in W-MSA, the shifted window enables cross-window attention to still yield the benefits of global attention. This is possible because the boundaries of window1 in W-MSA layer are shared with windows W2, W4 and W5 in SW-MSA layer. Hence global attention happens indirectly via “local attention on shifted windows”. Patch Merging Layer Patch Merging layer reduces the number of tokens as the network gets deeper and deeper. The first patch merging layer concatenates the features of each group of 2×2 neighbouring patches. SWIN Transformers on Dog Breed Classification The package in PyPI  has pretrained TF-Keras variants of the SWIN Transformers and is built based on the . Its code is available in . tfswin is used to train the dog breed images. tfswin official pytorch implementation github from tfswin import SwinTransformerBase224, preprocess_input
def build_model1(swintransformer):
  tf.keras.backend.clear_session()

  inputs = Input(shape=(resize_height, resize_width, 3))

  outputs = Lambda(preprocess_input)(inputs)
  outputs = swintransformer(outputs)
  outputs = Dense(num_classes, activation='softmax')(outputs)

  swin_model = Model(inputs=inputs, outputs=outputs)

  return swin_model

#build the model
swintransformer = SwinTransformerBase224(include_top=False,pooling='avg')
swin_model1     = build_model1(swintransformer)

#set the layers of the pretrained model as non-trainable
for layer in swin_model1.layers[2].layers:   
  layer.trainable = False

swin_model1.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),loss='categorical_crossentropy',metrics=['accuracy']) Convolutional Based Models To begin with, various ResNet variants were trained with different replacements for top layers, and partially to totally freezing the network. #Logloss of the test set using various ResNet variants
+------------+---------------+-------------------------+----------+
| Model Name |   Retrained   |  Top Layers Replacement | Log_Loss |
+------------+---------------+-------------------------+----------+
|  ResNet50  |      None     |   ConvBlock_FC_Output   | 0.96463  |
|  ResNet50  |      None     | GlobalAvgPooling_Output | 0.58147  |
|  ResNet50  | last 4 layers |   ConvBlock_FC_Output   | 2.10158  |
|  ResNet50  | last 4 layers | GlobalAvgPooling_Output | 0.57019  |
+------------+---------------+-------------------------+----------+ Code corresponding to the ResNet50 model with least log loss from tensorflow.keras.layers import Input,Conv2D,Dense,BatchNormalization,Flatten,Concatenate, Dropout,MaxPooling2D
from tensorflow.keras.models import Model
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input

def build_model():

  tf.keras.backend.clear_session()

  inputs = Input(shape=(resize_height, resize_width, 3))

  #added preprocess_input method as a layer to convert input images to those expected by Resnet
  processed_inputs = preprocess_input(inputs)  

  #use the pretrained ResNet model (Parameter pooling = 'avg' will take care of the Gobal Average Pooling of the ResNet model features)
  base_model = ResNet50(weights="imagenet", include_top=False,pooling='avg')(processed_inputs) 
  
  #output layer
  output = Dense(units=num_classes,activation='softmax',name='Output')(base_model)

  resnet_model = Model(inputs=inputs, outputs=output)
  return resnet_model

#build the model
resnet_model = build_model()

#set the layers of the resnet pretrained model as non-trainable except for its last 4 layers which needs to be re-trained for this data
for layer in resnet_model.layers[3].layers[:-4]:   
  layer.trainable = False

#compile the model
resnet_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),loss='categorical_crossentropy',metrics=['accuracy'])
print(resnet_model.summary())

history = resnet_model.fit(train_ds,
                    epochs=50,
                    validation_data=val_ds, callbacks=callbacks_list) More sophisticated pre-trained Convolutional models like EfficientNet, NASNet, InceptionResNet, Xception and InceptionV3 were also trained using the dog breed images both in standalone and also in an ensembled fashion. The log loss on the test data (consisting of 10K images) from these models were lower than any of the ResNet variants. #Logloss of the standalone model variants
+----------------------------+-------------+
|         Model Name         |   Log_Loss  |
+----------------------------+-------------+
|      EfficientNetV2M       |   0.28347   |
|      Inception ResNet      |   0.28623   |
|        NasNetLarge         |   0.33285   |
|          Xception          |   0.34187   |
|        Inception_V3        |   0.54297   |
| EfficientNetV2M_GlobalAveg |   0.50423   |
|   InceptionV3_GlobalAveg   |   0.46402   |
+----------------------------+-------------+ For standalone models, the layers of the pre-trained models were frozen and their top layers were replaced by either Convolutional Layer followed by MaxPooling Layer Global Average Pooling Layer EfficientNet architecture is a rightly scaled model in all 3 dimensions - depth, width and resolution. NASNet-Large is the outcome of model architecture search done using various hyperparameter search algorithms, optimization algorithms and reinforcement learning algorithms. InceptionV3 factorized larger convolutions into smaller asynchronous convolutions and also included auxiliary classifiers to improve model convergence. Xception architecture does depth-wise separable convolutions instead of the inception modules in the Inception architecture. The log loss from the ensemble models reduced by > 0.1 than the best performing standalone model. +--------------------------------------------------------------------------+-----------+
|                             Model Name                                   |  Log_Loss |
+--------------------------------------------------------------------------+-----------+  
| Ensemble1 - EfficientNEt,InceptionResNet,NasNet,Xception)                |  0.17363  |
| Ensemble2 - EfficientNEt,InceptionResNet,NasNet,Xception and InceptionV3 |  0.16914  |
| Ensemble3 - Ensemble2 with 50% dropout.                                  |  0.16678  |
| Ensemble4 - Ensemble of various EfficientNet Architecture                |  0.16519  |
+--------------------------------------------------------------------------+-----------+ Each of these models accepts varied input formats and in Keras they have their own preprocessing functions. Benchmarking Outcome +----------------------------------+------------+----------------------+----------+
|            Model Name            | Parameters | Train time (seconds) | Log_Loss |
+----------------------------------+------------+----------------------+----------+
|  EfficientNet_ConvBlock_Output   |   54.7M    |        ~260s         | 0.28347  |
| InceptionResNet_ConvBlock_Output |   56.1M    |        ~260s         | 0.28623  |
|   NASNetLarge_ConvBlock_Output   |   89.6M    |        ~330s         | 0.33285  |
|    XCeption_ConvBlock_Output     |   23.3M    |        ~240s         | 0.34187  |
|   InceptionV3_ConvBlock_Output   |   24.2M    |        ~225s         | 0.54297  |
|      EfficientNet_GlobalAvg      |   53.3M    |        ~260s         | 0.50423  |
|      InceptionV3_GlobalAvg       |    22M     |        ~215s         | 0.46402  |
|           swin_base224           |   86.8M    |        ~550s         | 0.47289  |
|           swin_base384           |    87M     |        ~600s         | 0.41902  |
|          swin_large384           |    195M    |        ~1000s        | 0.42207  |
+----------------------------------+------------+----------------------+----------+ SWIN Transformers have performed better than all of the ResNet50 variants and InceptionV3 model. Log-Loss of SWIN Transformer on this data is slightly higher compared to InceptionResNet, EfficientNet, Xception and NasNet Large models when their outputs are processed subsequently by convolutional layer followed by Maxpooling. SWIN however performs is as good as EfficientNet model when their average outputs are directly processed by the output layer. SWIN models are larger compared to any of the Convolutional models and hence will take a hit on the system throughput and latency. This study proved useful in understanding the application of transformer-based models for computer vision. You can find the code to the notebooks on my and the GUI for this usecase here. Github References: https://www.kaggle.com/competitions/dog-breed-identification https://arxiv.org/pdf/2103.14030.pdf https://www.youtube.com/watch?v=tFYxJZBAbE8&t=362s https://towardsdatascience.com/swin-vision-transformers-hacking-the-human-eye-4223ba9764c3 https://github.com/shkarupa-alex/tfswin

Image Classification in 2022

Clasificación de imágenes en 2022

Classification des images en 2022

2022 में छवि वर्गीकरण

2022 年の画像分類

Classificação de imagens em 2022

Phân loại hình ảnh năm 2022

2022年的图像分类

Can software project estimation be as realistic as possible? This article will walk you through how we, at UpsilonIT , approach the estimating process. We will also share some hands-on techniques and useful tips on making it more accurate and robust. Why Estimation Is an Essential Stage of Any Software Project? Many outsourcing software projects fail because their estimate is not reliable. Failure means the developer did not meet the deadline, overran the budget, or did not deliver what was expected. A study conducted by found that 66% of software projects have cost overruns. A third of them go beyond the estimated schedule, and almost 20% of them fall short of promised benefits. McKinsey Sometimes it seems that the process of estimation can significantly increase the cost of the pre-development phase. But it is crucial to realize: the resources spent at the initial stage of the project are a good investment in the outcome. The more accurate the estimation, the more customer-satisfying the result will be. That is why a correct estimate is a crucial part of project management procedures, and it is really worth it. What Is Project Estimation? Project estimation is a process of calculating and predicting the costs necessary for its successful realization. In software development, estimations imply making predictions regarding the most realistic scale of efforts that should be put into developing a software product. This information allows suggesting the amount of money that needs to be invested in the project, the number of specialists who would work on the latter, and, finally, the time required for all that. Accurate Estimating: Complicated but Crucial Task As estimates are based upon incomplete, imperfect knowledge and assumptions about the future, they all have uncertainty. There is no such thing as a precise, single-value estimate. A more traditional ‘waterfall’ approach implies that all the software project specifications must be provided upfront before the estimation process starts. But now, everyone is going agile. Stakeholders want to begin immediately after preparing brief requirements and then add new features literally on-the-go. MVPs (Minimum Viable Products) are released quickly, feedback is obtained, and improvements are made iteratively. In those conditions, planning and estimating can be rather challenging. So the question we want to answer is, This article will walk you through how we approach the software project estimation process at UpsilonIT and share some tips that help us make it more accurate and robust. ‘How can we cover all those uncertainties and make the estimation as realistic as possible?’ Estimation Process: How We Run It at UpsilonIT Let’s look at the whole process of project analysis and estimations at UpsilonIT — from the moment of figuring out the client’s expectations to giving the final project calculation. 1. Preparation We always start by digging into the client’s main challenges to paint a clear picture of their business and development goals. It’s important to gather as much information on the future project as possible. There are two main sources where we get it: Online video calls via Skype, Zoom, or Hangouts: we discuss the project in general by asking questions and getting answers Documents and data provided by the client: specifications, mockups, wireframes, use cases, user stories, software specifications, etc. All of this helps us estimate the cost of building the client’s product more precisely. The more information we get our hands on at this stage, the more accurate our estimation will be. Here is a list of things we usually try to clarify at the very beginning: What is the pain that you are solving for your customers? What are the main features of the product? What platforms will the product be based on (web, mobile, back-end)? What existing components or parts of the system do you already have? Can you describe the main use cases of the new application? Can you prioritize the functions of the future product? Do you have a specific technical stack that needs to be used for this project?What is your expected timeframe for the project? What is your expected budget for this project? Once we are sure that nothing is missing and that we’re in line with the client’s expectations, we go to the next phase: assembling the team. 2. Assembling the Right Time After collecting all the information, we gather professionals who have the most extensive competence in the respective domain. The team responsible for the project estimation is usually composed of senior-level specialists and includes: CTO Senior back-end developer Senior front-end developer Other specialists upon request: a project manager, UX/UI designer, QA specialist, technical writer, and others. In the vast majority of cases, specialists who make the estimation subsequently become a part of a project development team. The key criteria for selecting that team’s members are: Professional expertise: the relevant experience of working with specific technologies and similar projects. Ability to start the project as soon as possible Other factors: presence in the client’s timezone, full- or part-time employment, language proficiency, etc. Pro Tip: Assembling the right team is one of the most sensitive questions in software outsourcing. A customer wants to be ensured that the most skilled specialists with relevant expertise will be assigned to the project. That will guarantee the expected quality and hassle-free delivery of the final product. Also, customers want to see ‘real faces’ not just abstract ‘senior front-end developers’ in the estimations, including in order to understand further whom they are speaking to on the conference call. That is why every estimate made by UpsilonIT is accompanied by a document of the CV format with key information about each team member, including his or her: Expertise Skill matrix Project history Personal info 3. Research & Analysis During this phase, the team members set up a number of meetings to discuss project requirements, exchange their opinions, and brainstorm ideas. If some details need more clarification, the discussions may alternate with the research sessions. At the end of the Research & Analysis phase, our team provides potential clients with the following deliverables: High-level architecture & infrastructure design Recommended tech stack Mockups and wireframes (if not provided by a client) Desired breakdown of the team Rough timelines 4. Defining Scope & Size of the Project The next step of our estimation process is building the scope of work that covers software requirements. At UpsilonIT, we use the Work Breakdown Structure (WBS), which visualizes each software development stage with related tasks. To achieve the project’s goal, we split it into milestones, which are broken down into tasks, subtasks, etc. At UpsilonIT, we define a milestone as a marker that indicates an achievement or a deliverable, for example, the completion of a large task or reaching a phase in a project. For most of the projects, we figure out the following milestones: Discovery phase: documentation analysis, choice of 3rd party software tools, decisions on possible integrations, preparation of wireframes (if not provided by a client), etc. Design phase: collaborative work with a product owner that can include design theme workshops, brand identity design, UX and UI design, prototyping, etc. Project setup phase: setting up of infrastructure, deployment of environments, configuring of security, etc. Development phase: development of functionality based on user stories (and also tasks grouped into milestones). It can include user authentication, messaging, payment integration, mapping, video streaming, reporting, etc. Testing phase: functional testing, usability testing, user acceptance testing (UAT). Delivery phase: transfer of documentation and intellectual property, deployment, and implementation of the solution, including publishing on marketplaces, training, presentation of the product, if necessary, and more. How does it work in practice? Here you can see an example of how we broke down into milestones and user stories the structure of a health & wellness digital platform. Another significant question is to decide whether to build a turnkey solution or start from the Minimum Viable Product (MVP). About 80% of startups and SMEs we worked with had an MVP first. From our experience, we recommend to ignore non-primary functions by creating an MVP before the finished product, especially if: Project requirements are unspecified Product market fit is not validated yet Features must be added and tested quickly Budget is limited Pro Tip: When defining the scope of the project, it can be very helpful to visualize everything needed to be accomplished, organize tasks into multiple levels, and display these elements graphically. For that purpose, we use the Miro platform. With Miro’s help, we develop the project’s hierarchical structure that makes it easier to oversee a complex project and make sure every task gets done. When creating a WBS for a future project, we follow four fundamental principles: Hierarchy: every ‘child’ on the graph has a hierarchical relationship with its parent task. 100% rule: every level of the graph must make up 100% of the ‘parent’ level, and it must have at least two ‘child’ elements. Mutually exclusive elements: every element at each level has to be mutually exclusive — there can’t be any overlap between deliverables or work. Outcome-oriented: graphic depiction must focus on the outcomes rather than the activities required to produce them. 5. Project Estimate by PERT Method After conducting a detailed scope breakdown, we start to create a more accurate estimate using a PERT (Program Evaluation and Review Technique) method. We calculate a realistic cost estimation using an , , and a within this approach. There are different ways of interpreting the numbers depending on how we want to weight potential risks, but in most cases, we use the formula: optimistic case estimate a pessimistic case estimate most likely case estimate (optimistic + pessimistic + (4 x most likely))/6 = expected estimate Let us consider an example to see how these formulas work. Let us assume we have to calculate the expected travel time it takes to deliver a shipment from point ‘A’ to point ‘B’. There could be 3 different scenarios: roads would be free of traffic congestion, and there will be no stopping at traffic signals. Optimistic scenario: there would be severe traffic bottlenecks (may be due to a major accident), or there will be some unscheduled stops (may be due to vehicle breakdown). Pessimistic scenario: there would be normal traffic conditions. Most likely scenario: Let us assume that our 3 estimates (optimistic, pessimistic & most likely) were 450 minutes, 2250 minutes, and 900 minutes respectively. Putting these values in the formulas, we get: Expected travel time = (450 + 2250 + 4 x 900))/6 = 1050 minutes It essentially means that, in most cases, the expected travel time will take 1050 minutes. How can it look in a real project estimate? Let us explain how we estimated the task by PERT method for a social sports application. ‘Credits (Bank) & Prize claim’ 7 days (including 7 days for back-end and 6 days for front-end development. Both teams work simultaneously). Optimistic scenario: 14 days (including 14 days for back-end and 12 days for front-end development). Pessimistic scenario: 11 days (including 11 days for back-end and 10 days for front-end development). Most likely scenario: Expected estimate: (7 + 14 + (4 x 11))/6 = 10.83 days At the output, we get an estimate which takes into account some uncertainty and risk in tasks. The PERT method can be quite time-consuming, but it can provide a high degree of accuracy. 6. Building a Timeline Chart After determining the total time needed for each task and milestone, we build the project timeline. We line tasks end-to-end, adjust their lengths to reflect the amount of time allotted, and then add milestones. We turn that all into a solid timeline chart with expected delivery dates provided. It is especially helpful to have such visualization on projects where several teams work simultaneously on several milestones. Such a timeline chart shapes the entire project, indicating which tasks are to be completed at which stage and which resources need to be involved along the way. It also allows us to quickly see the proportion of time allocated to each project task or milestone and provides a visual interpretation of how the project will run phase by phase. 7. Calculating Costs After estimating the number of hours of work required to complete the defined scope, it’s time to price the project. We calculate the total project cost by multiplying the number of man-hours (estimated using the PERT method) by the applicable rates. For ease of calculation, let’s make the following assumptions: One man-hour = work completed in an hour of uninterrupted effort by a specialist. One day = a standard 8-hour working day. In IT projects, hourly rates range depending on the specialist’s professional category, which is defined by his or her job functionality, level of seniority, uniqueness of skills, etc. Of course, it will not cost the same for one hour of a junior programmer as a senior one. Therefore, when calculating the cost of the project, we take into account that the multiplication of the cost must be factorized by the hourly rate of each professional category: Total cost = (RH1 x H1) + … + (RHn x Hn) In this formula, RH1 is equal to the hourly rate of the professional category #1, and H1 is the total number of hours estimated by the professional category #1. How does it work in practice? For example, we estimated a software project and got the following numbers: The total cost of the project will be: 50x180 + 40x150 + 30x60 = $16800 8. What’s Next? Having put all things together and made our calculations, we share it with the client in person, explaining our estimation instead of just sending over a number without context. We schedule a meeting and go over the document with them, row by row. It allows us to discuss any questions that may have come up along the way. If it becomes clear that the project isn’t doable within the client’s limitations, we can also help to prioritize the functionality and start by developing a minimum viable product (MVP) or even a proof-of-concept (PoC). We try to include this kind of out-of-the-box thinking in our estimates whenever possible, to add value at every step of our collaboration. Conclusion Estimating is prone to error. The overall complexity becomes fully visible only at the point of execution. In the conditions of uncertainty and lack of information, there is a temptation to provide lower numbers to outbid the competition. But that doesn’t mean that the final price for your product will be lower. It may just mean more stress, more unforeseen costs along the way, and as a result, your dissatisfaction. And this is definitely not how great products are built. Our goal is to predict as closely as possible. Of course, some things can change, especially if a project evolves unexpectedly or if a client comes up with new ideas during development. But in most cases, when we give an estimate, we stick to it. The best way to find this out is to partner with us. You can follow our processes, share your thoughts with our developers, and enjoy our transparent communication and quality approach firsthand. We will be happy to do our best to make your ideas happen. Follow UpsilonIT on Hacker Noon for the latest tech know-hows, best software development practices, and insightful case studies from industry experts! Remember: your likes and comments are fuel for our future posts!

Accurate Project Estimations: How To Make Them?

Embeddings have pervaded the data scientist’s toolkit, and dramatically changed how NLP, computer vision, and recommender systems work. However, many data scientists find them archaic and confusing. Many more use them blindly without understanding what they are. In this article, we’ll deep dive into what embeddings are, how they work, and how they are often operationalized in real-world systems. What’s an embedding? To understand embeddings, we must first understand the basic requirements of a machine learning model. Specifically, most machine learning algorithms can only take low-dimensional numerical data as inputs. In the neural network below each of the input features must be numeric. That means that in domains such as recommender systems, we must transform non-numeric variables (ex. items and users) into numbers and vectors. We could try to represent items by a product ID; however, neural networks treat numerical inputs as continuous variables. That means higher numbers are “greater than” lower numbers. It also sees numbers that are similar as being similar items. This makes perfect sense for a field like “age” but is nonsensical when the numbers represent a categorical variable. Prior to embeddings, one of the most common methods used was one-hot encoding. One-Hot Encoding One-hot encoding was a common method for representing categorical variables. This unsupervised technique maps a single category to a vector and generates a binary representation. The actual process is simple. We create a vector with a size equal to the number of categories, with all the values set to 0. We then set the row or rows associated with the given ID or IDs to 1. An illustration of One Hot Encoding This technically works in turning a category into a set of continuous variables, but we literally end up with a huge vector of 0s with a single or a handful of 1’s. This simplicity comes with drawbacks. For variables with many unique categories, it creates an unmanageable number of dimensions. Since each item is technically equidistant in vector space, it omits context around similarity. In vector space, categories with little variance are not any closer together than those with high variance. This means that the terms “Hotdog” and “Hamburger” are no closer together than “Hotdog” and “Pepsi”. As a result, we have no way of evaluating the relationship between two entities. We could generate more one-to-one mappings, or attempt to group them and look for similarities. This requires extensive work and manual labeling that’s typically infeasible. Intuitively, we want to be able to create a denser representation of the categories and maintain some of the implicit relationship information between items. We need a way to reduce the number of categorical variables so we can place items of similar categories closer together. That’s exactly what an embedding is. Embeddings solve the encoding problem Embeddings are dense numerical representations of real-world objects and relationships, expressed as a vector. The vector space quantifies the semantic similarity between categories. Embedding vectors that are close to each other are considered similar. Sometimes, they are used directly for the “Similar items to this” section in an e-commerce store. Other times, embeddings are passed to other models. In those cases, the model can share learnings across similar items rather than treating them as two completely unique categories, as is the case with one-hot encodings. For this reason, embeddings can be used to accurately represent sparse data like clickstreams, text, and e-commerce purchases as features to downstream models. On the other hand, embeddings are much more expensive to compute than one-hot encodings and are far less interpretable. How are Embeddings created? A common way to create an embedding requires us to first set up a supervised machine learning problem. As a side-effect, training that model encodes categories into embedding vectors. For example, we can set up a model that predicts the next movie a user will watch based on what they are watching now. An embedding model will factorize the input into a vector and that vector will be used to predict the next movie. This means that similar vectors are movies that are commonly watched after similar movies. This makes for a great representation to be used for personalization. So even though we are solving a supervised problem, often called the surrogate problem, the actual creation of embeddings is an unsupervised process. Defining a surrogate problem is an art, and dramatically affects the behavior of the embeddings. For example, YouTube’s recommender team realized that using the “predict the next video a user is going to click on” resulted in clickbait becoming rampantly recommended. They moved to “predict the next video and how long they are going to watch it” as a surrogate problem and achieved far better results. Common Embedding Models Below are some common embedding models to learn more about. Principal Component Analysis (PCA) One method for generating embeddings is called Principal Component Analysis (PCA). PCA reduces the dimensionality of an entity by compressing variables into a smaller subset. This allows the model to behave more effectively but makes variables more difficult to interpret, and generally leads to a loss of information. A popular implementation of PCA is a technique called SVD. SVD Singular Value Decomposition, also known as SVD, is a dimensionality reduction technique. SVD reduces the quantity of data set features from N-dimensions to K-dimensions via matrix factorization. For example, let’s represent a user’s video ratings as a matrix of size (Number of users) x (Number of Items) where the value of each cell is the rating that a user gave that item. We first pick a number, k, which is our embedding vector size, and use SVD to turn it into two matrices. One will be (Number of users) x k and the other will be k x (Number of items). In the resulting matrices, if we multiply a user vector by an item vector, we should get our predicted user rating. If we were to multiply both matrices, we’d end up with the original matrix, but densely filled with all of our predicted ratings. It follows that two items that have similar vectors would result in a similar rating from the same user. In this way, we end up creating user and item embeddings. Word2Vec Word2vec generates embeddings from words. Words are encoded into one-hot vectors and fed into a hidden layer that generates hidden weights. Those hidden weights are then used to predict other nearby words. Although these hidden weights are used for training, word2vec will not use them for the task it was trained on. Instead, the hidden weights are returned as embeddings and the model is tossed out. Words that are found in similar contexts will have similar embeddings. Beyond that, embeddings can be used to form analogies. For example, the vector from king to man is very similar to the one from queen to woman. One problem with Word2Vec is that single words have one vector mapping. This means that all semantic uses for a word are combined into one representation. For example, the word “play” in “I’m going to see a play” and “I want to play” will have the same embedding, without the ability to distinguish context. BERT Bidirectional Encoder Representations of Transformers, also known as BERT, is a pre-trained model that solves Word2Vec’s context problems. BERT is trained in two steps. First, it is trained across a huge corpus of data like Wikipedia to generate similar embeddings as Word2Vec. The end-user performs the second training step. They train on a corpus of data that fits their context well, for example, medical literature. BERT will be fine-tuned for that specific use case. Also, to create a word embedding, BERT takes into account the context of the word. That means that the word “play” in “I’m going to see a play” and “I want to play” will correctly have different embeddings. BERT has become the go-to transformer model for generating text embeddings. Embeddings in the Real World Embedding usage started in research labs and quickly became state of the art. Since then, embeddings have found themselves in production machine learning systems across a variety of different fields including NLP, recommender systems, and computer vision. Recommender Systems A recommender system predicts the preferences and ratings of users for a variety of entities/products. The two most common approaches are collaborative filtering and content-based. Collaborative filtering uses actions to train and form recommendations. Modern collaborative filtering systems almost all use embeddings. As an example, we can use the SVD method defined above to build a recommender system. In that system, multiplying a user embedding by an item embedding generates a rating prediction. This provides a clear relationship between users and products. Similar items beget similar ratings from similar users. This attribute can also be used in downstream models. For example, Youtube’s recommender uses embeddings as inputs to a neural network that predicts watch time. Semantic Search Users expect search bars to be smarter than a regex. Whether it’s a customer support page, a blog, or Google, a search bar should understand the intent and context of a query, not just look at words. Search engines used to be built around TF-IDF, which also creates an embedding from text. This kind of semantic search worked by finding a document embedding that’s closest to the query embedding using the nearest neighbor. Today, semantic search utilizes more sophisticated embeddings like BERT and may use them in downstream models. In fact, even Google uses BERT on a large percentage of their queries: . https://searchengineland.com/google-bert-used-on-almost-every-english-query-342193/ Computer Vision In computer vision, embeddings are often used as a way to translate between different contexts. For example, if training a self-driving car, we can transform the image from the car into an embedding and then decide what to do based on that embedded context. By doing so, we can perform transfer learning. We can take a generated image from a game like Grand Theft Auto, turn it into an embedding in the same vector space, and train the driving model without having to feed it tons of expensive, real-world images. Tesla is doing this in practice today. Another interesting example is the AI Art Machine: . It will generate an image based on user input text. For example, if we type in “Nostalgia”, we get the following image. https://colab.research.google.com/drive/1n_xrgKDlGQcCF6O-eL3NOd_x4NSqAUjK#scrollTo=TnMw4FrN6JeB It works by transforming the user’s text and an image into an embedding in the same latent space. It’s composed of four transformers: Image -> Embedding, Text -> Embedding, Embedding -> Text, Image -> Text. With all these, transformations we can translate text to image and visa-versa using an embedding as an intermediate representation. Embedding Operations In the above examples, we see that there are a few common operations applied to embeddings. Any production system that uses embeddings should be able to implement some or all of the below. Averaging Using something like word2vec, we can end up with an embedding for each word, but we often need an embedding for a full sentence. Similarly, in recommender systems, we may know the items a user clicked on recently, but their user embedding may not have been retrained in days. In these situations, we can average embeddings to create higher-level embeddings. In the sentence example, we can create a sentence embedding by averaging each of the word embeddings. In the recommender system, we can create a user embedding by averaging the last N items they clicked. Subtraction/Addition We mentioned earlier how word embeddings also encode analogies via vector differences. Adding and subtracting vectors can be used for a variety of tasks. For example, we can find the average difference between a coat from a cheap brand and a luxury brand. We can store that delta and use it whenever we want to recommend a luxury item that’s similar to the current item that a user is looking at. We can find the difference between a coke and a diet coke and apply it to other drinks, even those that don’t have diet equivalents, to find the closest thing to a diet version. Nearest Neighbor Nearest neighbor (NN) is often the most useful embedding operation. It finds things that are similar to the current embedding. In recommender systems, we can create a user embedding and find items that are most relevant to them. In a search engine, we can find a document that’s most similar to a search query. Nearest neighbor is a computationally expensive operation however. Performed naively it is O(N*K), where N is the number of items and K is the size of each embedding. However, in most cases when we need nearest neighbors, an approximation would suffice. If we recommend five items to a user, and one is technically the sixth closest item, the user probably won’t care. Approximate nearest neighbor (ANN) algorithms typically drop the complexity of a lookup to O(log(n)). Implementations of ANN There are many different algorithms to efficiently find approximate nearest neighbors, and many implementations of each of them. We’ll list a few of the most common algorithms and implementations below and break down how they work at a high level. Spotify’s Annoy In Spotify’s ANN implementation (Annoy), the embeddings are turned into a forest of trees. Each tree is built using . At every intermediate node in the tree, a random hyperplane is chosen, which divides the space into two subspaces. This hyperplane is chosen by sampling two points from the subset and taking the hyperplane equidistant from them. This is performed k times to generate a forest. Lookups are done via in-order traversal of the nearest tree. Annoy’s approach allows the index to be split into multiple static files, the index can be mapped in memory, and the number of trees can be tweaked to change speed and accuracy. random projections Here’s how Annoy breaks down the embeddings into an index of multiple trees via random projections. Locality Sensitive Hashing (LSH) Locality Sensitive Hashing (LSH) is another common approach. LSH employs a hash table and stores nearby points in their respective buckets. To create the index, LSH runs many hashing functions which place similar points in the same bucket. In doing so, LSH keeps points with large distances in separate buckets. To retrieve the nearest neighbor, the point in question is hashed, a lookup is done and the closest query point is returned. Some pros include a sub-linear run time, zero reliance on data distribution, and the ability to fine-tune accuracy with the existing data structure. Facebook’s FAISS and Hierarchical Navigable Small World Graphs (HNSW) Facebook’s implementation, FAISS, uses Hierarchical Navigable Small World Graphs (HNSW). HNSW typically performs well in accuracy and recall. It utilizes a hierarchical graph to create an average path towards certain areas. This graph has a hierarchical, transitive structure with a small average distance between nodes. HNSW traverses the graph and finds the closest adjacent node in each iteration and keeps track of the “best” neighbors seen so far. HNSW has a polylogarithmic time complexity (O(logN)). HNSW works by creating a hierarchical index to allow for faster nearest neighbor lookup. How embeddings are operationalized today Moving embeddings out of labs and into real-world systems, has surfaced real gaps in the current data infrastructure capabilities. For example, traditional databases and caches don’t support operations like nearest neighbor lookups. Specialized approximate nearest neighbor indices lack durable storage and other features required for full production use. MLOps systems lack dedicated methods to manage versioning, access, and training for embeddings. Modern ML systems need an embedding store: a database built from the ground up around the machine learning workflow with embeddings. Getting embeddings into production isn’t easy. The most common ways we’ve seen embeddings operationalized today are via Redis, Postgres, and S3 + Annoy/FAISS. We’ll cover how each of those implementations typically looks and the challenges with each. Redis Redis is a super-fast in-memory object-store. It makes storing and getting embeddings by key very fast. However, it doesn’t have any native embedding operations. It can’t do nearest-neighbor lookups and it can’t add or average vectors. All of this must be done on the model service. It also doesn’t fit cleanly in a typical MLOps workflow. It doesn’t support versioning, rolling back, or maintaining immutability. When training, the Redis client doesn’t automatically cache embeddings and this can cause unnecessary pressure and cost. It also doesn’t support partitioning embeddings and creating sub-indices. Postgres Postgres is far more versatile but far slower than Redis. Via plugins, it can be made to perform some of the vector operations manually. However, it does not have a performant nearest neighbor index. Also, having a Postgres lookup in the critical path of a model may add too much latency. Finally, Postgres doesn’t have a great way to cache embeddings on the client when training, causing very, very slow training. S3 files + Annoy/FAISS Annoy or FAISS are often used to operationalize embeddings when nearest-neighbor lookups are required. Both of these systems build indices of embeddings for approximate nearest neighbor operations, but they do not handle durable storage or other vector operations. Alone, they only solve one problem. Typically, companies will store their embeddings in S3 or a similar object storage service to fill in the rest of the gaps. They will load the embeddings and create the ANN index directly on the model when required. Updating embeddings becomes tough, and the system typically ends up with a lot of ad-hoc rules to manage versioning and rollbacks. FAISS and Annoy are great, but they need a full embedding store built around them. The Embedding Hub Machine learning systems that use an embedding need a type of data infrastructure that: - Stores their embeddings durably and with high availability - Allow for approximate nearest neighbor operations - Enable other operations like partitioning, sub-indices, and averaging - Manage versioning, access control, and rollbacks painlessly Embedding Hub sits atop your existing infrastructure. There are a few examples of companies like Pinterest who have built their own in-house, but the gap is still yet to be filled. Some startups have raised a few millions to build proprietary systems, and some database companies have tried to add nearest neighbors as a feature on top. There is a clear gap for an open-source embedding store, and we’ve decided to try to fill it. We’ve made our initial release of Embedding Hub on . Feel free to try it out, leave your comments and issues, and hand us a star! Github Conclusion Embeddings are a critical part of the data science toolkit, and continue to gain in popularity. Embeddings have allowed teams to break the state of the art in multiple disciplines from NLP to recommender systems. As they grow in popularity, a lot more focus will go into operationalizing them in real-world systems. We think that an embedding store will become a critical piece of machine learning infrastructure and that’s why we’ve open-sourced ours. GIve it a try, and if this sounds like an awesome project to work on full-time, we’re hiring! This article was first published on the featureform blog

FACTORIZE

factorize.ai

-- employees

Since n.d.

COMPANY RANKING

FACTORIZE

EVERGREEN INDEX #4039

Representing this company? Click here to claim and customize this page!

Factorize WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

FACTORIZE

factorize.ai

-- employees

Since n.d.

COMPANY RANKING

FACTORIZE

EVERGREEN INDEX #4039

RELATED COMPANIESRANK

STORIES ABOUT FACTORIZE

Representing this company? Click here to claim and customize this page!

Factorize WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!