Enkempass

enkempass

for in is an ECMAScript1 (ES1) feature. This variant of looping appeared in the first version of JavaScript in 1997. It is a very old and not very good way to loop through iterable objects. It is not recommended to use it in modern JavaScript. It is better to use instead. for...in for...of JavaScript uses prototype inheritance, and operator will return even to inherited properties. Let's take a look at the example: in true const obj = { a: 1, b: 2, c: 3 };
for (const key in obj) {
    console.log(key);
}
// Output:
// a
// b
// c Everything looks fine and predictable here. But what if we add a property to the prototype of the object? Let's add a property to the prototype of : d obj obj.__proto__.d = 4
for (const key in obj) {
    console.log(key);
}
// Output:
// a
// b
// c
// d In that example, we intentionally added a property to the object's prototype. But in real life, it is not that easy to control the prototype of the object. It is possible that some libraries will add a property to the prototype of the object. And in that case, will return that property as well. for...in There are a few ways how to avoid id. The first one is using method: hasOwnProperty const obj = { a: 1, b: 2, c: 3 };
obj.__proto__.d = 4

for (const key in obj) {
    if(obj.hasOwnProperty(key)) {
        console.log(key);
    }
}

// Output:
// a
// b
// c The second one is using method: Object.keys const obj = { a: 1, b: 2, c: 3 };
obj.__proto__.d = 4
for (const key of Object.keys(obj)) {
    console.log(key);
}

// Output:
// a
// b
// c As you can see, both options are not very convenient. In the last example, I used loop. And it's quite an often case then developers confuse and loops. They do look very similar, but the result is different. My suggestion will be to use for these purposes. is a collection of key-value pairs. It is a good alternative to the object. It is more convenient to use instead of the object. Let's take a look at the example: for...of for...in for...of Map Map Map const obj = { a: 1, b: 2, c: 3 };
obj.__proto__.d = 4

const m = new Map(Object.entries(obj));

for (const key of m.keys()) {
  console.log(key)
}
// Output:
// a
// b
// c .forEach() There are quite a few array methods in JavaScript. , , , to name a few. All of them have a distinct meaning. on the other hand, is a way to iterate over an array. It cannot transform the array. It cannot return a value. It can only produce side effects. .map .reduce .find .concat forEach Also, it's not possible to break the loop and pass an async function to it. According to some researchers, it's twice as slow as loop. So, my suggestion will be to use loop instead of . for for forEach labels Probably you have never used labels in JavaScript. But if you have, you should stop using them. Labels are a way to mark a block of code. It is possible to use or to break or continue the execution of the code. But it is not recommended to use them. It is better to use loop with or instead of labels. Let's take a look at the example: break continue for break continue let str = '';

loop1:
  for (let i = 0; i < 5; i++) {
    if (i === 1) {
      continue loop1;
    }

    if (i >= 4) {
      break loop1;
    }

    str = str + i;
  }

console.log(str);
// expected output: "023" In the previous example, removing labels will not change the result. That example was pretty simple. But in real life, it is not that easy to understand the code. It can get messy really fast. Object.create() There are two types of inheritance in JavaScript. The first one is prototype inheritance. The second one is class inheritance. is a way to create an object with a specific prototype. Let's take a look at the example: Object.create const obj = { a: 1, b: 2, c: 3 };

const obj2 = Object.create(obj);

console.log(obj2.a); // 1
console.log(obj2); // {} Prototype inheritance is quite an old way of inheritance. Also, it's very convenient because you don't need to create classes, interfaces, abstract classes, etc. All you need is to have a base object and create a new object with that base object as a prototype. But that flexibility comes with a price. It can result in unpredictable object structure. Let's take a look at the example: const obj = { a: 1, b: 2, c: 3 };

const obj2 = Object.create(obj);
obj2.d = 4;

const obj3 = Object.assign({}, obj2);

console.log(obj3.d); // 4

console.log(obj3.a); // undefined
console.log(obj2.a) // 1 So it's much safer to use class-based inheritance in JavaScript. A named function using the arrow operator Arrow functions had their debut in javascript in 2015. Arrow functions are a new way to write anonymous function expressions and are similar to lambda functions in some other programming languages. Nowadays, there are a couple of ways to create an anonymous function in javascript. // traditional way
(function (a) {
  console.log(a);
});

// arrow functions
a => console.log(a); Arrow functions have much more compact syntax and therefore are more readable in cases where you need to pass anonymous functions using array methods like , , etc. map filter reduce So they are great for anonymous functions. But what if you need to create a named function? Let's take a look at the example: function foo() {
  console.log('foo')
}
const bar = () => {
  console.log('bar')
}

foo() // foo
bar() // bar Both these functions will behave in the same way, but the creation of an arrow function is a bit more clunky. There is one advantage of an arrow function. It's possible to override with another value, and for it's not possible because it's defined as constant. foo bar foo = 'not a function and this is fine';
bar = 'not a function and this is not fine'; // TypeError: Assignment to constant variable. null There is one unavoidable data type in javascript. And it is undefined. Sometimes people use and interchangeably. But that approach is not correct. In JavaScript means a full absence of anything. For example, a function that returns nothing technically will return undefined. Missing function parameters will also have as their value. null undefined undefined undefined function foo(a) {
    console.log(a);
    a++;
}

const result = foo(); // undefined

console.log(result); // undefined is a special type that also means the absence of value. But means that values were intentionally set to nothing. null null In order to use default parameters in a function, you need to use instead of . Let's take a look at the example: undefined null function func(a = 1, b = 2) {
 return a + b; 
}

console.log(func(undefined, 3)) // 4

console.log(func(null, 3)) // 3 null converted to 0 is unavoidable. It will appear every time the value is missed intentionally or by mistake. But is not. JavaScript is a very flexible language, so it's a developer's responsibility to keep the code safe. Also, it's much easier to check the value for undefined and much harder for null. Undefined null console.log(typeof undefined) // undefined
console.log(typeof null) // object Conclusion We don't have silver bullets in software development. There are always trade-offs. We have to choose the best solution for our project. But we should always keep in mind that there are better ways to do things. And we should always try to use the best practices. I welcome you all to share your thoughts and suggestions in the comments below. Thank you for reading!

6 Things You Should Not Do in JavaScript

Software vendors are presenting a plethora of low-code software solutions to reduce the mundanity of software development. But, what are their actual advantages, how can they reduce, and also cause technical debt? As a tech journalist/writer, one of the standard press pitches I get is low code by companies shilling their specific solution(s). Low-code is a term twinned with everything from accelerating digital transformation during COVID-19 to faster deployment and a solution for developer shortages. What is low-code? Low-code is a means to develop applications with visual, intent-driven development tools reducing the need to create lines of custom code. Instead, various tools and platforms do the heavy lifting, including drag-and-drop interfaces, reusable code templates, process modeling, and SDKs, into the software development process. Thus, everything the no-code vendor thinks the user needs to build an app is already built into the tool. According to , the global low-code development platform (LCDP) market generated revenue of $10.3 billion in 2019. Driving the market is increased demand for business digitization, less dependency on IT professionals, and stringent government norms (like GDPR). The is fragmented and competitive, with the presence of a large number of players. Key players that hold significant market share include Oracle Corporation, Salesforce, Appian Corporation, Siemens AG, and Outsystems. Gartner asserts that By 2023, over 50% of medium to large enterprises will have adopted an LCDP as one of their strategic application platforms. market research low-code development platform market Not to be confused with no code You’ve probably heard of no-code, especially if you work in the Internet of Things, or IoT. It’s also characterized by user-friendly, drag-and-drop, visual interfaces and typically comes pre-configured with server-side and database applications or a basic interface that can connect with an appropriate database. Its underlying mission is to make app development accessible to those without any technical knowledge or training. I’m sure I’ve written a few headlines over the years with the moniker ‘Do you really want your CEO creating your apps?’ No-code can reduce the cost of employing developers, but it offers limitations in terms of customization. In other words, it’s a bit like comparing Canva to Indesign or more bespoke design programs. You could argue that a blogging/eCommerce platform like WordPress is a bridge between low and no code, it can be launched without any code but requires coding for anything beyond templated capabilities. There’s plenty of no-code examples on the market like Zapier, Airtable, and Webflow. What are the advantages of no-code app development platforms? The use of templates speeds up prototyping, testing, and deployment. Some claim that low-code app development is up to 20 times faster than traditional development, with the platform doing the coding, freeing the team to focus on design and customization. Speed: Developing low-code software makes it easy to create prototype apps and MVPs to market for valuable feedback with minimal effort. Easy MVPs: Low-code development breaks down the walls between business and tech without uttering the word DevOps. It offers a more straightforward way to collaborate on feature and design development that’s very accessible to non-devs who can provide input on design and functionality. This is particularly relevant in creating apps for niche audiences. For example, if you were making an app for construction businesses, working closely with sales or business development teams (who have a rich, detailed history of the specific industry and the end-user needs) can create a highly responsive, effective product. Breaks down silos: apps built using low-code development platforms (LCDP) are typically progressive web apps, deployable across various devices and operating systems. This paves the way for future omnichannel communications and offerings. The platform does the heavy lifting for security, maintenance, upgrades, cross-platform support etc. Frameworks and infrastructure are already done. Easy maintainability: simply put, low-code is about automating the mundane aspects of app development, freeing skilled developers to focus their energy on more mission-critical, challenging, and creating pursuits. Simplicity and less pressure on IT: Low-code frameworks offer effective integration between legacy, on-premises, and cloud-based applications. It's touted as particularly effective in industries where legacy systems are ripe such as enterprise resource planning, CRMs, banking, and HR. Unified Cloud and On-premise Applications: How does low code help avoid technical debt? Low code platforms are designed to manage updates through automation, reducing the overhead costs of running an application and relieving the dev team from managing technical debt. As a result, development cycles are faster, more streamlined, with less waste and maintainability. Developers spend less time or meaning there’s more time for adding new features or functionalities. refactoring code If you have a lot of technical debt and no visibility on it, try out the and s that allow engineers to bookmark code, create issues, and show data during sprint planning meetings. Stepsize VS Code JetBrains extension What are the downsides to low code? Forsyth Alexander suggests that low code platforms can be challenging to master quickly. He , asserts “Low-code tools can stop just shy of enabling the development of enterprise apps. Scalability, high quality, high performance, and other non-functional requirements are not always easy to meet with low-code, nor is it easy to change them with the platform.” Low-code development is symbolic of growth in business staff (not developers) driving the development process on a fundamental level. One of the critical tenants of low-code is an army (ok, maybe a deskful is more accurate) of “citizen developers”, folks who can be relied upon (and trained if required) to build apps with easy-low code interfaces. How CTOs should manage these people compared to the dev team is anyone’s guess, especially when it comes to practices like standups, planning, allocation of work etc. Having folks somewhere else in the building who do some of your team’s work and have their own role-specific tasks, responsibilities, and skills could become unwieldy if not well managed. No-code development (especially by non-devs) can conversely prevent shadow IT when done well and cause it without appropriate supervision. Technical debt can proliferate without appropriate safeguards to ensure standards such as security, compliance, and performance are met. According to Maxwell Flitton, the low-code movement can be a scam sharing the experience of its fans: “They’ve smugly brought all the hype and touted that they could build the new Facebook with a point and click site builder. However, when they want to add more features, they realize that their platform just doesn’t support it, and they’re begging a developer to “write a bit of code” to just add this one feature. Problem is, you can’t just write a “bit of code”, you have to rewrite the whole system. That’s right, the system that they have pointed and clicked away is close to worthless. They have locked themselves into a system that can be quickly replicated and restricts them from truly customizing their product. It just doesn’t hold any value.” Myles F. Suer shared based on a Twitter discussion around low code amongst CIOs. The discussion raised the vision of a future clogged where “the compound interest of tech debt created from low code development is going to be brutal as organizations try to maintain N# of microapps.” He asserts, his insights “Low code without integration, for this reason, is viewed as a tech debt expander.” Similarly, Jacob Goerz that low-code might not be solving the real business problem “If business users are empowered to build their own tools and can build them rapidly, we are trading one form of technical debt for another. For instance, traffic: if we improve roads, we do not get better traffic conditions, and we get more cars. By empowering business users to build applications, we do not solve technical debt challenges, we get more applications.” contends How to avoid this? One solution raised in the CIO discussion was governance and the teaching of low code in entrepreneurial programs in business schools. There’s also the need to look beyond the capabilities of an LCDP: “Security, governance, and testing are often an afterthought when an enterprise platform is being adopted. Many companies think a platform’s built-in capabilities are sufficient, and this can lead organizations that are implementing a low- or no-code platform to use manual testing and ad-hoc governance control, undermining employees’ ability to use the platforms and leading to misalignment on objectives”. However, it’s worth remembering the underlying aims of improving the life of developers. According to Maxwell, low-code is emblematic of shortcuts and open-source practices that developers have been creating to relieve the monotony of dull tasks forever and are increasingly offered through machine learning automation: “Login managers, database drivers, etc., are all being open-sourced and are free to download, enabling the developer to do things in a few lines of code. There are whole code libraries that you can just plugin, and providers like AWS and Google cloud offer one-click deployments, and this highlights the importance of the competition. If you have a low/no-code tech product, all we need is two mildly experienced software engineers and a couple of weeks, and they will smoke your product.” However low-code you go, you need to get on top of technical debt If you’re considering further exploration of low-code development, you need to measure and make efforts to reduce technical debt. offers a tool to help your team track and prioritize technical debt based on time lost, team morale, and business impact. It enables you to monitor and identify the existing debt in real-time without workflow disruptions. Stepsize Also published on: https://www.stepsize.com/blog/can-low-code-really-solve-the-problem-of-technical-debt

Is Low-Code the Solution to Technical Debt?

SQL vs NoSQL — What is better for you? Whether you have worked with databases, are about to work with one, or haven’t worked at all, this topic is something you should know irrespectively. You need to always think about how your data looks like first, how you are about to query it and what are your scalability requirements. Whether it's you choosing the perfect database for your personal project or it’s an organization, It is an important decision whether to pick a relational (SQL) or non-relational (NoSQL) data structure. This article is not about which type is better but about which type suits you for your current project requirements. Each one has clear advantages and disadvantages which must be kept in mind. If you just want to see the key-points mentioned in this article skip directly to the end. Now with that said, let’s get started What is SQL? SQL or Structured Query Language is pronounced as “ or sometimes as “ is a standard language to access and manipulate Relational Databases. S-Q-L” see-quel” It gives the users the power to create queries to INSERT, DELETE, UPDATE, SEARCH, ALTER, and to do a lot more with the Database. It is a very powerful language and is preferred by lots of developers as it can handle very complex queries as well. Common and Popular Examples of Relational Databases which use SQL are — MySQL Oracle SQLite Postgres MS-SQL A very Basic syntax pattern of SQL statement is — General--SELECT * FROM <TABLENAME> WHERE <ATTRIBUTE> = <VALUE> Practical--SELECT * FROM EMPLOYEE WHERE Salary > 90000 What is NoSQL? As the name suggests NoSQL is “ ” SQL, in other words, it is a non-relational database and is unstructured. Due to its unstructured nature, it is sometimes called as not UnQL It follows a database design which can take all sorts of data models. It mainly consists of key-value pairs, graphs, documents, etc. It is especially used to store large sets of distributed data. Common and Popular Examples of NoSQL Databases are — Mongo DB BigTable Cassandra Hbase Redis Information may be stored in many ways in this type of unstructured databases, but a widely followed approach is to store the information in format. JSON {role: "employee",salary: "90000"} Comparing SQL vs NoSQL Now that we have a basic understanding of what SQL and NoSQL database means, it would be much easier to compare the two types and figure out which one better suits your needs. Let’s go ahead and compare them now. Database Structure A SQL Database follows a which can have an and every data present inside the database is , it is basically used to manipulate . table like structure unlimited number of rows properly structured with Predefined Schemas Relational Databases Management Systems A NoSQL Database is a where the data is very . Unlike SQL it cannot have unlimited rows but it follows a and can store all sorts of data models with large distributed data in the form of . Distributed Database unstructured with Dynamic Schema Standard Schema Definition key-value pairs, graph databases, documents or wide-column stores Syntax SQL is a very powerful language and is very handy when it comes to access as well as manipulate not just simple but very as well. It uses to define and manipulate data. An example of this syntax is — complex queries SQL syntax SELECT * FROM STUDENT WHERE NAME = "Sayan" AND MARKS > 80; Whereas in NoSQL the queries are mainly focused on the and it is not a good option to manipulate complex queries. As there are different types of NoSQL databases available therefore it , the syntax differs from database to database. The most commonly used format to store data is collection of large documents doesn’t follow a single syntax base JSON. {name:"Sayan",marks:"85"} Hierarchy SQL Databases are stored in the form of tables consisting of n number of rows thus it a hierarchical approach rather all the data is present individually in each row under a specific attribute with its own significance. cannot follow Whereas NoSQL is a for hierarchical data storage as it stores the data in the form of in JSON format and thus serves as a better fit for large distributed data. very good fit key-value pairs Scalability A SQL Database is always . The meaning of this sentence is that if there is more load on the server then it . This will work only on a server. Thus by this, you may already be aware that this requires on a single server. vertically scalable can be handled by increasing the CPU, SSD, RAM, etc single more strain A NoSQL Database, on the other hand, is . This means that if there is excess load on the server then it will simply to handle the large traffic. It is simple and and prevents the server from being non-responsive. horizontally scalable add a few more servers very efficient Stability A SQL Database is and is always preferred when doing heavy duty transactions. The reason being it keeps the data very through . It follows Property which stands for more stable secure data integrity and atomicity ACID Atomicity, Consistency, Isolation, and Durability. Click to know more about ACID A NoSQL Database is than a SQL Database when there is on the server or even when performing . It follows which stands for less stable high load complex transactions Brewers CAP Theorem Consistency, Availability, and Partition Tolerance. Click to know more about Brewers CAP Theorem What is the right choice for your business? This decision completely depends on the business type and the ideation of the business. If you want to work with pre-defined structures and set schemas then SQL Databases is the correct choice for you. If you are building anything similar to an inventory monitoring system or accounting system then SQL Database is a better fit. But if you want to build an application with dynamic schema or if your business has a very rapid growth and no schema definition then NoSQL Databases is what you need. If you cannot determine the type of schema your application wants or you want to keep it changing from time to time like in case of mobile apps, analytics or content management then NoSQL is a better fit. Key-points in Short Thank you for reading this. I really appreciate all the effort. If you’re having any doubts comment below and I’ll try to answer them. Peace Out. ✌️

SQL vs NoSQL — What is better for you?

The missing manual to Twitter I wished I had Photo by on Tyler B Unsplash There are many good reasons why you might want to start using Twitter. lists some of the more important ones in “ ”. Rachel Thomas Making Peace with Personal Branding For me, the appeal is simple. I get to listen and occasionally talk to people doing amazing things in the field I care about (Machine Learning with a focus on Deep Learning). I also sometimes fantasize that what I write is helpful to others. Mostly that is just me dreaming things up. But Twitter is not without quirks and some things are not what they appear. Below I present the missing manual — the one I wished I had when I was starting out with the platform. Stay awhile and listen Are you already on Twitter? Or maybe you are thinking of signing up? These are the things you need to think about: What are you interested in? Any passions or hobbies? Are there any writers or otherwise accomplished people in your field of interest that you would like to learn more about? And then all you have to do is follow the people from #2. From now on they will be able to share their thoughts with you. They might also introduce you to other people you might find interesting. Isn’t this great? Never before have there existed an easy and open way such as this to tune in to what a person has to say. Using Twitter in this fashion has a lot of positives and avoids nearly all of its troublesome parts. But you are right to point out that Twitter at its core is a social network. There is value in speaking out and allowing other people to get to know you. And that is where problems start. Twitter made a lot of design decisions that make this really hard. Below I share the strategy I use. It works for me quite well. But before we go further let’s take a closer look at the platform itself. Understanding the nature of the beast If you are a human, chances are social networks have not been designed with you in mind. Let me be more specific. Over many years many billions of dollars have been poured into optimizing social networks for one thing and one thing only — maximizing profits. This goal often stands in stark opposition to maintaining the well-being of users. A lot has been written on this subject. You might for instance find it quite interesting to learn . why the social media bosses do not use social media Most of this boils down to the fact that keeping users happy and relaxed doesn’t drive engagement. And you want to be happy, right? You want to have peace of mind. And so we need to proceed with caution as we are about to start using a platform that has goals that are not aligned with what is good for us. So how do we start talking? Pick one theme Want your voice to make it through the noise and be heard? Pick a single theme to speak to. There is not enough time for a person who is interested in skateboarding to pick out that you have something interesting to say when your feed is flooded with pictures of your cat. Or your breakfast. This might just work though. ( ) source There will come a time for an occasional personal but only as people get to know you better. tweet Increasing your follower count 101 Why do you need more followers in the first place? Unless you are trying to sell something — in which case stay away from me — you don’t. Remember the times when not everyone wanted to be a celebrity? When people just wanted to meet people? You now open FB and everyone is The Most Successful Person Ever Living the Perfect Live®. That is just wrong. It is even worse with Twitter. The appeal to grow your audience is very strong. But it is a path that leads to suffering. Find a couple of people, maybe 3 or 4, who are doing something interesting. Start talking to them. Maybe they will talk back to you. Once you have 2–3 people who are happy to talk to you, you are set. Be content and thankful. You’ve made it. If a person you admire shares some of your work with their followers, or talks back to you, that is a cherry on top. And there are some genuinely good people out there who will help you be heard if you have something interesting to say. But this cannot be your goal. If you want to chase popularity, this will be high school all over again. You will go crazy. It’s async or nothing at all The first rule of social networks is to disable push notifications. Nothing should be able to interrupt your thoughts whenever it pleases. Especially if it has algorithms baked in to make you addicted. Bring it on Twitter. Is this all you got? But even if you disable push notifications the situation is still not straightforward. I could tell you all about B.F Skinner. The American scientist who claimed free will didn’t exist and who made animals do his bidding. It’s intriguing to what extent his work influenced the design choices of large social platforms. Instead let me refer you to an article by an insider who makes a great point. Care to learn ? how a slot machine was put in a billion pockets I am already hooked on social media. What can I do? It depends and sometimes it might be wise to seek professional help. If the use is moderate maybe you can hack the habit and change it into something useful. I often find myself taking the phone out of my pocket and either about to press or pressing the Twitter icon. But luckily I have a very slow mobile phone. Even if I do press on the icon, there is a second or two of white screen before the app loads. In that brief moment I reclaim my freedom. I got you Twitter. I quickly cancel out and press on Mendeley. This is an application that I use for organizing scientific papers I want to read. I take a deep breath. Congratulate myself. And I then enjoy reading something that is good for me. The point is not to not use . The point is to not use it all the time. social media Give yourself the time to think and breath. What is your Mendeley? Twitter as a side dish Twitter works best when you talk to people and exchange ideas. When you receive helpful advice and learn from others. But to put the advice to good use, you need to give it attention. And that will not happen while you have the Twitter client open. Or while you care whether someone liked what you wrote. More importantly, your loved ones might or might not be on Twitter. This doesn’t matter. You cannot look into their eyes through the screen of your phone. You cannot hug them or go for a walk with them. Fight for your attention and give it to the people and the things you love. Your ability to do so will not last forever. Ironically, you can find me on Twitter here . I mostly tweet about Machine Learning. If that is something that you find interesting, feel free to follow. Just make sure you don’t check your feed too often.

How to stop worrying and start tweeting

Authors:
(1) Hoon Kim, Beeble AI, and contributed equally to this work;
(2) Minje Jang, Beeble AI, and contributed equally to this work;
(3) Wonjun Yoon, Beeble AI, and contributed equally to this work;
(4) Jisoo Lee, Beeble AI, and contributed equally to this work;
(5) Donghyun Na, Beeble AI, and contributed equally to this work;
(6) Sanghyun Woo, New York University, and contributed equally to this work. Editor's Note: This is Part 10 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Table of Links Abstract and 1. Introduction
2. Related Work
3. SwitchLight and 3.1. Preliminaries
3.2. Problem Formulation
3.3. Architecture
3.4. Objectives
4. Multi-Masked Autoencoder Pre-training
5. Data
6. Experiments
7. Conclusion Appendix A. Implementation Details
B. User Study Interface
C. Video Demonstration
D. Additional Qualitative Results & References 7. Conclusion We introduce SwitchLight, an architecture based on Cook-Torrance rendering physics, enhanced with a selfsupervised pre-training framework. This co-designed approach significantly outperforms previous models. Our future plans include scaling the current model beyond images to encompass video and 3D data. We hope our proposal serve as a new foundational model for relighting tasks. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Hoon Kim, Beeble AI, and contributed equally to this work; (2) Minje Jang, Beeble AI, and contributed equally to this work; (3) Wonjun Yoon, Beeble AI, and contributed equally to this work; (4) Jisoo Lee, Beeble AI, and contributed equally to this work; (5) Donghyun Na, Beeble AI, and contributed equally to this work; (6) Sanghyun Woo, New York University, and contributed equally to this work. Authors: Authors: (1) Hoon Kim, Beeble AI, and contributed equally to this work; (2) Minje Jang, Beeble AI, and contributed equally to this work; (3) Wonjun Yoon, Beeble AI, and contributed equally to this work; (4) Jisoo Lee, Beeble AI, and contributed equally to this work; (5) Donghyun Na, Beeble AI, and contributed equally to this work; (6) Sanghyun Woo, New York University, and contributed equally to this work. Editor's Note: This is Part 10 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 10 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 10 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Editor's Note: This is Part 10 of 14 of a study introducing a method for improving how light and shadows can be applied to human portraits in digital images. Read the rest below. Table of Links Abstract and 1. Introduction 2. Related Work 3. SwitchLight and 3.1. Preliminaries 3.2. Problem Formulation 3.3. Architecture 3.4. Objectives 4. Multi-Masked Autoencoder Pre-training 5. Data 6. Experiments 7. Conclusion Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2. Related Work 3. SwitchLight and 3.1. Preliminaries 3. SwitchLight and 3.1. Preliminaries 3.2. Problem Formulation 3.2. Problem Formulation 3.3. Architecture 3.3. Architecture 3.4. Objectives 3.4. Objectives 4. Multi-Masked Autoencoder Pre-training 4. Multi-Masked Autoencoder Pre-training 5. Data 5. Data 6. Experiments 6. Experiments 7. Conclusion 7. Conclusion Appendix A. Implementation Details B. User Study Interface C. Video Demonstration D. Additional Qualitative Results & References A. Implementation Details A. Implementation Details B. User Study Interface B. User Study Interface C. Video Demonstration C. Video Demonstration D. Additional Qualitative Results & References D. Additional Qualitative Results & References 7. Conclusion We introduce SwitchLight, an architecture based on Cook-Torrance rendering physics, enhanced with a selfsupervised pre-training framework. This co-designed approach significantly outperforms previous models. Our future plans include scaling the current model beyond images to encompass video and 3D data. We hope our proposal serve as a new foundational model for relighting tasks. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

New AI Relighting Model Outperforms Previous Models

Vilnius, Lithuania, December 20th, 2024/Chainwire/--Europe’s leading cryptocurrency exchange, WhiteBit, concludes the year with notable achievements. The latest estimates reveal the company’s capitalization has reached $38.9 billion, while the annual trading volume surged to $2.7 trillion — a 200% increase compared to the previous year. These figures encompass all exchange markets (spot and futures), showcasing the company’s rapid growth and the increasing trust of its users. For context, WhiteBIT’s trading volume surpassed major players like Kraken, whose annual trading volume stood at $628 billion. WhiteBIT’s trading volume exceeds the GDP of countries like Italy or Canada and approaches the GDP of France, valued at $3 trillion. According to WhiteBIT Founder and CEO Volodymyr Nosov, this milestone reflects the exchange’s innovative strategies and strategic focus: “Reaching $2.7 trillion is the result of our dedicated teamwork and the trust of our clients. We remain committed to developing cutting-edge technologies tailored to the needs of both retail and institutional traders.” Top 2 in Traffic In the fall of 2024, WhiteBIT secured its position as the second-most visited crypto exchange, with 33 million visits to the platform. This achievement positions it just behind Binance, reflecting the steady interest in WhiteBIT’s services. Key Drivers of Growth One of the primary contributors to WhiteBIT’s record trading volume has been the onboarding of institutional clients. The exchange introduced a market maker program tailored to the requirements of large-scale traders. With low fees and rebates as high as -0.01%, the program provided conditions for high-frequency trading, drawing professional traders to the platform. Institutional loans also became a cornerstone for building trust within the professional community, enabling large-scale transactions. The platform’s technical infrastructure also contributed, with features such as colocation to reduce latency, sub-accounts for strategy diversification, and a robust API supporting its services for institutional clients. The introduction of a new affiliate program contributed by engaging communities of retail traders, supporting user acquisition, and enhancing trading activity, which played a role in WhiteBIT’s strong performance. Currently, the exchange serves over 1,300 institutional clients. Expanding Global Reach Today, over 30 million people worldwide use services and products from the WhiteBIT Group holding. These include the centralized exchange WhiteBIT, the crypto payment solution Whitepay, the decentralized exchange WhiteSwap, the NFT marketplace white.market, and the gaming platform Pocket Rocket. Additionally, WhiteBIT Group encompasses Whitechain, its proprietary blockchain, and WhiteBIT Coin (WBT), which has experienced 349.39% growth over the year. With 600+ trading pairs and more than 300 digital assets, WhiteBIT continues to strengthen its competitive edge in the global market. In a recent interview, CEO Volodymyr Nosov confirmed WhiteBIT’s plans to enter the US market. Commitment to Security WhiteBIT stands out as one of the most secure cryptocurrency exchanges globally. Ranked among top 5 in security by CER.live, it is the first crypto exchange to achieve the highest-level Cryptocurrency Security Standard (CCSS) certification. The platform also holds a Payment Card Industry Data Security Standard (PCI DSS) certification, safeguarding customer payment information, and has been recognized by Hacken for excellence in cybersecurity. WhiteBIT remains committed to continuous growth, solidifying its position in the international market, and expanding its impact on the global crypto ecosystem. About WhiteBIT WhiteBIT is the largest European centralized cryptocurrency exchange founded in Ukraine in 2018, offers over 600 trading pairs, 300 assets, and supports 9 fiat currencies. As part of the WhiteBIT Group holding, the platform serves more than 30 million customers globally. WhiteBIT collaborates with Visa, FACEIT, FC Barcelona, Trabzonspor, the Ukrainian national football team, and lifecell. The company is dedicated to driving the widespread adoption of blockchain technology in Ukraine and worldwide. Contact WhiteBIT PR Service pr@whitebit.con This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here Vilnius, Lithuania, December 20th, 2024/Chainwire/--Europe’s leading cryptocurrency exchange, WhiteBit, concludes the year with notable achievements. The latest estimates reveal the company’s capitalization has reached $38.9 billion, while the annual trading volume surged to $2.7 trillion — a 200% increase compared to the previous year. These figures encompass all exchange markets (spot and futures), showcasing the company’s rapid growth and the increasing trust of its users. reached reached surged surged For context, WhiteBIT’s trading volume surpassed major players like Kraken, whose annual trading volume stood at $628 billion. WhiteBIT’s trading volume exceeds the GDP of countries like Italy or Canada and approaches the GDP of France, valued at $3 trillion. exceeds exceeds According to WhiteBIT Founder and CEO Volodymyr Nosov, this milestone reflects the exchange’s innovative strategies and strategic focus: “Reaching $2.7 trillion is the result of our dedicated teamwork and the trust of our clients. We remain committed to developing cutting-edge technologies tailored to the needs of both retail and institutional traders.” “Reaching $2.7 trillion is the result of our dedicated teamwork and the trust of our clients. We remain committed to developing cutting-edge technologies tailored to the needs of both retail and institutional traders.” Top 2 in Traffic In the fall of 2024, WhiteBIT secured its position as the second-most visited crypto exchange, with 33 million visits to the platform. This achievement positions it just behind Binance, reflecting the steady interest in WhiteBIT’s services. secured secured Key Drivers of Growth One of the primary contributors to WhiteBIT’s record trading volume has been the onboarding of institutional clients . The exchange introduced a market maker program tailored to the requirements of large-scale traders. institutional clients institutional clients With low fees and rebates as high as -0.01%, the program provided conditions for high-frequency trading, drawing professional traders to the platform. Institutional loans also became a cornerstone for building trust within the professional community, enabling large-scale transactions. The platform’s technical infrastructure also contributed, with features such as colocation to reduce latency, sub-accounts for strategy diversification, and a robust API supporting its services for institutional clients. The introduction of a new affiliate program contributed by engaging communities of retail traders, supporting user acquisition, and enhancing trading activity, which played a role in WhiteBIT’s strong performance. Currently, the exchange serves over 1,300 institutional clients. affiliate program affiliate program Expanding Global Reach Today, over 30 million people worldwide use services and products from the WhiteBIT Group holding. These include the centralized exchange WhiteBIT, the crypto payment solution Whitepay, the decentralized exchange WhiteSwap, the NFT marketplace white.market, and the gaming platform Pocket Rocket. Additionally, WhiteBIT Group encompasses Whitechain, its proprietary blockchain, and WhiteBIT Coin (WBT), which has experienced 349.39% growth over the year. experienced experienced With 600+ trading pairs and more than 300 digital assets, WhiteBIT continues to strengthen its competitive edge in the global market. In a recent interview, CEO Volodymyr Nosov confirmed WhiteBIT’s plans to enter the US market. confirmed confirmed Commitment to Security WhiteBIT stands out as one of the most secure cryptocurrency exchanges globally. Ranked among top 5 in security by CER.live, it is the first crypto exchange to achieve the highest-level Cryptocurrency Security Standard (CCSS) certification. Ranked Ranked achieve achieve The platform also holds a Payment Card Industry Data Security Standard (PCI DSS) certification, safeguarding customer payment information, and has been recognized by Hacken for excellence in cybersecurity. WhiteBIT remains committed to continuous growth, solidifying its position in the international market, and expanding its impact on the global crypto ecosystem. About WhiteBIT WhiteBIT is the largest European centralized cryptocurrency exchange founded in Ukraine in 2018, offers over 600 trading pairs, 300 assets, and supports 9 fiat currencies. WhiteBIT WhiteBIT As part of the WhiteBIT Group holding, the platform serves more than 30 million customers globally. WhiteBIT collaborates with Visa, FACEIT, FC Barcelona, Trabzonspor, the Ukrainian national football team, and lifecell. The company is dedicated to driving the widespread adoption of blockchain technology in Ukraine and worldwide. Contact WhiteBIT PR Service pr@whitebit.con This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here here here

WhiteBIT’s Institutional Focus Drives Trading Volume To Record $2.7 Trillion In 2024

WhiteBIT se institusionele fokus dryf handelsvolume om $2,7 triljoen in 2024 aan te teken

التركيز المؤسسي لشركة WhiteBIT يدفع حجم التداول إلى تسجيل 2.7 تريليون دولار في عام 2024

Институционалният фокус на WhiteBIT кара обема на търговията да запише $2,7 трилиона през 2024 г.

হোয়াইটবিট-এর প্রাতিষ্ঠানিক ফোকাস 2024 সালে $2.7 ট্রিলিয়ন ডলার রেকর্ড করতে ট্রেডিং ভলিউম ড্রাইভ করে

Der institutionelle Fokus von WhiteBIT lässt das Handelsvolumen im Jahr 2024 auf 2,7 Billionen US-Dollar ansteigen

El enfoque institucional de WhiteBIT impulsa el volumen de operaciones a un récord de 2,7 billones de dólares en 2024

המיקוד המוסדי של WhiteBIT מניע את נפח המסחר לשיא של 2.7 טריליון דולר בשנת 2024

WhiteBITの機関投資家向け投資により、2024年の取引量は2.7兆ドルに達する見込み

WhiteBIT-тің институционалдық фокусы сауда көлемін 2024 жылы 2,7 триллион долларға түсіреді

WhiteBIT institucionālais fokuss liek tirdzniecības apjomam 2024. gadā reģistrēt 2,7 triljonus USD

WhiteBIT:s institutionella fokus driver handelsvolymen till 2,7 biljoner dollar 2024

Ang Institusyonal na Focus ng WhiteBIT ay Nagtutulak sa Dami ng Trading Upang Magtala ng $2.7 Trilyon Noong 2024

Authors:
(1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors;
(2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors;
(3) Sahal Shaji, Mohamed bin Zayed University of AI;
(4) Abdelrahman Shaker, Mohamed bin Zayed University of AI;
(5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University;
(6) Hisham Cholakkal, Mohamed bin Zayed University of AI;
(7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University;
(8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University;
(9) Ming-Hsuan Yang, University of California - Merced and Google Research;
(10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 2 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction
2. Related Work
3. Method
4. Data Annotation Pipeline
5. Experiments
6. Conclusion and References Supplementary Material (Part 1) A. Additional Implementation Details
B. Additional Downstream Tasks
C. Additional Qualitative Results Supplementary Material (Part 2) D. Dataset Visualization
E. Limitations and Future Work
F. Ethics and Societal Impact 2. Related Work LMMs provide a versatile interface for a diverse array of tasks, encompassing language and vision. Prominent models such as BLIP-2 [24], LLaVA [29], InstructBLIP [6] and MiniGPT-4 [61] first conduct image-text feature alignment followed by instruction tuning. Other representative works include Otter [22], mPLUG-Owl [52], LLaMaAdapter [56], Video-ChatGPT [32], InternGPT [31]. However, these approaches lack region-specific understanding. Recent works like Kosmos-2 [35], Shikra [5], GPT4RoI [57], VisionLLM [44], Ferret [53] and All-Seeing [45] aim to allow region-specific conversation. Some methods [5, 35, 45, 53] input location bins and bounding boxes with image data for region-level understanding, relying on the LLM exclusively for interpreting these regions. GPT4RoI advances this by using spatial boxes and RoI-aligned features for input and training on region-text pairs. BuboGPT [59] utilizes an off-the-shelf grounding model [30] and matches the groundings with the language response. In contrast, LISA [21] utilizes embeddings from the vision language model and the SAM [18] decoder to generate output segmentation masks. However, LISA cannot comprehend specific image regions or handle multiple instances. To classify the LMM landscape, methods can be partitioned into four distinct categories (see Tab. 1 - separated via dotted lines). The first encompasses models effective in textual responses but lacking in region-specific capabilities [6, 8, 22, 29, 51, 52, 61]. In contrast, among models that handle region inputs or offer visual grounding, three more categories emerge. The first of these incorporates external vision modules [31, 59], and the next relies exclusively on LMMs for region understanding [5, 35, 36, 44]. The last category combines specialized vision modules with LMMs, trained end-to-end for a comprehensive understanding of regions [21, 45, 57]. Our approach belongs to the last category and distinctly offers pixel-level grounding together with multi-turn conversations and the flexibility to operate on both input images and specific regions. Further, we provide large-scale instance-level grounded visual understanding dataset that allows generalizability of GLaMM to multiple vision-language tasks. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Authors: Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 2 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 2 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 2 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 2 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction 2. Related Work 3. Method 4. Data Annotation Pipeline 5. Experiments 6. Conclusion and References Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method 3. Method 4. Data Annotation Pipeline 4. Data Annotation Pipeline 5. Experiments 5. Experiments 6. Conclusion and References 6. Conclusion and References Supplementary Material (Part 1) Supplementary Material (Part 1) A. Additional Implementation Details B. Additional Downstream Tasks C. Additional Qualitative Results A. Additional Implementation Details A. Additional Implementation Details B. Additional Downstream Tasks B. Additional Downstream Tasks C. Additional Qualitative Results C. Additional Qualitative Results Supplementary Material (Part 2) Supplementary Material (Part 2) D. Dataset Visualization E. Limitations and Future Work F. Ethics and Societal Impact D. Dataset Visualization D. Dataset Visualization E. Limitations and Future Work E. Limitations and Future Work F. Ethics and Societal Impact F. Ethics and Societal Impact 2. Related Work LMMs provide a versatile interface for a diverse array of tasks, encompassing language and vision. Prominent models such as BLIP-2 [24], LLaVA [29], InstructBLIP [6] and MiniGPT-4 [61] first conduct image-text feature alignment followed by instruction tuning. Other representative works include Otter [22], mPLUG-Owl [52], LLaMaAdapter [56], Video-ChatGPT [32], InternGPT [31]. However, these approaches lack region-specific understanding. Recent works like Kosmos-2 [35], Shikra [5], GPT4RoI [57], VisionLLM [44], Ferret [53] and All-Seeing [45] aim to allow region-specific conversation. Some methods [5, 35, 45, 53] input location bins and bounding boxes with image data for region-level understanding, relying on the LLM exclusively for interpreting these regions. GPT4RoI advances this by using spatial boxes and RoI-aligned features for input and training on region-text pairs. BuboGPT [59] utilizes an off-the-shelf grounding model [30] and matches the groundings with the language response. In contrast, LISA [21] utilizes embeddings from the vision language model and the SAM [18] decoder to generate output segmentation masks. However, LISA cannot comprehend specific image regions or handle multiple instances. To classify the LMM landscape, methods can be partitioned into four distinct categories (see Tab. 1 - separated via dotted lines). The first encompasses models effective in textual responses but lacking in region-specific capabilities [6, 8, 22, 29, 51, 52, 61]. In contrast, among models that handle region inputs or offer visual grounding, three more categories emerge. The first of these incorporates external vision modules [31, 59], and the next relies exclusively on LMMs for region understanding [5, 35, 36, 44]. The last category combines specialized vision modules with LMMs, trained end-to-end for a comprehensive understanding of regions [21, 45, 57]. Our approach belongs to the last category and distinctly offers pixel-level grounding together with multi-turn conversations and the flexibility to operate on both input images and specific regions. Further, we provide large-scale instance-level grounded visual understanding dataset that allows generalizability of GLaMM to multiple vision-language tasks. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

This New AI Model Is Excelling in Understanding and Interacting with Images

Authors:
(1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors;
(2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors;
(3) Sahal Shaji, Mohamed bin Zayed University of AI;
(4) Abdelrahman Shaker, Mohamed bin Zayed University of AI;
(5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University;
(6) Hisham Cholakkal, Mohamed bin Zayed University of AI;
(7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University;
(8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University;
(9) Ming-Hsuan Yang, University of California - Merced and Google Research;
(10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction
2. Related Work
3. Method
4. Data Annotation Pipeline
5. Experiments
6. Conclusion and References Supplementary Material (Part 1) A. Additional Implementation Details
B. Additional Downstream Tasks
C. Additional Qualitative Results Supplementary Material (Part 2) D. Dataset Visualization
E. Limitations and Future Work
F. Ethics and Societal Impact Abstract Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations. 1. Introduction Fueled by the generative AI wave, Large Multimodal Models (LMMs) have emerged as a pivotal advancement, bridging the gap between vision and language tasks [2]. Initial efforts like [6, 8, 22, 29, 52, 61] demonstrate effective textual responses based on input images. Although these models are sophisticated, they cannot still ground their responses in the visual context. Such grounding is crucial for advanced applications like detailed visual understanding, interactive embodied agents, and localized content manipulation. Recent efforts have started to address this limitation by enabling models to process user-defined regions specified via bounding boxes [5, 31, 35, 36, 57]. A few recent works have explored grounded text response generation [5, 21, 35, 59] but do not provide detailed pixel-level groundings. Parallel to these, efforts have been made in the referring segmentation literature to ground textual descriptions in natural images [21]. However, they are limited to grounding a single object and cannot engage in natural, coherent conversations, thereby restricting their practical applicability in interactive tasks that demand a deep understanding of both visual and textual content. To address these limitations of existing works, we introduce Grounding LMM (GLaMM), that simultaneously provides in-depth region understanding, pixel-level groundings, and conversational abilities through an end-to-end training approach (see Fig. 1 and Tab. 1). To address the lack of benchmarks for visually grounded conversations, we introduce the novel task of Grounded Conversation Generation (GCG). The GCG task aims to produce natural language responses interleaved with object segmentation masks. This challenging task unifies several existing tasks in computer vision that are typically treated in isolation, i.e., referring expression segmentation, image and region-level captioning, phrase grounding, and vision-language conversations. Thereby, our unified model and proposed pretraining dataset can effectively transfer to several downstream tasks (referring expression segmentation, region-level captioning, image captioning, and conversational-style QA). We present GLaMM as the first model specifically designed for this challenging task. Unlike prior works, GLaMM can work with both textual and visual prompts and can generate visually grounded outputs, thus offering a versatile user experience. Detailed region-level understanding requires the laborious process of collecting large-scale annotations for image regions. We propose an automated pipeline to annotate the large-scale Grounding-anything Dataset (GranD) to alleviate the manual labeling effort. Leveraging the automated pipeline with dedicated verification steps, GranD comprises 7.5M unique concepts anchored in 810M regions, each with a segmentation mask. Using state-ofthe-art vision and language models, the dataset annotates SAM [18] images through a multi-level hierarchical scheme that enhances annotation quality. With 11M images, 84M referring expressions, and 33M grounded captions, GranD sets a new benchmark in comprehensiveness. In addition to the automatically generated dataset for the GCG, we provide the first high-quality dataset for grounded conversations obtained by revamping the existing manually annotated datasets [16, 37, 49] for GCG using GPT-4 [34] incontext learning. We refer to the high-quality dataset as GranDf , denoting its suitability for fine-tuning. Our work has three main contributions: • We present GLaMM, the first model capable of generating natural language responses seamlessly integrated with object segmentation masks. Unlike existing models, GLaMM accommodates textual and visual prompts, facilitating enhanced multimodal user interaction. • Recognizing the lack of standardized benchmarks for visually grounded conversations, we propose the new Grounded Conversation Generation (GCG) task. We also introduce a comprehensive evaluation protocol to measure the efficacy of models for GCG that unifies multiple isolated tasks, filling a significant gap in the literature. • To facilitate model training and evaluation, we create Grounding-anything Dataset (GranD), a large-scale densely annotated dataset. Developed using an automatic annotation pipeline and verification criteria, it encompasses 7.5M unique concepts grounded in 810M regions. Additionally, we propose GranDf , a high-quality dataset explicitly designed for the GCG task finetuning, by repurposing existing open-source datasets. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Authors: Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 1 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction 2. Related Work 3. Method 4. Data Annotation Pipeline 5. Experiments 6. Conclusion and References Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method 3. Method 4. Data Annotation Pipeline 4. Data Annotation Pipeline 5. Experiments 5. Experiments 6. Conclusion and References 6. Conclusion and References Supplementary Material (Part 1) Supplementary Material (Part 1) A. Additional Implementation Details B. Additional Downstream Tasks C. Additional Qualitative Results A. Additional Implementation Details A. Additional Implementation Details B. Additional Downstream Tasks B. Additional Downstream Tasks C. Additional Qualitative Results C. Additional Qualitative Results Supplementary Material (Part 2) Supplementary Material (Part 2) D. Dataset Visualization E. Limitations and Future Work F. Ethics and Societal Impact D. Dataset Visualization D. Dataset Visualization E. Limitations and Future Work E. Limitations and Future Work F. Ethics and Societal Impact F. Ethics and Societal Impact Abstract Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations. Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations. 1. Introduction Fueled by the generative AI wave, Large Multimodal Models (LMMs) have emerged as a pivotal advancement, bridging the gap between vision and language tasks [2]. Initial efforts like [6, 8, 22, 29, 52, 61] demonstrate effective textual responses based on input images. Although these models are sophisticated, they cannot still ground their responses in the visual context. Such grounding is crucial for advanced applications like detailed visual understanding, interactive embodied agents, and localized content manipulation. Recent efforts have started to address this limitation by enabling models to process user-defined regions specified via bounding boxes [5, 31, 35, 36, 57]. A few recent works have explored grounded text response generation [5, 21, 35, 59] but do not provide detailed pixel-level groundings. Parallel to these, efforts have been made in the referring segmentation literature to ground textual descriptions in natural images [21]. However, they are limited to grounding a single object and cannot engage in natural, coherent conversations, thereby restricting their practical applicability in interactive tasks that demand a deep understanding of both visual and textual content. To address these limitations of existing works, we introduce Grounding LMM (GLaMM), that simultaneously provides in-depth region understanding, pixel-level groundings, and conversational abilities through an end-to-end training approach (see Fig. 1 and Tab. 1). To address the lack of benchmarks for visually grounded conversations, we introduce the novel task of Grounded Conversation Generation (GCG). The GCG task aims to produce natural language responses interleaved with object segmentation masks. This challenging task unifies several existing tasks in computer vision that are typically treated in isolation, i.e., referring expression segmentation, image and region-level captioning, phrase grounding, and vision-language conversations. Thereby, our unified model and proposed pretraining dataset can effectively transfer to several downstream tasks (referring expression segmentation, region-level captioning, image captioning, and conversational-style QA). We present GLaMM as the first model specifically designed for this challenging task. Unlike prior works, GLaMM can work with both textual and visual prompts and can generate visually grounded outputs, thus offering a versatile user experience. Detailed region-level understanding requires the laborious process of collecting large-scale annotations for image regions. We propose an automated pipeline to annotate the large-scale Grounding-anything Dataset (GranD) to alleviate the manual labeling effort. Leveraging the automated pipeline with dedicated verification steps, GranD comprises 7.5M unique concepts anchored in 810M regions, each with a segmentation mask. Using state-ofthe-art vision and language models, the dataset annotates SAM [18] images through a multi-level hierarchical scheme that enhances annotation quality. With 11M images, 84M referring expressions, and 33M grounded captions, GranD sets a new benchmark in comprehensiveness. In addition to the automatically generated dataset for the GCG, we provide the first high-quality dataset for grounded conversations obtained by revamping the existing manually annotated datasets [16, 37, 49] for GCG using GPT-4 [34] incontext learning. We refer to the high-quality dataset as GranDf , denoting its suitability for fine-tuning. Our work has three main contributions: • We present GLaMM, the first model capable of generating natural language responses seamlessly integrated with object segmentation masks. Unlike existing models, GLaMM accommodates textual and visual prompts, facilitating enhanced multimodal user interaction. • Recognizing the lack of standardized benchmarks for visually grounded conversations, we propose the new Grounded Conversation Generation (GCG) task. We also introduce a comprehensive evaluation protocol to measure the efficacy of models for GCG that unifies multiple isolated tasks, filling a significant gap in the literature. • To facilitate model training and evaluation, we create Grounding-anything Dataset (GranD), a large-scale densely annotated dataset. Developed using an automatic annotation pipeline and verification criteria, it encompasses 7.5M unique concepts grounded in 810M regions. Additionally, we propose GranDf , a high-quality dataset explicitly designed for the GCG task finetuning, by repurposing existing open-source datasets. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Researchers in UAE Create AI That Can Describe Images in Perfect Detail

Navorsers in die VAE skep KI wat beelde in perfekte detail kan beskryf

Forskere i UAE skaber AI, der kan beskrive billeder i perfekte detaljer

Investigadores de los Emiratos Árabes Unidos crean una IA que puede describir imágenes con perfecto detalle

Des chercheurs des Émirats arabes unis créent une IA capable de décrire des images avec une précision parfaite

חוקרים באיחוד האמירויות יוצרים בינה מלאכותית שיכולה לתאר תמונות בפירוט מושלם

Chèchè nan UAE kreye AI ki ka dekri imaj an detay pafè

UAEの研究者が画像を完璧に詳細に描写できるAIを開発

ນັກຄົ້ນຄວ້າໃນ UAE ສ້າງ AI ທີ່ສາມາດອະທິບາຍຮູບພາບໃນລາຍລະອຽດທີ່ສົມບູນແບບ

Истражувачите во ОАЕ создаваат вештачка интелигенција што може да ги опише сликите со совршени детали

Арабын Нэгдсэн Эмират улсын судлаачид дүрсийг нарийвчлан дүрсэлж чадах хиймэл оюун ухаан бүтээжээ

Abashakashatsi bo muri UAE barema AI ishobora gusobanura amashusho muburyo burambuye

Cilmi-baarayaasha UAE waxay abuuraan AI Kaas oo ku sifayn kara muuqaallada si faahfaahsan

Table of Links Abstract and 1 Introduction 2 Related Work 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 3 Hierspeech++ and 3.1 Speech Representations 3.2 Hierarchical Speech Synthesizer 3.3 Text-to-Vec 3.4 Speech Super-resolution 3.5 Model Architecture 4 Speech Synthesis Tasks 4.1 Voice Conversion and 4.2 Text-to-Speech 4.3 Style Prompt Replication 5 Experiment and Result, and Dataset 5.2 Preprocessing and 5.3 Training 5.4 Evaluation Metrics 5.5 Ablation Study 5.6 Zero-shot Voice Conversion 5.7 High-diversity but High-fidelity Speech Synthesis 5.8 Zero-shot Text-to-Speech 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.10 Speech Super-resolution 5.11 Additional Experiments with Other Baselines 6 Limitation and Quick Fix 7 Conclusion, Acknowledgement and References 3.3 Text-to-Vec For TTS, we introduce a text-to-vec (TTV) model that generates a semantic representation and F0 from a text sequence. Following VITS [35], we utilize a variational autoencoder and a monotonic alignment search (MAS) to align the text and speech internally, as shown in Fig 4. We replace the linear spectrogram with a self-supervised speech representation for the input of posterior encoder, and we reconstruct the same self-supervised speech representation for the output of TTV. Furthermore, we predict a F0 with four× larger resolutions than the self-supervised speech representation. We use a text sequence and prosody prompt as conditional information to generate a self-supervised speech representation of the data. We utilize a prosody conditional text representation as the prior information. A prosody style representation is extracted from the full-length input speech as a global style embedding. Owing to the semantic information of self supervised speech representation, we can transfer a prosody style in the TTV framework which is almost irrelevant to the voice style. To increase the linguistic capacity of a semantic representation, a latent representation is fed to the phoneme encoder, and the connectionist temporal classification (CTC) loss is minimized. We found that this could improve text speech alignment by significantly decreasing the CER and WER of synthetic speech. Furthermore, we use a Transformer based normalizing flow with AdaLN-Zero for better prosody adaptation. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors:
(1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 3 Hierspeech++ and 3.1 Speech Representations 3 Hierspeech++ and 3.1 Speech Representations 3.2 Hierarchical Speech Synthesizer 3.2 Hierarchical Speech Synthesizer 3.3 Text-to-Vec 3.3 Text-to-Vec 3.4 Speech Super-resolution 3.4 Speech Super-resolution 3.5 Model Architecture 3.5 Model Architecture 4 Speech Synthesis Tasks 4.1 Voice Conversion and 4.2 Text-to-Speech 4.1 Voice Conversion and 4.2 Text-to-Speech 4.3 Style Prompt Replication 4.3 Style Prompt Replication 5 Experiment and Result, and Dataset 5 Experiment and Result, and Dataset 5.2 Preprocessing and 5.3 Training 5.2 Preprocessing and 5.3 Training 5.4 Evaluation Metrics 5.4 Evaluation Metrics 5.5 Ablation Study 5.5 Ablation Study 5.6 Zero-shot Voice Conversion 5.6 Zero-shot Voice Conversion 5.7 High-diversity but High-fidelity Speech Synthesis 5.7 High-diversity but High-fidelity Speech Synthesis 5.8 Zero-shot Text-to-Speech 5.8 Zero-shot Text-to-Speech 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.10 Speech Super-resolution 5.10 Speech Super-resolution 5.11 Additional Experiments with Other Baselines 5.11 Additional Experiments with Other Baselines 6 Limitation and Quick Fix 6 Limitation and Quick Fix 7 Conclusion, Acknowledgement and References 7 Conclusion, Acknowledgement and References 3.3 Text-to-Vec For TTS, we introduce a text-to-vec (TTV) model that generates a semantic representation and F0 from a text sequence. Following VITS [35], we utilize a variational autoencoder and a monotonic alignment search (MAS) to align the text and speech internally, as shown in Fig 4. We replace the linear spectrogram with a self-supervised speech representation for the input of posterior encoder, and we reconstruct the same self-supervised speech representation for the output of TTV. Furthermore, we predict a F0 with four× larger resolutions than the self-supervised speech representation. We use a text sequence and prosody prompt as conditional information to generate a self-supervised speech representation of the data. We utilize a prosody conditional text representation as the prior information. A prosody style representation is extracted from the full-length input speech as a global style embedding. Owing to the semantic information of self supervised speech representation, we can transfer a prosody style in the TTV framework which is almost irrelevant to the voice style. To increase the linguistic capacity of a semantic representation, a latent representation is fed to the phoneme encoder, and the connectionist temporal classification (CTC) loss is minimized. We found that this could improve text speech alignment by significantly decreasing the CER and WER of synthetic speech. Furthermore, we use a Transformer based normalizing flow with AdaLN-Zero for better prosody adaptation. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors: (1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author. Authors: Authors: (1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author.

A Text-To-Vec Model That Can Generate A Semantic Representation and F0 From A Text Sequence

Table of Links Abstract and 1 Introduction 2 Background and 2.1 Transformer-Based Large Language Models 2.2 LLM Service & Autoregressive Generation 2.3 Batching Techniques for LLMs 3 Memory Challenges in LLM Serving 3.1 Memory Management in Existing Systems 4 Method and 4.1 PagedAttention 4.2 KV Cache Manager 4.3 Decoding with PagedAttention and vLLM 4.4 Application to Other Decoding Scenarios 4.5 Scheduling and Preemption 4.6 Distributed Execution 5 Implementation 6 Evaluation and 6.1 Experimental Setup 6.2 Basic Sampling 6.3 Parallel Sampling and Beam Search 6.4 Shared prefix 6.5 Chatbot 7 Ablation Studies 8 Discussion 9 Related Work 10 Conclusion, Acknowledgement and References Abstract High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM’s source code is publicly available at https://github.com/vllm-project/vllm. 1 Introduction The emergence of large language models (LLMs) like GPT [5, 37] and PaLM [9] have enabled new applications such as programming assistants [6, 18] and universal chatbots [19, 35] that are starting to profoundly impact our work and daily routines. Many cloud companies [34, 44] are racing to provide these applications as hosted services. However, running these applications is very expensive, requiring a large number of hardware accelerators such as GPUs. According to recent estimates, processing an LLM request can be 10× more expensive than a traditional keyword query [43]. Given these high costs, increasing the throughput—and hence reducing the cost per request—of LLM serving systems is becoming more important. At the core of LLMs lies an autoregressive Transformer model [53]. This model generates words (tokens), one at a time, based on the input (prompt) and the previous sequence of the output’s tokens it has generated so far. For each request, this expensive process is repeated until the model outputs a termination token. This sequential generation process makes the workload memory-bound, underutilizing the computation power of GPUs and limiting the serving throughput. Improving the throughput is possible by batching multiple requests together. However, to process many requests in a batch, the memory space for each request should be efficiently managed. For example, Fig. 1 (left) illustrates the memory distribution for a 13B-parameter LLM on an NVIDIA A100 GPU with 40GB RAM. Approximately 65% of the memory is allocated for the model weights, which remain static during serving. Close to 30% of the memory is used to store the dynamic states of the requests. For Transformers, these states consist of the key and value tensors associated with the attention mechanism, commonly referred to as KV cache [41], which represent the context from earlier tokens to generate new output tokens in sequence. The remaining small percentage of memory is used for other data, including activations – the ephemeral tensors created when evaluating the LLM. Since the model weights are constant and the activations only occupy a small fraction of the GPU memory, the way the KV cache is managed is critical in determining the maximum batch size. When managed inefficiently, the KV cache memory can significantly limit the batch size and consequently the throughput of the LLM, as illustrated in Fig. 1 (right). In this paper, we observe that existing LLM serving systems [31, 60] fall short of managing the KV cache memory efficiently. This is mainly because they store the KV cache of a request in contiguous memory space, as most deep learning frameworks [33, 39] require tensors to be stored in contiguous memory. However, unlike the tensors in the traditional deep learning workloads, the KV cache has unique characteristics: it dynamically grows and shrinks over time as the model generates new tokens, and its lifetime and length are not known a priori. These characteristics make the existing systems’ approach significantly inefficient in two ways: First, the existing systems [31, 60] suffer from internal and external memory fragmentation. To store the KV cache of a request in contiguous space, they pre-allocate a contiguous chunk of memory with the request’s maximum length (e.g., 2048 tokens). This can result in severe internal fragmentation, since the request’s actual length can be much shorter than its maximum length (e.g., Fig. 11). Moreover, even if the actual length is known a priori, the pre-allocation is still inefficient: As the entire chunk is reserved during the request’s lifetime, other shorter requests cannot utilize any part of the chunk that is currently unused. Besides, external memory fragmentation can also be significant, since the preallocated size can be different for each request. Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems. Second, the existing systems cannot exploit the opportunities for memory sharing. LLM services often use advanced decoding algorithms, such as parallel sampling and beam search, that generate multiple outputs per request. In these scenarios, the request consists of multiple sequences that can partially share their KV cache. However, memory sharing is not possible in the existing systems because the KV cache of the sequences is stored in separate contiguous spaces. To address the above limitations, we propose PagedAttention, an attention algorithm inspired by the operating system’s (OS) solution to memory fragmentation and sharing: virtual memory with paging. PagedAttention divides the request’s KV cache into blocks, each of which can contain the attention keys and values of a fixed number of tokens. In PagedAttention, the blocks for the KV cache are not necessarily stored in contiguous space. Therefore, we can manage the KV cache in a more flexible way as in OS’s virtual memory: one can think of blocks as pages, tokens as bytes, and requests as processes. This design alleviates internal fragmentation by using relatively small blocks and allocating them on demand. Moreover, it eliminates external fragmentation as all blocks have the same size. Finally, it enables memory sharing at the granularity of a block, across the different sequences associated with the same request or even across the different requests. In this work, we build vLLM, a high-throughput distributed LLM serving engine on top of PagedAttention that achieves near-zero waste in KV cache memory. vLLM uses block-level memory management and preemptive request scheduling that are co-designed with PagedAttention. vLLM supports popular LLMs such as GPT [5], OPT [62], and LLaMA [52] with varying sizes, including the ones exceeding the memory capacity of a single GPU. Our evaluations on various models and workloads show that vLLM improves the LLM serving throughput by 2-4× compared to the state-of-the-art systems [31, 60], without affecting the model accuracy at all. The improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms (§4.3). In summary, we make the following contributions: • We identify the challenges in memory allocation in serving LLMs and quantify their impact on serving performance. • We propose PagedAttention, an attention algorithm that operates on KV cache stored in non-contiguous paged memory, which is inspired by the virtual memory and paging in OS. • We design and implement vLLM, a distributed LLM serving engine built on top of PagedAttention. • We evaluate vLLM on various scenarios and demonstrate that it substantially outperforms the previous state-of-theart solutions such as FasterTransformer [31] and Orca [60]. This paper is available on arxiv under CC BY 4.0 DEED license. Authors:
(1) Woosuk Kwon, UC Berkeley with Equal contribution;
(2) Zhuohan Li, UC Berkeley with Equal contribution;
(3) Siyuan Zhuang, UC Berkeley;
(4) Ying Sheng, UC Berkeley and Stanford University;
(5) Lianmin Zheng, UC Berkeley;
(6) Cody Hao Yu, Independent Researcher;
(7) Cody Hao Yu, Independent Researcher;
(8) Joseph E. Gonzalez, UC Berkeley;
(9) Hao Zhang, UC San Diego;
(10) Ion Stoica, UC Berkeley. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and 2.1 Transformer-Based Large Language Models 2 Background and 2.1 Transformer-Based Large Language Models 2.2 LLM Service & Autoregressive Generation 2.2 LLM Service & Autoregressive Generation 2.3 Batching Techniques for LLMs 2.3 Batching Techniques for LLMs 3 Memory Challenges in LLM Serving 3 Memory Challenges in LLM Serving 3.1 Memory Management in Existing Systems 3.1 Memory Management in Existing Systems 4 Method and 4.1 PagedAttention 4 Method and 4.1 PagedAttention 4.2 KV Cache Manager 4.2 KV Cache Manager 4.3 Decoding with PagedAttention and vLLM 4.3 Decoding with PagedAttention and vLLM 4.4 Application to Other Decoding Scenarios 4.4 Application to Other Decoding Scenarios 4.5 Scheduling and Preemption 4.5 Scheduling and Preemption 4.6 Distributed Execution 4.6 Distributed Execution 5 Implementation 5 Implementation 6 Evaluation and 6.1 Experimental Setup 6 Evaluation and 6.1 Experimental Setup 6.2 Basic Sampling 6.2 Basic Sampling 6.3 Parallel Sampling and Beam Search 6.3 Parallel Sampling and Beam Search 6.4 Shared prefix 6.4 Shared prefix 6.5 Chatbot 6.5 Chatbot 7 Ablation Studies 7 Ablation Studies 8 Discussion 8 Discussion 9 Related Work 9 Related Work 10 Conclusion, Acknowledgement and References 10 Conclusion, Acknowledgement and References Abstract High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM’s source code is publicly available at https://github.com/vllm-project/vllm. 1 Introduction The emergence of large language models (LLMs) like GPT [5, 37] and PaLM [9] have enabled new applications such as programming assistants [6, 18] and universal chatbots [19, 35] that are starting to profoundly impact our work and daily routines. Many cloud companies [34, 44] are racing to provide these applications as hosted services. However, running these applications is very expensive, requiring a large number of hardware accelerators such as GPUs. According to recent estimates, processing an LLM request can be 10× more expensive than a traditional keyword query [43]. Given these high costs, increasing the throughput—and hence reducing the cost per request—of LLM serving systems is becoming more important. At the core of LLMs lies an autoregressive Transformer model [53]. This model generates words (tokens), one at a time, based on the input (prompt) and the previous sequence of the output’s tokens it has generated so far. For each request, this expensive process is repeated until the model outputs a termination token. This sequential generation process makes the workload memory-bound, underutilizing the computation power of GPUs and limiting the serving throughput. Improving the throughput is possible by batching multiple requests together. However, to process many requests in a batch, the memory space for each request should be efficiently managed. For example, Fig. 1 (left) illustrates the memory distribution for a 13B-parameter LLM on an NVIDIA A100 GPU with 40GB RAM. Approximately 65% of the memory is allocated for the model weights, which remain static during serving. Close to 30% of the memory is used to store the dynamic states of the requests. For Transformers, these states consist of the key and value tensors associated with the attention mechanism, commonly referred to as KV cache [41], which represent the context from earlier tokens to generate new output tokens in sequence. The remaining small percentage of memory is used for other data, including activations – the ephemeral tensors created when evaluating the LLM. Since the model weights are constant and the activations only occupy a small fraction of the GPU memory, the way the KV cache is managed is critical in determining the maximum batch size. When managed inefficiently, the KV cache memory can significantly limit the batch size and consequently the throughput of the LLM, as illustrated in Fig. 1 (right). In this paper, we observe that existing LLM serving systems [31, 60] fall short of managing the KV cache memory efficiently. This is mainly because they store the KV cache of a request in contiguous memory space, as most deep learning frameworks [33, 39] require tensors to be stored in contiguous memory. However, unlike the tensors in the traditional deep learning workloads, the KV cache has unique characteristics: it dynamically grows and shrinks over time as the model generates new tokens, and its lifetime and length are not known a priori. These characteristics make the existing systems’ approach significantly inefficient in two ways: First, the existing systems [31, 60] suffer from internal and external memory fragmentation. To store the KV cache of a request in contiguous space, they pre-allocate a contiguous chunk of memory with the request’s maximum length (e.g., 2048 tokens). This can result in severe internal fragmentation, since the request’s actual length can be much shorter than its maximum length (e.g., Fig. 11). Moreover, even if the actual length is known a priori, the pre-allocation is still inefficient: As the entire chunk is reserved during the request’s lifetime, other shorter requests cannot utilize any part of the chunk that is currently unused. Besides, external memory fragmentation can also be significant, since the preallocated size can be different for each request. Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems. Second, the existing systems cannot exploit the opportunities for memory sharing. LLM services often use advanced decoding algorithms, such as parallel sampling and beam search, that generate multiple outputs per request. In these scenarios, the request consists of multiple sequences that can partially share their KV cache. However, memory sharing is not possible in the existing systems because the KV cache of the sequences is stored in separate contiguous spaces. To address the above limitations, we propose PagedAttention, an attention algorithm inspired by the operating system’s (OS) solution to memory fragmentation and sharing: virtual memory with paging. PagedAttention divides the request’s KV cache into blocks, each of which can contain the attention keys and values of a fixed number of tokens. In PagedAttention, the blocks for the KV cache are not necessarily stored in contiguous space. Therefore, we can manage the KV cache in a more flexible way as in OS’s virtual memory: one can think of blocks as pages, tokens as bytes, and requests as processes. This design alleviates internal fragmentation by using relatively small blocks and allocating them on demand. Moreover, it eliminates external fragmentation as all blocks have the same size. Finally, it enables memory sharing at the granularity of a block, across the different sequences associated with the same request or even across the different requests. In this work, we build vLLM, a high-throughput distributed LLM serving engine on top of PagedAttention that achieves near-zero waste in KV cache memory. vLLM uses block-level memory management and preemptive request scheduling that are co-designed with PagedAttention. vLLM supports popular LLMs such as GPT [5], OPT [62], and LLaMA [52] with varying sizes, including the ones exceeding the memory capacity of a single GPU. Our evaluations on various models and workloads show that vLLM improves the LLM serving throughput by 2-4× compared to the state-of-the-art systems [31, 60], without affecting the model accuracy at all. The improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms (§4.3). In summary, we make the following contributions: • We identify the challenges in memory allocation in serving LLMs and quantify their impact on serving performance. • We propose PagedAttention, an attention algorithm that operates on KV cache stored in non-contiguous paged memory, which is inspired by the virtual memory and paging in OS. • We design and implement vLLM, a distributed LLM serving engine built on top of PagedAttention. • We evaluate vLLM on various scenarios and demonstrate that it substantially outperforms the previous state-of-theart solutions such as FasterTransformer [31] and Orca [60]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv Authors: (1) Woosuk Kwon, UC Berkeley with Equal contribution; (2) Zhuohan Li, UC Berkeley with Equal contribution; (3) Siyuan Zhuang, UC Berkeley; (4) Ying Sheng, UC Berkeley and Stanford University; (5) Lianmin Zheng, UC Berkeley; (6) Cody Hao Yu, Independent Researcher; (7) Cody Hao Yu, Independent Researcher; (8) Joseph E. Gonzalez, UC Berkeley; (9) Hao Zhang, UC San Diego; (10) Ion Stoica, UC Berkeley. Authors: Authors: (1) Woosuk Kwon, UC Berkeley with Equal contribution; (2) Zhuohan Li, UC Berkeley with Equal contribution; (3) Siyuan Zhuang, UC Berkeley; (4) Ying Sheng, UC Berkeley and Stanford University; (5) Lianmin Zheng, UC Berkeley; (6) Cody Hao Yu, Independent Researcher; (7) Cody Hao Yu, Independent Researcher; (8) Joseph E. Gonzalez, UC Berkeley; (9) Hao Zhang, UC San Diego; (10) Ion Stoica, UC Berkeley.

Stories, News & Mentions about ENKEMPASS

Enkempass’s Stories on HackerNoon