Grailed

grailed

When hip-hip artist Travis Scott appeared on Fortnite for a 10-minute virtual concert in April, he created one of the culturally defining moments of 2020. At a time when a global pandemic has put a halt to the concert industry, Scott managed to attract an audience of millions without even technically showing up for a live show. His avatar did all the heavy lifting during a 10-minute fantasy designed for Fornite, “Astronomical” It didn’t matter whether you were a gamer -- you probably about his appearance even if you didn’t show up.  And Scott -- whose meteoric rise to fame in the 2010s has made him both popular ( ) and -- is on fire. knew all 17 songs from his 2018 album Astroworld charted on the Billboard Hot 100 wealthy Travis Scott’s Fortnite moment demonstrates how an elite group of musicians, the new music moguls, have become brands unto themselves, wielding power through their cultural relevance, and oftentimes through digital. What Is a Music Mogul? Musicians used to align themselves with non-music brands such as McDonald’s to gain more power, visibility, and wealth. But an elite group of musicians, such as Jay-Z and Kanye West, have become so powerful that they’ve inverted the model. Some have changed the model completely by creating non-music brands. For example, is credited for compelling the beauty industry to create more inclusive products, a phenomenon known as the Fenty Effect. Arguably, Rihanna as a fashion and beauty brand has eclipsed Rihanna the musician. Travis Scott is inheriting the mantel from this high-flying group — a next-generation music mogul. Rihanna’s Fenty beauty line Travis Scott Achieves Cultural Relevance Travis Scott is firing on all cylinders as a musician and a brand. He ranks among the Top 20 most-listened-to artists on Spotify. With Kid Cudi, he released a song, “The Scotts,” that became his third U.S. Number One single. His collaboration with Rosalía, “TKN,” became a viral sensation on TikTok. A virtual performance on Fortnite attracted 12 million viewers in April. But his popularity extends beyond music. He’s also a culturally relevant brand. Brands become culturally relevant when they connect with an audience through their attitudes, beliefs, and behaviors. Sometimes cultural relevance means attitudes, beliefs, and behaviors, too. When brands achieve cultural relevance, they become so inextricably linked with our lives that we become lifelong members of their tribes. Scott defines cultural relevance in 2020 by connecting his music and his brand with people through mainstream behaviors such as: shaping Gaming , and Travis Scott is positioning himself to get a chunk of that money. His April Fortnite appearance had the same show-stopping power as any Coachella main stage performance — in fact, probably more. Online gaming is a $159 billion market in 2020 The performance drew 12.3 million concurrent players (i.e., watching at the same moment) and 27.7 million unique players (i.e., the number of different people who have signed on) globally across five airings over three days. But that’s not all. The concert generated a boatload of cash for Scott, including his appearance fee (estimated to be $500,000) and millions from the sale of in-game virtual merchandise such as a replica of Scott’s performance avatar and tons of other branded goodies — the online equivalent of concert Ts and sweatshirts sold at physical concerts. In this era of social distancing, physical concerts are not going to be feasible for a long, long time. “Astronomical” could be a template for the music industry, or at least one segment of the industry consisting of stars with the power to connect with fans through the rapidly growing online gaming industry. As Mark Mulligan of MiDIA Research told Billboard, “Games are the new venues of tomorrow. Travis Scott performing in Fortnite was the gaming equivalent of having a stadium-sized setup.” Scott, an avid gamer himself, follows a long line of musicians who have connected with fans through online gaming. Snoop Dogg, David Bowie, and Nile Rodgers are among the musicians who pioneered commercial collaborations with games such as NBA Live (Snoop Dogg), Omikron: The Nomad Soul (David Bowie), and HALO 2 (Nile Rodgers). The pioneers inspired many musicians to form relationships with gaming experiences. But Scott has taken the model to another level with the “Astronomical” concert. Fast Food , with a rich history that arguably dates back to Run-DMC name checking KFC and McDonald’s in “You Be Illin’” and NWA and LL Cool J both showing Burger King love in “I Ain’t That 1” and “The Bristol Hotel,” respectively. L.A.-based Fatburger achieved national fame in 1992 when Ice Cube name checked the chain in the song “It Was a Good Day.” Fast food and hip hop are culturally intertwined After that, Fatburger became the unofficial burger stand of choice for the hip-hop world, with artists such as Biggie Smalls calling out the restaurant either in song or word. Another famous L.A. eatery, Roscoe’s House of Chicken & Waffles, is so beloved by hip-hop stars that Snoop Dogg . offered to buy the joint when it faced financial problems The fast food industry understands the power of hip-hop to confer cultural relevance, too. Back in 2005, . Having deep pockets doesn’t make you culturally relevant, of course. But you can borrow cultural relevance by forming the right relationships. And that’s what McDonald’s did by co-branding with Travis Scott recently. McDonald’s enlisted the help of a marketing firm to recruit hip-hop artists to mention Big Macs in their songs From September 8 to October 4, McDonald's partnered with Scott on a marketing campaign that featured the Travis Scott value meal, which became so popular that shortages resulted. The Travis Scott meal promotional campaign also included the sale of related merchandise such , and Scott starred in a commercial for Mickey D’s. as a $90 pillow that sold out in days In turn, Scott made himself more relevant to one of the most powerful cultural institutions in the United States, Fast Food Nation. McDonald’s US chief marketing officer Morgan Flatley that McDonald’s decided to team up with Scott because of his cultural impact, especially when it comes to younger customers: told Business Insider His ability to kind of see where culture is going and have a hand in where culture is going is really unique. Then you couple that with his huge followership and his fans, social-media footprint, and . . . 3 billion streams. He just has an incredible audience. The relationship with McDonald’s also spawned an amusing behavior among younger customers who rolled up to McDonald’s drive-through lanes and placed their orders for the Travis Scott Meal with their own creative spin, such as announcing cryptically, “You know what I want,” or “You know why we’re here,” and then blasting Scott’s hit song, “Sicko Mode.” The “Sicko Mode”-style ordering also lit up TikTok, with fans posting their “Sicko Mode” moments on videos that went viral. Fans had so much fun coming up with their own distinct ways to order the Travis Scott Meal that McDonald’s executives , giving a heads up regarding the different vernacular one might expect in the drive-through lane, and encouraging employees to just roll with it. sent a memo to employees “To reduce confusion, please make crew aware of these monikers or alternate ordering methods,” the memo said. This is what cultural relevance is all about: influencing how people actually talk and behave. And all this happened in just weeks. Fashion Hip-hop has wielded such an enormous impact on fashion that . Hip-hop artists have taken their influence to the bank, such as Cardi B teaming up with the viral brand Fashion Nova to launch her first clothing collection or Kanye West teaming with the Gap to sell his own Yeezy line. Travis Scott turns heads with his many signature looks, and he’s also been a fashion creator. For example: entire books have been written about the subject In 2019, he collaborated with the Houston Astros to launch a limited edition baseball cap.Since 2017, he’s worked with Nike on many redesigns of popular Nike styles such as Air Force 1 and the Dunk.In 2017, he collaborated with Helmut Lang on a clothing collection that included denim, shoes, and shirts costing anywhere from $175 to $1,715. His record label, Cactus Jack, has morphed into an all-purpose merchandise operation that includes clothing ranging from hoodies to T shirts, and the way he combines T shirts, flannel, and jeans has influenced the way his fans dress. As Mike Destefano of Complex magazine , Scott makes it easy to adopt his various looks: points out Part of the appeal of his style is that it’s relatively accessible. Fans can buy the same Supreme Kermit T-shirt (at Grailed, eBay, or stores like Round Two), Stone Island jacket, and Jordan 4s Scott rocks. Even if they can’t cop the exact same pieces, his rockstar look can be easily recreated. “Travis cuts through because he’s authentic,” Ksubi CEO Craig King told Complex. “He has a true street style and his followers clearly relate to him.” Without question, Scott’s biggest influence is seen in sneakerhead culture. His own designs, such as the Jordan 1 Retro High Travis Scott, regularly sell out instantly and command huge premiums on the sneaker resale market. , Scott’s Air Force 1 collaboration with Nike is one of the most highly coveted Air Force 1 styles ever. He’s managed to crank out a number of Nike collaborations in a short amount of time, and each one has been wildly successful. Part of his appeal is how he drops sneakers to coincide with show-stopping events, such as the release of a Nike Air Force 1 “Sail” variation right before he released his popular Astroworld album, or the Travis Scott x Air Jordan 1 Low “Cactus Jack,” which he wore during his Super Bowl appearance in February 2019. According to sneakerhead site JustFreshKicks Why has Travis Scott become such a powerful force in sneakerhead culture? Well, he’s a sneakerhead himself who gets the nuances of sneakerhead style. As Yu-Ming Wu, Stadium Goods’ chief marketing officer, Cam Wolf of GQ: told A lot of rappers and entertainers like to think of themselves as sneakerheads, but Travis Scott is a rare breed with a deep appreciation of the culture. He often brings back older, hyped sneakers which helps educate new heads and remind the old heads. His love and respect for the industry is a big part of his sneaker cult following. His collaborations with Nike are a natural by-product of his love of sneaker style, not the other way around. As Cam Wolf noted, Those on the other side of the taste spectrum will tell you that his shoes are a reflection of the personal style that has made Scott a beloved fashion figure. Over the years, Scott has built up his fashion bonafides by wearing archival Raf Simons — the really good stuff from, like, the fall/winter 2001 collection — in the “Antidote” music video or taking a vintage Celine blouse seemingly pilfered from Kanye West on a seaside vacation (a dream!). Past Nike press releases detail Scott’s influences: everything from lil’ military-inspired pockets and Houston-Oilers blue, to burlap accents, a reference to farm product and a metonym for resilience and hard work. Sneaker influencer, DJ, and consultant Kish Kash also said that Scott’s personal style “adds the respect . . . It means [the shoe designs] aren’t contrived.” Where Is Travis Scott Headed Next? Travis Scott could have coasted through 2020 dropping songs and riding the wave of his success with sneakerheads. Instead, he made one of the defining cultural events of the year with his Fortnite concert. The COVID-19 pandemic will influence how his brand evolves in the near term. Physical tours, once a crucial element of any musician’s ability to extend their influence commercially and culturally, are not an option. Scott, whose Astroworld concert featured him singing on a roller coaster, would be the perfect choice for branded theme park attractions, but theme parks are struggling. Arguably, a Travis Scott-themed ride could help bring back fans to parks. Whatever he does, his Cactus Jack brand is key. I expect him to follow in the footsteps of Kanye West and launch his own Cactus Jack clothing collaboration with a major retailer. Of course, brick-and-mortar retail is still reeling from the impact of COVID-19. Everything will come down to how well Scott can hustle Cactus Jack fashion online by leveraging the reach of a well-known retailer that can endure the ravages of the pandemic — I’d predict Target or Walmart. It’s not a question of if, but when. Given Scott’s love of world building (as demonstrated through his immersive appearance on Fortnite), I could easily see him embracing augmented reality and virtual reality in a creative way. These two technologies (especially virtual reality) would have limited appeal — their use is growing, but they’re not exactly mainstream. But a Travis Scott virtual reality title is not out of the question given the popularity of his Fortnite appearance. Travis Scott is winning in 2020 by capitalizing on the power of digital and choosing his offline opportunities carefully and wisely.

From Fortnite to Fashion: How Travis Scott Creates Cultural Relevance

React Demo: CSS column-count + array reordering Live Demo: https://masonry-css-js.netlify.com/ React: https://github.com/jessekorzan/masonry-css-js JQuery: https://github.com/jessekorzan/css-masonry The problem… column-count works great except for left-to-right scan reading Masonry Layouts… popularized by sites like Pinterest. Loved by non-coding designers and holy-grailed in corporate boardrooms. Naturally, many sharp front-enders have worked on solutions. We have several libraries, techniques and hacks to achieve this marvel of content display. There are times when a masonry layout is an appropriate design decision. Depending on what else that design has to accommodate, these solutions might be a good fit. For me, especially in exploratory efforts, a pure-CSS method is a flexible and fast way to iterate. I don’t want to rely on DOM manipulation or end up with a contrived and constrained JS solution. This often takes the design off course (opportunity costs) or produces an awkward handoff with the real engineers. “Dude, srsly? 213 lines in jQuery to display some boxes?” The Problem My problem is relatively mundane. I want to run a masonry layout of recent news posts. Have left-to-right scan reading. NOT use JS to control or manipulate the view (no DOM manipulation with JS, for example). Use CSS column-count. It’s effective, simple and gets me close out of the box. No div-itis. One parent .wrapper for all .items in the layout. The Solution Using the React demo here, , you can grok what’s going. https://github.com/jessekorzan/masonry-css-js You can use CSS column-count “as is” for a masonry layout. If you want a nice left-to-right scan reading experience, you simply re-order the array of content before you render to the DOM. The main take away here... Not intending this to be a tutorial, here’s the gist of what’s happening: Grab some content (pseudo feed in JSON) Re-order the array of content so that the flow appears left-to-right Render the view CSS styling hooks (column-count) produces a nice cleanly ordered masonry layout https://masonry-css-js.netlify.com/ Magic algo to re-order the array BEFORE output to DOM Couple notes: column-count: X… the array re-order routine requires a matching X My demo is in React, but you can adapt the JS array re-order routine for anything Without some help from my teammates, I’d have never come up with the JS re-order routine (thanks NickB) To be responsive… it needs a little more JS (if you didn’t want just a single-column layout at narrower breakpoints) So there you have it. Which is what I need to keep moving forward with my current effort on a feature. And the DOM is 100% CSS. — I share this here for other “designers who code”. Maybe it helps in a prototype or reduce the cringing when the brief has that line… you know that line. “A Pinterest-style layout is required because ____________ and __________.” — Live Demo: https://masonry-css-js.netlify.com/ GitHub: https://github.com/jessekorzan/masonry-css-js

Easy CSS Masonry Layout w/ Left-To-Right Content Flow

Flow is a permissionless layer-1 blockchain built to support the high-scale use cases of games, virtual worlds, and the digital assets that power them. The blockchain was created by the team behind Cryptokitties, Dapper Labs, and NBA Top Shot. One core attribute that differentiates Flow from the other blockchains is its way of providing to objects within the blockchain. Fine-grained access is a new method of providing access to users that follows the principle of least privilege. This makes it highly secure and fault-resistant. Flow is one of the unique blockchains to have this, as it is built on the fundamentals of Capability-based security. fine-grained permissions But how does this help you as a developer? If you’re writing smart contracts, then you likely have to regularly define access to objects and functions within your contract. Understanding the different types of access better will help you to define access types in code more accurately. This will allow you to write highly secure and efficient smart contracts. Let’s dive in. Understanding fine-grained permissions As opposed to Role-based access control (RBAC), where permissions are defined loosely for every role and not specifically for every individual, fine-grained permissions allow developers to define permissions for objects at the lowest level. This allows permissions to be defined for every object. This also allows you to add new permissions types, such as time-expiry and geography-based permissions. Let’s understand the core principle behind fine-grained permissions. The principle of least privilege The concept of least privilege means that in a computing environment, any module (such as a process, a user, or a programmer) only has the necessary access that it needs to do its specific job. Any other unnecessary permissions for its specific job are either revoked or unavailable. For example, a programmer whose job is to create backups does not need to make user data updates, so it should only have read permissions on the database of users. This principle is the core principle behind fine-grained permissions. Capability-based Security A capability is an unforgeable token of authority. In capability-based authorization, your identity does not matter. If you’ve been sent an access token by the owner/admin that grants you the capability to access a resource and you can execute that capability, then you will have access. Hence, at runtime, the application does not check what your identity is but only that you can access the requested resource. Thus, fine-grained access can be implemented on the foundations of capability-based security. History of fine-grained access In the 1980s, operating systems would define permissions on objects as “read”, “write”, and “execute”. This was called Access Control Lists. In the 1990s, Role-based Access Control was introduced, which allowed you to create groups and assign users to groups. Then, the concepts of attribute-based access control and capability-based access control arrived, which brought fine-grained access into the picture. In the world of blockchains, however, fine-grained access is new and brought by Cadence, the language used to write Flow smart contracts. Fine-grained access in Cadence Cadence, the language used to write flow smart contracts, has built-in special keywords to allow for fine-grained permissions to resources. Below, we dive into the common keywords and the usage of these keywords to provide fine-grained access. The access keywords Flow defines certain keywords, known as access keywords, with different parameters to define the access privileges of a certain resource. Below are the four primary keywords: Pub or access(all) Using this keyword means the declaration is accessible in all scopes. However, this also means that the declaration is not updatable except when you’re updating the value at an index of an array. Pub or access(all) are the least restrictive types of access and should generally be avoided. Below is an example of using the pub keyword: pub contract Car {    pub let carBrand: String    pub let carNameByBrand: {String: String}    pub fun returnCarBrand(): String {        return self.carBrand    }}} Following the above code example, if there’s an object called CarObject of type Car, then the statement below invocation is valid. CarObject.carBrand //Equals the value of greeting in the object The values with access type pub are generally not updatable except when updating the value at a certain index of an array. CarObject.carNameByBrand = {} //Not ValidCarObject.carNameByBrand[“Porsche”] = “Carrera” //Valid access(self) access(self) means that the declaration is only visible in the current and inner scopes. For example, an access(self) field can only be accessed by the functions of the type it is a part of. When using access(self), it is common practice to define setters and getter methods to allow other objects and contracts to access this field. This keyword makes the usage very safe but also restrictive. Below is an example of using access(self): pub contract Car {    access(self) let carBrand: String    access(self) let carNameByBrand: {String: String}    pub fun returnCarBrand(): String {        return self.carBrand    }    pub fun returnCarName(_ brand: String): String? {        return self.carNameByBrand[brand]    }}} access(contract) access(contract) means that the declaration is only accessible in the scope of the contract that defines it. Hence, functions defined in other contracts under the same account or functions defined in contracts under other accounts will not be able to access it. This keyword is unique to Cadence and comes in handy when you have any contract-specific variables you don’t want to expose to other contracts. Below is an example of using the access(contract) keyword: pub contract CarContract {    pub struct Car {        access(contract) var carBrand: String    }    pub fun returnCarBrand(_brand: Car): String {        return Car.carBrand    }}} In the above example, if you have a method in a different contract that has an object of the Car struct, you cannot get the carBrand variable directly. However, you can call the returnCarBrand function to get the value of carBrand. access(account) This keyword means that the declaration is only accessible inside the entire account owned by the current smart contract. Hence, other contracts in the same account can access a variable freely. This type of access is very powerful, allowing you to keep a variable private to your account. You are also able to deploy a smart contract that needs to use the variable in the future, if needed. This way, several contracts in the same account can share information seamlessly with one another, promoting the modularity of code. Below is an example of how to use it: pub contract CarContract {    pub struct Car {        access(account) var carBrand: String    }    pub fun returnCarBrand(_brand: Car): String {        return Car.carBrand    }}} In the above example, if you have a method in a different contract that has an object of the Car struct, you can get the carBrand variable directly. Conclusion The concept of least privilege and capability-based security play a significant role in Flow's precise permissions. The access keywords provide clear definitions of variable and function access, as we have covered in this article. One of the most noteworthy features that fundamentally strengthens the security of the Flow blockchain is its fine-grained access control. Blockchain technology has long been plagued by bugs and hacks, most of which occur because of ill-conceived access controls. However, fine-grained access control provides developers with a remarkable opportunity to effortlessly and smoothly secure their smart contracts. Ultimately, this is what sets Flow apart in the world of blockchain. On the topic of fine-grained permissions, hopefully you found this in-depth exploration valuable and helpful. For more information, refer to the . Or explore the and get some hands-on experience! official documentation Flow Docs

Better Smart Contracts with Fine-Grained Permissions using Cadence

Beyond Login: Implement Fine-Grained Authorization With ZITADEL

লগইন এর বাইরে: ZITADEL এর সাথে সূক্ষ্ম-দানাযুক্ত অনুমোদন বাস্তবায়ন করুন

Über die Anmeldung hinaus: Implementieren Sie eine differenzierte Autorisierung mit ZITADEL

Más allá del inicio de sesión: implemente una autorización detallada con ZITADEL

Au-delà de la connexion : implémentez une autorisation détaillée avec ZITADEL

लॉगिन से परे: ZITADEL के साथ बढ़िया प्राधिकरण लागू करें

ログインを超えて: ZITADEL を使用したきめ細かい認証の実装

로그인 그 이상: ZITADEL로 세분화된 인증 구현

Além do login: implemente autorização refinada com ZITADEL

Помимо входа в систему: реализация детальной авторизации с помощью ZITADEL

Giriş Yapmanın Ötesinde: ZITADEL ile Ayrıntılı Yetkilendirmeyi Uygulayın

Ngoài đăng nhập: Triển khai ủy quyền chi tiết với ZITADEL

超越登录：使用 ZITADEL 实施细粒度授权

In Firefox 95, we’re shipping a novel sandboxing technology called — developed in collaboration with researchers at the University of California San Diego and the University of Texas — that makes it easy and efficient to isolate subcomponents to make the browser more secure. This technology opens up new opportunities beyond what’s been possible with traditional process-based sandboxing, and we look forward to expanding its usage and (hopefully) seeing it adopted in other browsers and software projects. RLBox This technique, which uses WebAssembly to isolate potentially-buggy code, builds on the we shipped last year to Mac and Linux users. Now, we’re bringing that technology to all supported Firefox platforms (desktop and mobile), and isolating five different modules: , , , and [1]. prototype Graphite Hunspell Ogg Expat Woff2 Going forward, we can treat these modules as untrusted code, and — assuming we did it right — even a zero-day vulnerability in any of them should pose no threat to Firefox. Accordingly, we’ve updated our to pay researchers for bypassing the sandbox even without a vulnerability in the isolated library. bug bounty program The Limits of Process Sandboxing All major browsers run Web content in its own sandboxed process, in theory preventing it from exploiting a browser vulnerability to compromise your computer. On desktop operating systems, Firefox also isolates each site in its own process in order to protect sites from each other. Unfortunately, threat actors routinely attack users by chaining together two vulnerabilities — one to compromise the sandboxed process containing the malicious site, and another to escape the sandbox [2]. To keep our users secure against the most well-funded adversaries, we need multiple layers of protection. Having already isolated things along trust boundaries, the next logical step is to isolate across functional boundaries. Historically, this has meant hoisting a subcomponent into its own process. For example, Firefox runs audio and video codecs in a dedicated, locked-down process with a limited interface to the rest of the system. However, there are some serious limitations to this approach. First, it requires decoupling the code and making it asynchronous, which is usually time-consuming and may impose a performance cost. Second, processes have a fixed memory overhead, and adding more of them increases the memory footprint of the application. For all of these reasons, nobody would seriously consider hoisting something like the XML parser into its own process. To isolate at that level of granularity, we need a different approach. Isolating with RLBox This is where RLBox comes in. Rather than hoisting the code into a separate process, we instead compile it into WebAssembly and then compile that WebAssembly into native code. This doesn’t result in us shipping any .wasm files in Firefox, since the WebAssembly step is only an intermediate representation in our build process. However, the transformation places two key restrictions on the target code: it can’t jump to unexpected parts of the rest of the program, and it can’t access memory outside of a specified region. Together, these restrictions ( ) between trusted and untrusted code, allowing us to run them in the same process largely as we were doing before. This, in turn, makes it easy to apply without major refactoring: the programmer only needs to sanitize any values that come from the sandbox (since they could be maliciously-crafted), a task which RLBox makes easy with a . make it safe to share an address space including the stack tainting layer The first step in this transformation is straightforward: we use to compile Firefox, and Clang knows how to emit WebAssembly, so we simply need to switch the output format for the given module from native code to wasm. For the second step, our prototype implementation used . Cranelift is excellent, but a second native code generator added complexity — and we realized that it would be simpler to just map the WebAssembly back into something that our existing build system could ingest. Clang Cranelift We accomplished this with , which performs a straightforward translation of WebAssembly into equivalent C code, which we can then feed back into Clang along with the rest of the Firefox source code. This approach is very simple, and automatically enables a number of important features that we support for regular Firefox code: profile-guided optimization, inlining across sandbox boundaries, crash reporting, debugger support, source-code indexing, and likely other things that we have yet to appreciate. wasm2c Next Steps RLBox is a big win for us on several fronts: it protects our users from accidental defects as well as supply-chain attacks, and it reduces the need for us to scramble when such issues are disclosed upstream. As such, we intend to continue applying to more components going forward. Some components are not a good fit for this approach — either because they depend too much on sharing memory with the rest of the program, or because they’re too performance-sensitive to accept the modest overhead incurred — but we’ve identified a number of other good candidates. Moreover, we hope to see this technology make its way into other browsers and software projects to make the ecosystem safer. is a standalone project that’s designed to be very modular and easy-to-use, and the team behind it would welcome other use-cases. RLBox Speaking of the team: I’d like to thank , , and for their tireless work in bringing this work from research concept to production. Shipping to hundreds of millions of users is hard, and they did some seriously impressive work. Shravan Narayan Deian Stefan Hovav Shacham Read more about RLBox and this announcement on the . UC San Diego Jacobs School of Engineering website [1] Cross-platform sandboxing for Graphite, Hunspell, and Ogg is shipping in Firefox 95, while Expat and Woff2 will ship in Firefox 96. [2] By using a syscall to to exploit a vulnerability in the OS, or by using an IPC message to exploit a vulnerability in a process hosting more-privileged parts of the browser. First published here

WebAssembly and Fine-Grained Sandboxing in Firefox 95

Authors:
(1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.;
(2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com;
(3) Aliaksandr Siarohin, Snap Inc.;
(4) Ivan Skorokhodov, Snap Inc.;
(5) Yanyu Li, Snap Inc.;
(6) Dahua Lin, CUHK;
(7) Xihui Liu, HKU;
(8) Ziwei Liu, NTU;
(9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 4 Human Verse Dataset 5 Experiments 5.1 Main Results 5.2 Ablation Study 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses A.2 MORE IMPLEMENTATION DETAILS We report implementation details like training hyper-parameters, and model architecture in Tab. 6. A.3 MORE ABLATION STUDY RESULTS We implement additional ablation study experiments on the second stage Structure-Guided Refiner. Note that due to the training resource limit and the resolution discrepancy between MS-COCO real images (512 × 512) and high-quality renderings (1024 × 1024), we conduct several toy ablation experiments in the lightweight 512 × 512 variant of our model: 1) w/o random dropout, where the all the input conditions are not dropout or masked out during the conditional training stage. 2) Only Text, where not any structural prediction is input to the model and we only use the text prompt as condition. 3) Condition on p, where we only use human pose skeleton p as input condition to the refiner network. 4) Condition on d that uses depth map d as input condition. 5) Condition on n that uses surface-normal n as input condition. And their combinations of 6) Condition on p, d; 7) Condition on p, n; 8) Condition on d, n, to verify the impact of each condition and the necessity of using such multi-level hierarchical structural guidance for fine-grained generation. The results are reported in Tab. 7. We can see that the random dropout conditioning scheme is crucial for more robust training with better image quality, especially in the two-stage generation pipeline. Besides, the structural map/guidance contains geometry and spatial relationship information, which are beneficial to image generation of higher quality. Another interesting phenomenon is that only conditioned on surface-normal n is better than conditioned on both the pose skeleton p and depth map d, which aligns with our intuition that surface-normal conveys rich structural information that mostly cover coarse-level skeleton and depth map, except for the keypoint location and foregroundbackground relationship. Overall, we can conclude from ablation results that: 1) Each condition (i.e., pose skeleton, depth map, and surface-normal) is important for higher-quality and more aligned generation, which validates the necessity of our first-stage Latent Structural Diffusion Model to jointly capture them. 2) The random dropout scheme for robust conditioning can essentially bridge the train-test error accumulation in two-stage pipeline, leading to better image results. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Authors: Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 3.3 Structure-Guided Refiner 4 Human Verse Dataset 4 Human Verse Dataset 5 Experiments 5 Experiments 5.1 Main Results 5.1 Main Results 5.2 Ablation Study 5.2 Ablation Study 6 Discussion and References 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses A.9 Licenses A.2 MORE IMPLEMENTATION DETAILS We report implementation details like training hyper-parameters, and model architecture in Tab. 6. A.3 MORE ABLATION STUDY RESULTS We implement additional ablation study experiments on the second stage Structure-Guided Refiner. Note that due to the training resource limit and the resolution discrepancy between MS-COCO real images (512 × 512) and high-quality renderings (1024 × 1024), we conduct several toy ablation experiments in the lightweight 512 × 512 variant of our model: 1) w/o random dropout, where the all the input conditions are not dropout or masked out during the conditional training stage. 2) Only Text, where not any structural prediction is input to the model and we only use the text prompt as condition. 3) Condition on p, where we only use human pose skeleton p as input condition to the refiner network. 4) Condition on d that uses depth map d as input condition. 5) Condition on n that uses surface-normal n as input condition. And their combinations of 6) Condition on p, d; 7) Condition on p, n; 8) Condition on d, n, to verify the impact of each condition and the necessity of using such multi-level hierarchical structural guidance for fine-grained generation. The results are reported in Tab. 7. We can see that the random dropout conditioning scheme is crucial for more robust training with better image quality, especially in the two-stage generation pipeline. Besides, the structural map/guidance contains geometry and spatial relationship information, which are beneficial to image generation of higher quality. Another interesting phenomenon is that only conditioned on surface-normal n is better than conditioned on both the pose skeleton p and depth map d, which aligns with our intuition that surface-normal conveys rich structural information that mostly cover coarse-level skeleton and depth map, except for the keypoint location and foregroundbackground relationship. Overall, we can conclude from ablation results that: 1) Each condition (i.e., pose skeleton, depth map, and surface-normal) is important for higher-quality and more aligned generation, which validates the necessity of our first-stage Latent Structural Diffusion Model to jointly capture them. 2) The random dropout scheme for robust conditioning can essentially bridge the train-test error accumulation in two-stage pipeline, leading to better image results. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How Pose, Depth, and Surface-Normal Impact HyperHuman’s Image Quality

How HyperHuman Pushes the Boundaries of Realistic Human Image Generation

Authors:
(1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.;
(2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com;
(3) Aliaksandr Siarohin, Snap Inc.;
(4) Ivan Skorokhodov, Snap Inc.;
(5) Yanyu Li, Snap Inc.;
(6) Dahua Lin, CUHK;
(7) Xihui Liu, HKU;
(8) Ziwei Liu, NTU;
(9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 4 Human Verse Dataset 5 Experiments 5.1 Main Results 5.2 Ablation Study 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses 4 HUMANVERSE DATASET Large-scale datasets with high quality samples, rich annotations, and diverse distribution are crucial for image generation tasks (Schuhmann et al., 2022; Podell et al., 2023), especially in the human domain (Liu et al., 2016;Jiang et al., 2023). To facilitate controllable human generation of high-fidelity, we establish a comprehensive human dataset with extensive annotations named HumanVerse. Please kindly refer to Appendix A.9 for more details about the dataset and annotation resources we use. Dataset Preprocessing. We curate from two principled datasets: LAION-2B-en (Schuhmann et al., 2022) and COYO-700M (Byeon et al., 2022). To isolate human images, we employ YOLOS (Fang et al., 2021) for human detection. Specifically, only those images containing 1 to 3 human bounding boxes are retained, where people should be visible with an area ratio exceeding 15%. We further rule out samples of poor aesthetics (< 4.5) or low resolution (< 200 × 200). This yields a high-quality subset by eliminating blurry and over-small humans. Unlike existing models that mostly train on full-body humans of simple context (Zhang & Agrawala, 2023), our dataset encompasses a wider spectrum, including various backgrounds and partial human regions such as clothing and limbs. 2D Human Poses. 2D human poses (skeleton of joint keypoints), which serve as one of the most flexible and easiest obtainable coarse-level condition signals, are widely used in controllable human generation studies (Ju et al., 2023b). To achieve accurate keypoint annotations, we resort to MMPose (Contributors, 2020) as inference interface and choose ViTPose-H (Xu et al., 2022) as backbone that performs best over several pose estimation benchmarks. In particular, the per-instance bounding box, keypoint coordinates and confidence are labeled, including whole-body skeleton (133 keypoints), body skeleton (17 keypoints), hand (21 keypoints), and facial landmarks (68 keypoints). Depth and Surface-Normal Maps are fine-grained structures that reflect the spatial geometry of images (Wu et al., 2022), which are commonly used in conditional generation (Mou et al., 2023). We apply Omnidata (Eftekhar et al., 2021) for monocular depth and normal. The MiDaS (Ranftl et al., 2022) is further annotated following recent depth-to-image pipelines (Rombach et al., 2022). Outpaint for Accurate Annotations. Diffusion models have shown promising results on image inpainting and outpainting, where the appearance and structure of unseen regions can be hallucinated based on the visible parts. Motivated by this, we propose to outpaint each image for a more holistic view given that most off-the-shelf structure estimators are trained on the “complete” image views. Although the outpainted region may be imperfect with artifacts, it can complement a more comprehensive human structure. To this end, we utilize the powerful SD-Inpaint to outpaint the surrounding areas of the original canvas. These images are further processed by off-the-shelf estimators, where we only use the labeling within the original image region for more accurate annotations. Overall Statistics. In summary, COYO subset contains 90, 948, 474 (91M) images and LAION-2B subset contains 248, 396, 109 (248M) images, which is 18.12% and 20.77% of fullset, respectively. The whole annotation process takes 640 16/32G NVIDIA V100 GPUs for two weeks in parallel. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Authors: Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 3.3 Structure-Guided Refiner 4 Human Verse Dataset 4 Human Verse Dataset 5 Experiments 5 Experiments 5.1 Main Results 5.1 Main Results 5.2 Ablation Study 5.2 Ablation Study 6 Discussion and References 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses A.9 Licenses 4 HUMANVERSE DATASET Large-scale datasets with high quality samples, rich annotations, and diverse distribution are crucial for image generation tasks (Schuhmann et al., 2022; Podell et al., 2023), especially in the human domain (Liu et al., 2016;Jiang et al., 2023). To facilitate controllable human generation of high-fidelity, we establish a comprehensive human dataset with extensive annotations named HumanVerse. Please kindly refer to Appendix A.9 for more details about the dataset and annotation resources we use. Dataset Preprocessing. We curate from two principled datasets: LAION-2B-en (Schuhmann et al., 2022) and COYO-700M (Byeon et al., 2022). To isolate human images, we employ YOLOS (Fang et al., 2021) for human detection. Specifically, only those images containing 1 to 3 human bounding boxes are retained, where people should be visible with an area ratio exceeding 15%. We further rule out samples of poor aesthetics (< 4.5) or low resolution (< 200 × 200). This yields a high-quality subset by eliminating blurry and over-small humans. Unlike existing models that mostly train on full-body humans of simple context (Zhang & Agrawala, 2023), our dataset encompasses a wider spectrum, including various backgrounds and partial human regions such as clothing and limbs. Dataset Preprocessing. 2D Human Poses. 2D human poses (skeleton of joint keypoints), which serve as one of the most flexible and easiest obtainable coarse-level condition signals, are widely used in controllable human generation studies (Ju et al., 2023b). To achieve accurate keypoint annotations, we resort to MMPose (Contributors, 2020) as inference interface and choose ViTPose-H (Xu et al., 2022) as backbone that performs best over several pose estimation benchmarks. In particular, the per-instance bounding box, keypoint coordinates and confidence are labeled, including whole-body skeleton (133 keypoints), body skeleton (17 keypoints), hand (21 keypoints), and facial landmarks (68 keypoints). 2D Human Poses. Depth and Surface-Normal Maps are fine-grained structures that reflect the spatial geometry of images (Wu et al., 2022), which are commonly used in conditional generation (Mou et al., 2023). We apply Omnidata (Eftekhar et al., 2021) for monocular depth and normal. The MiDaS (Ranftl et al., 2022) is further annotated following recent depth-to-image pipelines (Rombach et al., 2022). Depth and Surface-Normal Maps Outpaint for Accurate Annotations. Diffusion models have shown promising results on image inpainting and outpainting, where the appearance and structure of unseen regions can be hallucinated based on the visible parts. Motivated by this, we propose to outpaint each image for a more holistic view given that most off-the-shelf structure estimators are trained on the “complete” image views. Although the outpainted region may be imperfect with artifacts, it can complement a more comprehensive human structure. To this end, we utilize the powerful SD-Inpaint to outpaint the surrounding areas of the original canvas. These images are further processed by off-the-shelf estimators, where we only use the labeling within the original image region for more accurate annotations. Outpaint for Accurate Annotations. Overall Statistics. In summary, COYO subset contains 90, 948, 474 (91M) images and LAION-2B subset contains 248, 396, 109 (248M) images, which is 18.12% and 20.77% of fullset, respectively. The whole annotation process takes 640 16/32G NVIDIA V100 GPUs for two weeks in parallel. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

HumanVerse: A Richly Annotated Dataset for Realistic, Controllable Human Image Synthesis

Authors:
(1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.;
(2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com;
(3) Aliaksandr Siarohin, Snap Inc.;
(4) Ivan Skorokhodov, Snap Inc.;
(5) Yanyu Li, Snap Inc.;
(6) Dahua Lin, CUHK;
(7) Xihui Liu, HKU;
(8) Ziwei Liu, NTU;
(9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 4 Human Verse Dataset 5 Experiments 5.1 Main Results 5.2 Ablation Study 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses 3.2 LATENT STRUCTURAL DIFFUSION MODEL To incorporate the body skeletons for pose control, the simplest way is by feature residual (Mou et al., 2023) or input concatenation (Ju et al., 2023b). However, three problems remain: 1) The sparse keypoints only depict the coarse human structure, while the fine-grained geometry and foregroundbackground relationship are ignored. Besides, the naive DM training is merely supervised by RGB signals, which fails to capture the inherent structural information. 2) The image RGB and structure representations are spatially aligned but substantially different in latent space. How to jointly model them remains challenging. 3) In contrast to the colorful RGB images, the structure maps are mostly monotonous with similar values in local regions, which are hard to learn by DMs (Lin et al., 2023). Unified Model for Simultaneous Denoising. Our solution to the first problem is to simultaneously denoise the depth and surface-normal along with the synthesized RGB image. We choose them as additional learning targets due to two reasons: 1) Depth and normal can be easily annotated for large-scale dataset, which are also used in recent controllable T2I generation (Zhang & Agrawala, 2023). 2) As two commonly-used structural guidance, they complement the spatial relationship and geometry information, where the depth (Deng et al., 2022), normal (Wang et al., 2022), or both (Yu et al., 2022b) are proven beneficial in recent 3D studies. To this end, a naive method is to train three separate networks to denoise the RGB, depth, and normal individually. But the spatial alignment between them is hard to preserve. Therefore, we propose to capture the joint distribution in a unified model by simultaneous denoising, which can be trained with simplified objective (Ho et al., 2020): Furthermore, we find that the number of shared modules can trade-off between the spatial alignment and distribution learning: On the one hand, more shared layers guarantee the more similar features of final output, leading to the paired texture and structure corresponding to the same image. On the other hand, the RGB, depth, and normal can be treated as different views of the same image, where predicting them from the same feature resembles an image-to-image translation task in essence. Empirically, we find the optimal design to replicate the conv_in, first DownBlock, last UpBlock, and conv_out for each expert branch, where each branch’s skip-connections are maintained separately (as depicted in Fig. 2). This yields both the spatial alignment and joint capture of image texture and structure. Note that such design is not limited to three targets, but can generalize to arbitrary number of paired distributions by simply involving more branches with little computation overhead. 3.3 STRUCTURE-GUIDED REFINER This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Authors: Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 3.3 Structure-Guided Refiner 4 Human Verse Dataset 4 Human Verse Dataset 5 Experiments 5 Experiments 5.1 Main Results 5.1 Main Results 5.2 Ablation Study 5.2 Ablation Study 6 Discussion and References 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses A.9 Licenses 3.2 LATENT STRUCTURAL DIFFUSION MODEL To incorporate the body skeletons for pose control, the simplest way is by feature residual (Mou et al., 2023) or input concatenation (Ju et al., 2023b). However, three problems remain: 1) The sparse keypoints only depict the coarse human structure, while the fine-grained geometry and foregroundbackground relationship are ignored. Besides, the naive DM training is merely supervised by RGB signals, which fails to capture the inherent structural information. 2) The image RGB and structure representations are spatially aligned but substantially different in latent space. How to jointly model them remains challenging. 3) In contrast to the colorful RGB images, the structure maps are mostly monotonous with similar values in local regions, which are hard to learn by DMs (Lin et al., 2023). Unified Model for Simultaneous Denoising. Our solution to the first problem is to simultaneously denoise the depth and surface-normal along with the synthesized RGB image. We choose them as additional learning targets due to two reasons: 1) Depth and normal can be easily annotated for large-scale dataset, which are also used in recent controllable T2I generation (Zhang & Agrawala, 2023). 2) As two commonly-used structural guidance, they complement the spatial relationship and geometry information, where the depth (Deng et al., 2022), normal (Wang et al., 2022), or both (Yu et al., 2022b) are proven beneficial in recent 3D studies. To this end, a naive method is to train three separate networks to denoise the RGB, depth, and normal individually. But the spatial alignment between them is hard to preserve. Therefore, we propose to capture the joint distribution in a unified model by simultaneous denoising, which can be trained with simplified objective (Ho et al., 2020): Unified Model for Simultaneous Denoising. Furthermore, we find that the number of shared modules can trade-off between the spatial alignment and distribution learning: On the one hand, more shared layers guarantee the more similar features of final output, leading to the paired texture and structure corresponding to the same image. On the other hand, the RGB, depth, and normal can be treated as different views of the same image, where predicting them from the same feature resembles an image-to-image translation task in essence. Empirically, we find the optimal design to replicate the conv_in, first DownBlock, last UpBlock, and conv_out for each expert branch, where each branch’s skip-connections are maintained separately (as depicted in Fig. 2). This yields both the spatial alignment and joint capture of image texture and structure. Note that such design is not limited to three targets, but can generalize to arbitrary number of paired distributions by simply involving more branches with little computation overhead. 3.3 STRUCTURE-GUIDED REFINER This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How HyperHuman Solves Structural Challenges with a Latent Diffusion Model

Authors:
(1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.;
(2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com;
(3) Aliaksandr Siarohin, Snap Inc.;
(4) Ivan Skorokhodov, Snap Inc.;
(5) Yanyu Li, Snap Inc.;
(6) Dahua Lin, CUHK;
(7) Xihui Liu, HKU;
(8) Ziwei Liu, NTU;
(9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 4 Human Verse Dataset 5 Experiments 5.1 Main Results 5.2 Ablation Study 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses 2 Related Work Text-to-Image Diffusion Models. Text-to-image (T2I) generation, the endeavor to synthesize highfidelity images from natural language descriptions, has made remarkable strides in recent years. Distinguished by the superior scalability and stable training, diffusion-based T2I models have eclipsed conventional GANs in terms of performance (Dhariwal & Nichol, 2021), becoming the predominant choice in generation (Nichol et al., 2021; Saharia et al., 2022; Balaji et al., 2022; Li et al., 2023). By formulating the generation as an iterative denoising process (Ho et al., 2020), exemplar works like Stable Diffusion (Rombach et al., 2022) and DALL·E 2 (Ramesh et al., 2022) demonstrate unprecedented quality. Despite this, they mostly fail to create high-fidelity humans. One main reason is that existing models lack inherent structural awareness for human, making them even struggle to generate human of reasonable anatomy, e.g., correct number of arms and legs. To this end, our proposed approach explicitly models human structures within the latent space of diffusion model. Controllable Human Image Generation. Traditional approaches for controllable human generation can be categorized into GAN-based (Zhu et al., 2017; Siarohin et al., 2019) and VAE-based (Ren et al., 2020; Yang et al., 2021), where the reference image and conditions are taken as input. To facilitate user-friendly applications, recent studies explore text prompts as generation guidance (Roy et al., 2022; Jiang et al., 2022), yet are confined to simple pose or style descriptions. The most relevant works that enable open-vocabulary pose-guided controllable human synthesis are ControlNet (Zhang & Agrawala, 2023), T2I-Adapter (Mou et al., 2023), and HumanSD (Ju et al., 2023b). However, they either suffer from inadequate pose control, or are confined to artistic styles of limited diversity. Besides, most previous studies merely take pose as input, while ignoring the multi-level correlations between human appearance and different types of structural information. In this work, we propose to incorporate structural awareness from coarse-level skeleton to fine-grained depth and surface-normal by joint denoising with expert branch, thus simultaneously capturing both the explicit appearance and latent structure in a unified framework for realistic human image synthesis. Datasets for Human Image Generation. Large datasets are crucial for image generation. Existing human-centric collections are mainly confronted with following drawbacks: 1) Low-resolution of poor quality. For example, Market-1501 (Zheng et al., 2015) contains noisy pedestrian images of resolution 128 × 64, and VITON (Han et al., 2018) has human-clothing pairs of 256 × 192, which are inadequate for training high-definition models. 2) Limited diversity of certain domain. For example, SHHQ (Fu et al., 2022) is mostly composed of full-body humans with clean background, and DeepFashion (Liu et al., 2016) focuses on fashion images of little pose variations. 3) Insufficient dataset scale, where LIP (Gong et al., 2017) and Human-Art (Ju et al., 2023a) only contain 50K samples. Furthermore, none of the existing datasets contain rich annotations, which typically label a singular aspect of images. In this work, we take a step further by curating in-the-wild HumanVerse dataset with comprehensive annotations like human pose, depth map, and surface-normal map. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Authors: Authors: (1) Xian Liu, Snap Inc., CUHK with Work done during an internship at Snap Inc.; (2) Jian Ren, Snap Inc. with Corresponding author: jren@snapchat.com; (3) Aliaksandr Siarohin, Snap Inc.; (4) Ivan Skorokhodov, Snap Inc.; (5) Yanyu Li, Snap Inc.; (6) Dahua Lin, CUHK; (7) Xihui Liu, HKU; (8) Ziwei Liu, NTU; (9) Sergey Tulyakov, Snap Inc. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Our Approach and 3.1 Preliminaries and Problem Setting 3 Our Approach and 3.1 Preliminaries and Problem Setting 3.2 Latent Structural Diffusion Model 3.2 Latent Structural Diffusion Model 3.3 Structure-Guided Refiner 3.3 Structure-Guided Refiner 4 Human Verse Dataset 4 Human Verse Dataset 5 Experiments 5 Experiments 5.1 Main Results 5.1 Main Results 5.2 Ablation Study 5.2 Ablation Study 6 Discussion and References 6 Discussion and References A Appendix and A.1 Additional Quantitative Results A Appendix and A.1 Additional Quantitative Results A.2 More Implementation Details and A.3 More Ablation Study Results A.2 More Implementation Details and A.3 More Ablation Study Results A.4 More User Study Details A.4 More User Study Details A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.5 Impact of Random Seed and Model Robustness and A.6 Boarder Impact and Ethical Consideration A.7 More Comparison Results and A.8 Additional Qualitative Results A.7 More Comparison Results and A.8 Additional Qualitative Results A.9 Licenses A.9 Licenses 2 Related Work Text-to-Image Diffusion Models. Text-to-image (T2I) generation, the endeavor to synthesize highfidelity images from natural language descriptions, has made remarkable strides in recent years. Distinguished by the superior scalability and stable training, diffusion-based T2I models have eclipsed conventional GANs in terms of performance (Dhariwal & Nichol, 2021), becoming the predominant choice in generation (Nichol et al., 2021; Saharia et al., 2022; Balaji et al., 2022; Li et al., 2023). By formulating the generation as an iterative denoising process (Ho et al., 2020), exemplar works like Stable Diffusion (Rombach et al., 2022) and DALL·E 2 (Ramesh et al., 2022) demonstrate unprecedented quality. Despite this, they mostly fail to create high-fidelity humans. One main reason is that existing models lack inherent structural awareness for human, making them even struggle to generate human of reasonable anatomy, e.g., correct number of arms and legs. To this end, our proposed approach explicitly models human structures within the latent space of diffusion model. Text-to-Image Diffusion Models. Controllable Human Image Generation. Traditional approaches for controllable human generation can be categorized into GAN-based (Zhu et al., 2017; Siarohin et al., 2019) and VAE-based (Ren et al., 2020; Yang et al., 2021), where the reference image and conditions are taken as input. To facilitate user-friendly applications, recent studies explore text prompts as generation guidance (Roy et al., 2022; Jiang et al., 2022), yet are confined to simple pose or style descriptions. The most relevant works that enable open-vocabulary pose-guided controllable human synthesis are ControlNet (Zhang & Agrawala, 2023), T2I-Adapter (Mou et al., 2023), and HumanSD (Ju et al., 2023b). However, they either suffer from inadequate pose control, or are confined to artistic styles of limited diversity. Besides, most previous studies merely take pose as input, while ignoring the multi-level correlations between human appearance and different types of structural information. In this work, we propose to incorporate structural awareness from coarse-level skeleton to fine-grained depth and surface-normal by joint denoising with expert branch, thus simultaneously capturing both the explicit appearance and latent structure in a unified framework for realistic human image synthesis. Controllable Human Image Generation. Datasets for Human Image Generation. Large datasets are crucial for image generation. Existing human-centric collections are mainly confronted with following drawbacks: 1) Low-resolution of poor quality. For example, Market-1501 (Zheng et al., 2015) contains noisy pedestrian images of resolution 128 × 64, and VITON (Han et al., 2018) has human-clothing pairs of 256 × 192, which are inadequate for training high-definition models. 2) Limited diversity of certain domain. For example, SHHQ (Fu et al., 2022) is mostly composed of full-body humans with clean background, and DeepFashion (Liu et al., 2016) focuses on fashion images of little pose variations. 3) Insufficient dataset scale, where LIP (Gong et al., 2017) and Human-Art (Ju et al., 2023a) only contain 50K samples. Furthermore, none of the existing datasets contain rich annotations, which typically label a singular aspect of images. In this work, we take a step further by curating in-the-wild HumanVerse dataset with comprehensive annotations like human pose, depth map, and surface-normal map. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about GRAILED

Grailed’s Stories on HackerNoon