Reelist

reelist

Startups of The Year 2023 Winners: North America 

Authors:
(1) Kedan Li, University of Illinois at Urbana-Champaign;
(2) Min Jin Chong, University of Illinois at Urbana-Champaign;
(3) Jingen Liu, JD AI Research;
(4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro
Related Work
Proposed Method
Experiments
Conclusions and References 2. Related Work Image synthesis: Spatial transformer networks estimate geometric transformations using neural networks [23]. Subsequent work [28,39] shows how to warp one object to another. Warping can be used to produce images of rigid objects [26,30] and non-rigid objects (e.g., clothing) [17,12,45]. In contrast to prior work, we use multiple spatial warpers. Our warps must be combined into a single image, and our U-Net for producing this image follows trends in inpainting (methods that fill in missing portions of an image, see [48,31,50,49]). Han et al. [16,52] show inpainting methods can complete missing clothing items on people. In our work, we use FID∞ to quantitatively evaluate our method. This is based on the Frchet Inception Distance (FID) [18], a common metric in generative image modelling [5,54,29]. Chong et al. [9] recently showed that FID is biased; extrapolation removes the bias, to an unbiased score (FID∞). Generating clothed people: Zhu et al. [57] used a conditional GAN to generate images based on pose skeleton and text descriptions of garment. SwapNet [38] learns to transfer clothes from person A to person B by disentangling clothing and pose features. Hsiao et al. [20] learned a fashion model synthesis network using per-garment encodings to enable convenient minimal edit to specific items. In contrast, we warp products onto real model images. Shape matching underlies our method to match product to model. Tsiao et al. [19] built a shape embedding to enable matching between human body and well-fitting clothing items. Prior work estimated the shape of human body [4,27], clothing items [10,25] and both [35,40], through 2D images. The DensePose [1] descriptor helps modeling the deformation and shading of cloth and, therefore, has been adopted by recent work [36,13,47,51,7,52]. Virtual try-on (VTO) maps a product to a model image. VITON [17] uses a U-Net to generate a coarse synthesis and a mask on the model where the product is presented. A mapping from the product mask to the on-model mask is learned through Thin plate spline (TPS) transformation [3]. The learned mapping is applied on the product image to create a warp. Following their work, Wang et al. [45] improved the architecture using a Geometric Matching Module [39] to estimate the TPS transformations parameters directly from pairs of product image and target person. They train a separate refinement network to combine the warp and the target image. VTNFP [53] extends the work by incorporatiing body segments prediction and later works follow similar procedure [37,24,42,22,2]. However, TPS transformation fails to produce reasonable warps, due to the noisiness of generated masks in our dataset, as shown in Figure 6 right. Instead, we adopt affine transformations which we have found to be more robust to imperfections instead of TPS transformation. A group of following work extended the task to multi-pose. Warping-GAN [11] combined adversarial training with GMM, and generate post and texture separately using a two stage network. MG-VTON [12] further refine the generation method using a three-stage generation network. Other work [21,55,51,7,46] followed similar procedure. Han et al. [15] argued that TPS transformation has low degree of freedom and proposed a flow-based method to create the warp. Much existing virtual try-on work [17,12,21,47,55,53,24,37] is evaluated on datasets that only have tops (t-shirt, shirt, etc.). Having only tops largely reduces the likelihood of shape mismatch as tops have simple and similar shapes. In our work, we extend the problem to include clothing items of all categories(tshirt, shirt, pants, shorts, dress, skirt, robe, jacket, coat, etc.), and propose a method for matching the shape between the source product and the target model. Evaluation shows that using pairs that match in shape significantly increases the generation quality for both our and prior work (table 4.3). In addition, real studio outfits are often covered by an unzipped/unbuttoned outerwear, which is also not presented in prior work [17,12,21,47,55,53,37]. This can cause partition or severe occlusion to the garment, and is not addressed by prior work as shown in Figure 6. We show that our multi-warp generation module ameliorates these difficulties. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Authors: Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro Related Work Proposed Method Experiments Conclusions and References Abstract and Intro Abstract and Intro Related Work Related Work Proposed Method Proposed Method Experiments Experiments Conclusions and References Conclusions and References 2. Related Work Image synthesis : Spatial transformer networks estimate geometric transformations using neural networks [23]. Subsequent work [28,39] shows how to warp one object to another. Warping can be used to produce images of rigid objects [26,30] and non-rigid objects (e.g., clothing) [17,12,45]. In contrast to prior work, we use multiple spatial warpers. Image synthesis Our warps must be combined into a single image, and our U-Net for producing this image follows trends in inpainting (methods that fill in missing portions of an image, see [48,31,50,49]). Han et al. [16,52] show inpainting methods can complete missing clothing items on people. In our work, we use FID∞ to quantitatively evaluate our method. This is based on the Frchet Inception Distance (FID) [18], a common metric in generative image modelling [5,54,29]. Chong et al. [9] recently showed that FID is biased; extrapolation removes the bias, to an unbiased score (FID∞). Generating clothed people: Zhu et al. [57] used a conditional GAN to generate images based on pose skeleton and text descriptions of garment. SwapNet [38] learns to transfer clothes from person A to person B by disentangling clothing and pose features. Hsiao et al. [20] learned a fashion model synthesis network using per-garment encodings to enable convenient minimal edit to specific items. In contrast, we warp products onto real model images. Generating clothed people: Shape matching underlies our method to match product to model. Tsiao et al. [19] built a shape embedding to enable matching between human body and well-fitting clothing items. Prior work estimated the shape of human body [4,27], clothing items [10,25] and both [35,40], through 2D images. The DensePose [1] descriptor helps modeling the deformation and shading of cloth and, therefore, has been adopted by recent work [36,13,47,51,7,52]. Shape matching Virtual try-on (VTO) maps a product to a model image. VITON [17] uses a U-Net to generate a coarse synthesis and a mask on the model where the product is presented. A mapping from the product mask to the on-model mask is learned through Thin plate spline (TPS) transformation [3]. The learned mapping is applied on the product image to create a warp. Following their work, Wang et al. [45] improved the architecture using a Geometric Matching Module [39] to estimate the TPS transformations parameters directly from pairs of product image and target person. They train a separate refinement network to combine the warp and the target image. VTNFP [53] extends the work by incorporatiing body segments prediction and later works follow similar procedure [37,24,42,22,2]. However, TPS transformation fails to produce reasonable warps, due to the noisiness of generated masks in our dataset, as shown in Figure 6 right. Instead, we adopt affine transformations which we have found to be more robust to imperfections instead of TPS transformation. A group of following work extended the task to multi-pose. Warping-GAN [11] combined adversarial training with GMM, and generate post and texture separately using a two stage network. MG-VTON [12] further refine the generation method using a three-stage generation network. Other work [21,55,51,7,46] followed similar procedure. Han et al. [15] argued that TPS transformation has low degree of freedom and proposed a flow-based method to create the warp. Virtual try-on Much existing virtual try-on work [17,12,21,47,55,53,24,37] is evaluated on datasets that only have tops (t-shirt, shirt, etc.). Having only tops largely reduces the likelihood of shape mismatch as tops have simple and similar shapes. In our work, we extend the problem to include clothing items of all categories(tshirt, shirt, pants, shorts, dress, skirt, robe, jacket, coat, etc.), and propose a method for matching the shape between the source product and the target model. Evaluation shows that using pairs that match in shape significantly increases the generation quality for both our and prior work (table 4.3). In addition, real studio outfits are often covered by an unzipped/unbuttoned outerwear, which is also not presented in prior work [17,12,21,47,55,53,37]. This can cause partition or severe occlusion to the garment, and is not addressed by prior work as shown in Figure 6. We show that our multi-warp generation module ameliorates these difficulties. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Toward Accurate, Realistic Virtual Try-on Through Shape Matching: Related Work

শেপ ম্যাচিং এর মাধ্যমে সঠিক, বাস্তবসম্মত ভার্চুয়াল ট্রাই-অন: সম্পর্কিত কাজ

Auf dem Weg zu präzisem, realistischem virtuellem Anprobieren durch Formabgleich: Verwandte Arbeiten

Hacia una prueba virtual precisa y realista mediante la combinación de formas: trabajo relacionado

Vers un essai virtuel précis et réaliste grâce à la correspondance de formes : travaux connexes

आकार मिलान के माध्यम से सटीक, यथार्थवादी आभासी ट्राई-ऑन की ओर: संबंधित कार्य

形状マッチングによる正確でリアルな仮想試着に向けて：関連研究

형태 매칭을 통한 정확하고 실감나는 가상 시착을 향하여: 관련 연구

Rumo a um teste virtual preciso e realista por meio da correspondência de formas: trabalho relacionado

На пути к точной и реалистичной виртуальной примерке посредством сопоставления форм: сопутствующие работы

Şekil Eşleştirme Yoluyla Doğru, Gerçekçi Sanal Denemeye Doğru: İlgili Çalışma

Hướng tới việc thử nghiệm ảo thực tế, chính xác thông qua việc khớp hình dạng: Công việc liên quan

通过形状匹配实现准确、逼真的虚拟试穿：相关工作

Authors:
(1) Kedan Li, University of Illinois at Urbana-Champaign;
(2) Min Jin Chong, University of Illinois at Urbana-Champaign;
(3) Jingen Liu, JD AI Research;
(4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro
Related Work
Proposed Method
Experiments
Conclusions and References Abstract A virtual try-on method takes a product image and an image of a model and produces an image of the model wearing the product. Most methods essentially compute warps from the product image to the model image and combine using image generation methods. However, obtaining a realistic image is challenging because the kinematics of garments is complex and because outline, texture, and shading cues in the image reveal errors to human viewers. The garment must have appropriate drapes; texture must be warped to be consistent with the shape of a draped garment; small details (buttons, collars, lapels, pockets, etc.) must be placed appropriately on the garment, and so on. Evaluation is particularly difficult and is usually qualitative. This paper uses quantitative evaluation on a challenging, novel dataset to demonstrate that (a) for any warping method, one can choose target models automatically to improve results, and (b) learning multiple coordinated specialized warpers offers further improvements on results. Target models are chosen by a learned embedding procedure that predicts a representation of the products the model is wearing. This prediction is used to match products to models. Specialized warpers are trained by a method that encourages a second warper to perform well in locations where the first works poorly. The warps are then combined using a U-Net. Qualitative evaluation confirms that these improvements are wholesale over outline, texture shading, and garment details. Keywords: Fashion, Virtual try-on, Image generation, Image warping 1. Introduction E-commerce means not being able to try on a product, which is difficult for fashion consumers [44]. Sites now routinely put up photoshoots of models wearing products, but volume and turnover mean doing so is very expensive and time consuming [34]. There is a need to generate realistic and accurate images of fashion models wearing different sets of clothing. One could use 3D models of posture [8,14]. The alternative – synthesize product-model images without 3D measurements [17,45,39,11,15] – is known as virtual try-on. These methods usually consist of two components: 1) a spatial transformer to warp the product image using some estimate of the model’s pose and 2) an image generation network that combines the coarsely aligned, warped product with the model image to produce a realistic image of the model wearing the product. It is much easier to transfer with simple garments like t-shirts, which are emphasized in the literature. General garments (unlike t-shirts) might open in front; have sophisticated drapes; have shaped structures like collars and cuffs; have buttons; and so on. These effects severely challenge existing methods (examples in Supplementary Materials). Warping is significantly improved if one uses the product image to choose a model image that is suited to that garment (Figure 1). At least in part, this is a result of how image generation networks are trained. We train using paired images – a product and a model wearing a product [17,45,53]. This means that the generation network always expects the target image to be appropriate for the product (so it is not trained to, for example, put a sweater onto a model wearing a dress, Figure 1). An alternative is to use adversarial training [11,12,38,13,37]; but it is hard to preserve specific product details (for example, a particular style of buttons; a decal on a t-shirt) in this framework. To deal with this difficulty, we learn an embedding space for choosing product-model pairs that will result in high-quality transfers (Figure 2). The embedding learns to predict what shape a garment in a model image would take if it were in a product image. Products are then matched to models wearing similarly shaped garments. Because models typically wear many garments, we use a spatial attention visual encoder to parse each category (top, bottom, outerwear, all-body, etc.) of garment and embed each separately. Another problem arises when a garment is open (for example, an unbuttoned coat). In this case, the target of the warp might have more than one connected component. Warpers tend to react by fitting one region well and the other poorly, resulting in misaligned details (the buttons of Figure 1). Such errors may make little contribution to the training loss, but are very apparent and are considered severe problems by real users. We show that using multiple coordinated specialized warps produces substantial quantitative and qualitative improvements in warping. Our warper produces multiple warps, trained to coordinate with each other. An inpainting network combines the warps and the masked model, and creates a synthesized image. The inpainting network essentially learns to choose between the warps, while also provides guidance to the warper, as they are trained jointly. Qualitative evaluation confirms that an important part of the improvement results from better predictions of buttons, pockets, labels, and the like. We show large scale quantitative evaluations of virtual try-on. We collected a new dataset of 422,756 pairs of product images and studio photos by mining fashion e-commerce sites. The dataset contains multiple product categories. We compare with prior work on the established VITON dataset [17] both quantitatively and qualitatively. Quantitative result shows that choosing the productmodel pairs using our shape embedding yields significant improvements for all image generation pipelines (table 4.3). Using multiple warps also consistently outperform the single warp baseline, demonstrated through both quantitative (table 4.3, figure 5) and qualitative (figure 7) results. Qualitative comparison with prior work shows that our system preserves the details of both the tochange garment and the target model more accurately than prior work. We conducted a user study simulating the cost for e-commerce to replace real model with synthesized model. Result shows 40% of our synthesized model are thought as real models. As a summary of our contributions: – we introduce a matching procedure that results in significant qualitative and quantitative improvements in virtual try-on, whatever warper is used. – we introduce a warping model that learns multiple coordinated-warps and consistently outperforms baselines on all test sets. – our generated results preserve details accurately and realistically enough to make shoppers think that some of the synthesized images are real. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Authors: Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro Related Work Proposed Method Experiments Conclusions and References Abstract and Intro Abstract and Intro Related Work Related Work Proposed Method Proposed Method Experiments Experiments Conclusions and References Conclusions and References Abstract A virtual try-on method takes a product image and an image of a model and produces an image of the model wearing the product. Most methods essentially compute warps from the product image to the model image and combine using image generation methods. However, obtaining a realistic image is challenging because the kinematics of garments is complex and because outline, texture, and shading cues in the image reveal errors to human viewers. The garment must have appropriate drapes; texture must be warped to be consistent with the shape of a draped garment; small details (buttons, collars, lapels, pockets, etc.) must be placed appropriately on the garment, and so on. Evaluation is particularly difficult and is usually qualitative. This paper uses quantitative evaluation on a challenging, novel dataset to demonstrate that (a) for any warping method, one can choose target models automatically to improve results, and (b) learning multiple coordinated specialized warpers offers further improvements on results. Target models are chosen by a learned embedding procedure that predicts a representation of the products the model is wearing. This prediction is used to match products to models. Specialized warpers are trained by a method that encourages a second warper to perform well in locations where the first works poorly. The warps are then combined using a U-Net. Qualitative evaluation confirms that these improvements are wholesale over outline, texture shading, and garment details. Keywords: Fashion, Virtual try-on, Image generation, Image warping Keywords: 1. Introduction E-commerce means not being able to try on a product, which is difficult for fashion consumers [44]. Sites now routinely put up photoshoots of models wearing products, but volume and turnover mean doing so is very expensive and time consuming [34]. There is a need to generate realistic and accurate images of fashion models wearing different sets of clothing. One could use 3D models of posture [8,14]. The alternative – synthesize product-model images without 3D measurements [17,45,39,11,15] – is known as virtual try-on. These methods usually consist of two components: 1) a spatial transformer to warp the product image using some estimate of the model’s pose and 2) an image generation network that combines the coarsely aligned, warped product with the model image to produce a realistic image of the model wearing the product. It is much easier to transfer with simple garments like t-shirts, which are emphasized in the literature. General garments (unlike t-shirts) might open in front; have sophisticated drapes; have shaped structures like collars and cuffs; have buttons; and so on. These effects severely challenge existing methods (examples in Supplementary Materials). Warping is significantly improved if one uses the product image to choose a model image that is suited to that garment (Figure 1). At least in part, this is a result of how image generation networks are trained. We train using paired images – a product and a model wearing a product [17,45,53]. This means that the generation network always expects the target image to be appropriate for the product (so it is not trained to, for example, put a sweater onto a model wearing a dress, Figure 1). An alternative is to use adversarial training [11,12,38,13,37]; but it is hard to preserve specific product details (for example, a particular style of buttons; a decal on a t-shirt) in this framework. To deal with this difficulty, we learn an embedding space for choosing product-model pairs that will result in high-quality transfers (Figure 2). The embedding learns to predict what shape a garment in a model image would take if it were in a product image. Products are then matched to models wearing similarly shaped garments. Because models typically wear many garments, we use a spatial attention visual encoder to parse each category (top, bottom, outerwear, all-body, etc.) of garment and embed each separately. Another problem arises when a garment is open (for example, an unbuttoned coat). In this case, the target of the warp might have more than one connected component. Warpers tend to react by fitting one region well and the other poorly, resulting in misaligned details (the buttons of Figure 1). Such errors may make little contribution to the training loss, but are very apparent and are considered severe problems by real users. We show that using multiple coordinated specialized warps produces substantial quantitative and qualitative improvements in warping. Our warper produces multiple warps, trained to coordinate with each other. An inpainting network combines the warps and the masked model, and creates a synthesized image. The inpainting network essentially learns to choose between the warps, while also provides guidance to the warper, as they are trained jointly. Qualitative evaluation confirms that an important part of the improvement results from better predictions of buttons, pockets, labels, and the like. We show large scale quantitative evaluations of virtual try-on. We collected a new dataset of 422,756 pairs of product images and studio photos by mining fashion e-commerce sites. The dataset contains multiple product categories. We compare with prior work on the established VITON dataset [17] both quantitatively and qualitatively. Quantitative result shows that choosing the productmodel pairs using our shape embedding yields significant improvements for all image generation pipelines (table 4.3). Using multiple warps also consistently outperform the single warp baseline, demonstrated through both quantitative (table 4.3, figure 5) and qualitative (figure 7) results. Qualitative comparison with prior work shows that our system preserves the details of both the tochange garment and the target model more accurately than prior work. We conducted a user study simulating the cost for e-commerce to replace real model with synthesized model. Result shows 40% of our synthesized model are thought as real models. As a summary of our contributions: – we introduce a matching procedure that results in significant qualitative and quantitative improvements in virtual try-on, whatever warper is used. – we introduce a warping model that learns multiple coordinated-warps and consistently outperforms baselines on all test sets. – our generated results preserve details accurately and realistically enough to make shoppers think that some of the synthesized images are real. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Toward Accurate, Realistic Virtual Try-on Through Shape Matching: Abstract & Intro

শেপ ম্যাচিং এর মাধ্যমে সঠিক, বাস্তবসম্মত ভার্চুয়াল ট্রাই-অন: বিমূর্ত এবং ভূমিকা

Auf dem Weg zum präzisen, realistischen virtuellen Anprobieren durch Formabgleich: Zusammenfassung und Einführung

Hacia una prueba virtual precisa y realista mediante la combinación de formas: resumen e introducción

Vers un essai virtuel précis et réaliste grâce à la correspondance de formes : résumé et introduction

आकृति मिलान के माध्यम से सटीक, यथार्थवादी आभासी ट्राई-ऑन की ओर: सार और परिचय

形状マッチングによる正確でリアルなバーチャル試着の実現に向けて：概要と導入

도형 매칭을 통한 정확하고 사실적인 가상 시착을 향하여: 요약 및 소개

Rumo a um teste virtual preciso e realista por meio da correspondência de formas: resumo e introdução

На пути к точной и реалистичной виртуальной примерке посредством сопоставления форм: краткое содержание и введение

Şekil Eşleştirme Yoluyla Doğru, Gerçekçi Sanal Denemeye Doğru: Soyut ve Giriş

Hướng tới việc thử nghiệm ảo thực tế, chính xác thông qua việc khớp hình dạng: Tóm tắt & giới thiệu

通过形状匹配实现准确、逼真的虚拟试穿：摘要和简介

Authors:
(1) Kedan Li, University of Illinois at Urbana-Champaign;
(2) Min Jin Chong, University of Illinois at Urbana-Champaign;
(3) Jingen Liu, JD AI Research;
(4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro
Related Work
Proposed Method
Experiments
Conclusions and References 4. Experiments 4.1 Datasets The VITON dataset [17] contains pairs of product image (front-view, laying flat, white background) and studio images, 2D pose maps and pose key-points. It has been used by many works [45,11,15,53,24,22,2,37]. Some works [47,15,13,51] on multi-pose matching used DeepFashion [33] or MVC [32] and other self-collected datasets [12,21,47,55]. These datasets have the same product worn by multiple people, but do not have a product image, therefore not suitable for our task. The VITON dataset only has tops. This likely biases performance up, because (for example): the drape of trousers is different from the drape of tops; some garments (robes, jackets, etc.) are often unzipped and open, creating warping issues; the drape of skirts is highly variable, and depends on details like pleating, the orientation of fabric grain and so on. To emphasize these real-world problem, we collected a new dataset of 422,756 fashion products through web-scraping fashion e-commerce sites. Each product contains a product image (front-view, laying flat, white background), a model image (single person, mostly front-view), and other metadata. We use all categories except shoes and accessories, and group them into four types (top, bottoms, outerwear, or all-body). Type details appear in the supplementary materials. We randomly split the data into 80% for training and 20% for testing. Because the dataset does not come with segmentation annotation, we use Deeplab v3 [6] pre-trained on ModaNet dataset [56] to obtain the segmentation masks for model images. A large portion of the segmentation masks are noisy, which further increases the difficulty (see Supplementary Materials). 4.2 Training Process We train our model on our newly collected dataset and the VITON dataset [17] to facilitate comparison with prior work. When training our method on VITON dataset, we only extract the part of the 2D pose map that corresponds to the product to obtain segmentation mask, and discard the rest. The details of the training procedure is in Supplementary Materials. We also attempted to train prior works on our dataset. However, prior work [45,17,11,15,53,24,22,13,47,51,7,37] require pose estimation annotations which is not available in our dataset. Thus, we only compare with prior work on the VITON dataset. 4.3 Quantitative Evaluation Quantitative comparison with state of the art is difficult. Reporting the FID in other papers is meaningless, because the value is biased and the bias depends on the parameters of the network used [9,37]. We use the FID∞ score, which is unbiased. We cannot compute FID∞ for most other methods, because results have not been released; in fact, recent methods (eg [15,53,24,24,42,22,2]) have not released an implementation. CP-VTON [45] has, and we use this as a point of comparison. Most evaluation is qualitative, and others [24,37] also computed the FID score on the original test set of VITON, which consists of only 2,032 synthesized pairs. Because of the small dataset, this FID score is not meaningful. The variance arising from the calculation will be high which leads to a large bias in the FID score, rendering it inaccurate. To ensure an accurate comparison, we created a larger test set of synthesized 50,000 pairs through random matching, following the procedure of the original work [17]. We created new test sets using our shape matching model by selecting the top 25 nearest neighbors in the shape embedding space for every item in the original test set. We produce two datasets each of 50,000 pairs using colored image and grayscale images to compute the shape embedding. The grayscale ablation tells us whether the shape embedding looks at color features. The number of warps is chosen by computing the L1 error and Perceptual error (using VGG19 pre-trained on ImageNet) using warpers with different k on the test set of our dataset. Here the warper is evaluated by mapping a product to a model wearing that product. As shown in figure 5, k = 2 consistently out-performs k = 1. However, having more than two warps also reduce performance using the current training configuration, possibly due to overfitting. We choose β by training a single warp model with different β values using 10% of the dataset, then evaluating on test. Table 1 shows that a β that is too large or two small cause the performance to drop. β = 3 happens to be the best, and therefore is adopted. Qualitative comparison are available in supplementary materials. With this data, we can compare CP-VTON, our method using a single warp (k = 1), and two warps (k = 2), and two warp blended. The blended model takes in the average of two warps instead of the concatenation. Results appear in Table 4.3. We find: – for all methods, choosing the model gets better results; – there is little to choose between color and grayscale matching, so the match attends mainly to garment shape; – having two warpers is better than having one; – combining with a u-net is much better than blending. We believe that quantitative results understate the improvement of using more warpers, because the quantitative measure is relatively crude. Qualitative evidence supports this (figure 7). 4.4 Qualitative Results We have looked carefully for matching examples in [15,24,53,37] to produce qualitative comparisons. Comparison against MG-VTON [12] is not applicable, as the work did not include any fixed-pose qualitative example. Note that the comparison favors prior work because our model trains and tests only using the region corresponding to the garment in the 2D pose map while prior work uses the full 2D pose map and key-point pose annotations. Generally, garment transfer is hard, but modern methods now mainly fail on details. This means that evaluating transfer requires careful attention to detail. Figure 6 shows some comparisons. In particular, attending to image detail around boundaries, textures, and garment details exposes some of the difficulties in the task. As shown in Figure 6 left, our method can handle complicated texture robustly (col. a, c) and preserve details of the logo accurately (col. b, e, f, g, i). The examples also show clear difference between our inpainting-based method and prior work – our method only modifies the area where the original cloth is presented. This property allows us to preserve the details of the limb (col. a, d, f, g, h, j) and other clothing items (col. a, b) better than most prior work. Some of our results (col. c, g) show color artifacts from the original cloth on the boundary, because the edge of the pose map is slightly misaligned (imperfect segmentation mask). This confirms that our method rely on fine-grain segmentation mask to produce high quality result. Some pairs are slightly mis-matched in shape(col. d, h). This will rarely occur with our method if the test set is constructed using the shape embedding. Therefore, our method does not attempt to address it. Two warps are very clearly better than one (Figure 7), likely because the second warp can fix the alignment and details that single warp model fails to address. Particular improvements occur for unbuttoned/unzipped outerwear and for product images with tags. These improvement may not be easily captured by quantitative evaluation because the differences in pixel values are small. We attempted to train the geometric matching module (using TPS transform) to create warps on our dataset, as it was frequently adopted by prior work [17,45,11]. However, TPS transform failed to adapt to partitions and significant occlusions (examples in Supplementary Materials). 4.5 User Study We used a user study to check how often users could identify synthesized images. A user is asked whether an image of a model wearing a product (which is shown) is real or synthesized. Display uses the highest possible resolution (512x512), as in figure 8. We used examples where the mask is good, giving a fair representation of the top 20 percentile of our results. Users are primed with two real vs. fake pairs before the study. Each participant is then tested with 50 pairs of 25 real and 25 fake, without repeating products. We test two populations of users (vision researchers, and randomly selected participants). Mostly, users are fooled by our images; there is a very high false-positive (i.e. synthesized image marked real by a user) rate (table 3). Figure 8 shows two examples of synthesized images that 70% of the general population reported as real. They are hard outerwear examples with region partition and complex shading. Nevertheless, our method managed to generate high quality synthesis. See supplementary material for all questions and complete results of the user study. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Authors: Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro Related Work Proposed Method Experiments Conclusions and References Abstract and Intro Abstract and Intro Related Work Related Work Proposed Method Proposed Method Experiments Experiments Conclusions and References Conclusions and References 4. Experiments 4.1 Datasets The VITON dataset [17] contains pairs of product image (front-view, laying flat, white background) and studio images, 2D pose maps and pose key-points. It has been used by many works [45,11,15,53,24,22,2,37]. Some works [47,15,13,51] on multi-pose matching used DeepFashion [33] or MVC [32] and other self-collected datasets [12,21,47,55]. These datasets have the same product worn by multiple people, but do not have a product image, therefore not suitable for our task. The VITON dataset only has tops. This likely biases performance up, because (for example): the drape of trousers is different from the drape of tops; some garments (robes, jackets, etc.) are often unzipped and open, creating warping issues; the drape of skirts is highly variable, and depends on details like pleating, the orientation of fabric grain and so on. To emphasize these real-world problem, we collected a new dataset of 422,756 fashion products through web-scraping fashion e-commerce sites. Each product contains a product image (front-view, laying flat, white background), a model image (single person, mostly front-view), and other metadata. We use all categories except shoes and accessories, and group them into four types (top, bottoms, outerwear, or all-body). Type details appear in the supplementary materials. We randomly split the data into 80% for training and 20% for testing. Because the dataset does not come with segmentation annotation, we use Deeplab v3 [6] pre-trained on ModaNet dataset [56] to obtain the segmentation masks for model images. A large portion of the segmentation masks are noisy, which further increases the difficulty (see Supplementary Materials). 4.2 Training Process We train our model on our newly collected dataset and the VITON dataset [17] to facilitate comparison with prior work. When training our method on VITON dataset, we only extract the part of the 2D pose map that corresponds to the product to obtain segmentation mask, and discard the rest. The details of the training procedure is in Supplementary Materials. We also attempted to train prior works on our dataset. However, prior work [45,17,11,15,53,24,22,13,47,51,7,37] require pose estimation annotations which is not available in our dataset. Thus, we only compare with prior work on the VITON dataset. 4.3 Quantitative Evaluation Quantitative comparison with state of the art is difficult. Reporting the FID in other papers is meaningless, because the value is biased and the bias depends on the parameters of the network used [9,37]. We use the FID∞ score, which is unbiased. We cannot compute FID∞ for most other methods, because results have not been released; in fact, recent methods (eg [15,53,24,24,42,22,2]) have not released an implementation. CP-VTON [45] has, and we use this as a point of comparison. Most evaluation is qualitative, and others [24,37] also computed the FID score on the original test set of VITON, which consists of only 2,032 synthesized pairs. Because of the small dataset, this FID score is not meaningful. The variance arising from the calculation will be high which leads to a large bias in the FID score, rendering it inaccurate. To ensure an accurate comparison, we created a larger test set of synthesized 50,000 pairs through random matching, following the procedure of the original work [17]. We created new test sets using our shape matching model by selecting the top 25 nearest neighbors in the shape embedding space for every item in the original test set. We produce two datasets each of 50,000 pairs using colored image and grayscale images to compute the shape embedding. The grayscale ablation tells us whether the shape embedding looks at color features. The number of warps is chosen by computing the L1 error and Perceptual error (using VGG19 pre-trained on ImageNet) using warpers with different k on the test set of our dataset. Here the warper is evaluated by mapping a product to a model wearing that product. As shown in figure 5, k = 2 consistently out-performs k = 1. However, having more than two warps also reduce performance using the current training configuration, possibly due to overfitting. We choose β by training a single warp model with different β values using 10% of the dataset, then evaluating on test. Table 1 shows that a β that is too large or two small cause the performance to drop. β = 3 happens to be the best, and therefore is adopted. Qualitative comparison are available in supplementary materials. With this data, we can compare CP-VTON, our method using a single warp (k = 1), and two warps (k = 2), and two warp blended. The blended model takes in the average of two warps instead of the concatenation. Results appear in Table 4.3. We find: – for all methods, choosing the model gets better results; – there is little to choose between color and grayscale matching, so the match attends mainly to garment shape; – having two warpers is better than having one; – combining with a u-net is much better than blending. We believe that quantitative results understate the improvement of using more warpers, because the quantitative measure is relatively crude. Qualitative evidence supports this (figure 7). 4.4 Qualitative Results We have looked carefully for matching examples in [15,24,53,37] to produce qualitative comparisons. Comparison against MG-VTON [12] is not applicable, as the work did not include any fixed-pose qualitative example. Note that the comparison favors prior work because our model trains and tests only using the region corresponding to the garment in the 2D pose map while prior work uses the full 2D pose map and key-point pose annotations. Generally, garment transfer is hard, but modern methods now mainly fail on details. This means that evaluating transfer requires careful attention to detail. Figure 6 shows some comparisons. In particular, attending to image detail around boundaries, textures, and garment details exposes some of the difficulties in the task. As shown in Figure 6 left, our method can handle complicated texture robustly (col. a, c) and preserve details of the logo accurately (col. b, e, f, g, i). The examples also show clear difference between our inpainting-based method and prior work – our method only modifies the area where the original cloth is presented. This property allows us to preserve the details of the limb (col. a, d, f, g, h, j) and other clothing items (col. a, b) better than most prior work. Some of our results (col. c, g) show color artifacts from the original cloth on the boundary, because the edge of the pose map is slightly misaligned (imperfect segmentation mask). This confirms that our method rely on fine-grain segmentation mask to produce high quality result. Some pairs are slightly mis-matched in shape(col. d, h). This will rarely occur with our method if the test set is constructed using the shape embedding. Therefore, our method does not attempt to address it. Two warps are very clearly better than one (Figure 7), likely because the second warp can fix the alignment and details that single warp model fails to address. Particular improvements occur for unbuttoned/unzipped outerwear and for product images with tags. These improvement may not be easily captured by quantitative evaluation because the differences in pixel values are small. We attempted to train the geometric matching module (using TPS transform) to create warps on our dataset, as it was frequently adopted by prior work [17,45,11]. However, TPS transform failed to adapt to partitions and significant occlusions (examples in Supplementary Materials). 4.5 User Study We used a user study to check how often users could identify synthesized images. A user is asked whether an image of a model wearing a product (which is shown) is real or synthesized. Display uses the highest possible resolution (512x512), as in figure 8. We used examples where the mask is good, giving a fair representation of the top 20 percentile of our results. Users are primed with two real vs. fake pairs before the study. Each participant is then tested with 50 pairs of 25 real and 25 fake, without repeating products. We test two populations of users (vision researchers, and randomly selected participants). Mostly, users are fooled by our images; there is a very high false-positive (i.e. synthesized image marked real by a user) rate (table 3). Figure 8 shows two examples of synthesized images that 70% of the general population reported as real. They are hard outerwear examples with region partition and complex shading. Nevertheless, our method managed to generate high quality synthesis. See supplementary material for all questions and complete results of the user study. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Toward Accurate, Realistic Virtual Try-on Through Shape Matching: Experiments

সঠিক, বাস্তবসম্মত ভার্চুয়াল ট্রাই-অন থ্রু শেপ ম্যাচিং: পরীক্ষা-নিরীক্ষার দিকে

Auf dem Weg zur präzisen, realistischen virtuellen Anprobe durch Formabgleich: Experimente

Hacia una prueba virtual precisa y realista mediante la combinación de formas: experimentos

Vers un essai virtuel précis et réaliste grâce à la correspondance de formes : expérimentations

आकार मिलान के माध्यम से सटीक, यथार्थवादी आभासी ट्राई-ऑन की ओर: प्रयोग

形状マッチングによる正確でリアルなバーチャル試着に向けて：実験

모양 매칭을 통한 정확하고 사실적인 가상 시착을 향한 실험

Rumo a um teste virtual preciso e realista por meio da correspondência de formas: experimentos

На пути к точной и реалистичной виртуальной примерке посредством сопоставления форм: эксперименты

Şekil Eşleştirme Yoluyla Doğru, Gerçekçi Sanal Denemeye Doğru: Deneyler

Hướng tới việc thử nghiệm ảo thực tế, chính xác thông qua việc khớp hình dạng: Thử nghiệm

通过形状匹配实现准确、逼真的虚拟试穿：实验

Toward Accurate, Realistic Virtual Try-on Through Shape Matching: Conclusions & References

সঠিক, বাস্তবসম্মত ভার্চুয়াল ট্রাই-অন থ্রু শেপ ম্যাচিং এর দিকে: উপসংহার এবং রেফারেন্স

Auf dem Weg zur präzisen, realistischen virtuellen Anprobe durch Formabgleich: Schlussfolgerungen und Referenzen

Hacia una prueba virtual precisa y realista mediante la combinación de formas: conclusiones y referencias

Vers un essai virtuel précis et réaliste grâce à la correspondance de formes : conclusions et références

आकार मिलान के माध्यम से सटीक, यथार्थवादी आभासी ट्राई-ऑन की ओर: निष्कर्ष और संदर्भ

形状マッチングによる正確でリアルなバーチャル試着に向けて：結論と参考文献

모양 매칭을 통한 정확하고 사실적인 가상 시착을 향하여: 결론 및 참고자료

Rumo a um teste virtual preciso e realista por meio da correspondência de formas: conclusões e referências

На пути к точной и реалистичной виртуальной примерке посредством сопоставления форм: выводы и ссылки

Şekil Eşleştirme Yoluyla Doğru, Gerçekçi Sanal Denemeye Doğru: Sonuçlar ve Referanslar

Hướng tới việc thử ảo chính xác, thực tế thông qua việc khớp hình dạng: Kết luận và tài liệu tham khảo

通过形状匹配实现准确、逼真的虚拟试穿：结论与参考文献

Authors:
(1) Kedan Li, University of Illinois at Urbana-Champaign;
(2) Min Jin Chong, University of Illinois at Urbana-Champaign;
(3) Jingen Liu, JD AI Research;
(4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro
Related Work
Proposed Method
Experiments
Conclusions and References 3. Proposed Method Our method has two components. A Shape Matching Net (SMN; Figure 2 and 3) learns an embedding for choosing shape-wise compatible garment-model pairs to perform transfer. Product and model images are matched by finding product (resp. model) images that are nearby in the embedding space. A Multi-warp Try-on Net (MTN; Figure 4) takes in a garment image, a model image and a mask covering the to-change garment on the model and generates a realistic synthesis image of the model wearing the provided garment. The network consists of a warper and an inpainting network, trained jointly. The warper produces k warps of the product image, each specialized on certain features. The inpainting network learns to combine warps by choosing which features to look for from each warp. SMN and MTN are trained separately. 3.1 Shape Matching Net The embedding loss is used to capture the feature correspondence of the two domain and help enforce the attention mechanism embed in the network architecture. Details about the spatial attention architecture are in Supplementary Materials. 3.2 Multi-warp Try-on Net As with prior work [17,45], our system also consists of two modules: (a) a warper to create multiple specialized warps, by aligning the product image with the mask; (b) an inpainting module to combine the warps with the masked model and produce the synthesis image. Unlike prior work [17,45], the two modules are trained jointly rather than separately, so the inpainter guides the warper. Cascade Loss: With multiple warps, each warp wi is trained to address the mistakes made by previous warps wj where j < i. For the k th warp, we compute the minimum loss among all the previous warps at every pixel, written as The cascade loss computes the average loss for all warps. An additional regularization terms is enforced on the transformation parameters, so all the later warps stay close to the first warp. The cascade loss enforce a hierarchy among all warps, making it more costly for an earlier warp to make a mistake than for a later warp. This prevents possible oscillation during the training (multiple warps compete for optimal). The idea is comparable with boosting, but yet different because all the warps share gradient, making it possible for earlier warps to adjust according to later warps. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Authors: Authors: (1) Kedan Li, University of Illinois at Urbana-Champaign; (2) Min Jin Chong, University of Illinois at Urbana-Champaign; (3) Jingen Liu, JD AI Research; (4) David Forsyth, University of Illinois at Urbana-Champaign. Table of Links Abstract and Intro Related Work Proposed Method Experiments Conclusions and References Abstract and Intro Abstract and Intro Related Work Related Work Proposed Method Proposed Method Experiments Experiments Conclusions and References Conclusions and References 3. Proposed Method Our method has two components. A Shape Matching Net (SMN; Figure 2 and 3) learns an embedding for choosing shape-wise compatible garment-model pairs to perform transfer. Product and model images are matched by finding product (resp. model) images that are nearby in the embedding space. A Multi-warp Try-on Net (MTN; Figure 4) takes in a garment image, a model image and a mask covering the to-change garment on the model and generates a realistic synthesis image of the model wearing the provided garment. The network consists of a warper and an inpainting network, trained jointly. The warper produces k warps of the product image, each specialized on certain features. The inpainting network learns to combine warps by choosing which features to look for from each warp. SMN and MTN are trained separately. Shape Matching Net Multi-warp Try-on Net 3.1 Shape Matching Net The embedding loss is used to capture the feature correspondence of the two domain and help enforce the attention mechanism embed in the network architecture. Details about the spatial attention architecture are in Supplementary Materials. 3.2 Multi-warp Try-on Net As with prior work [17,45], our system also consists of two modules: (a) a warper to create multiple specialized warps , by aligning the product image with the mask; (b) an inpainting module to combine the warps with the masked model and produce the synthesis image. Unlike prior work [17,45], the two modules are trained jointly rather than separately, so the inpainter guides the warper. multiple specialized warps Cascade Loss: With multiple warps, each warp wi is trained to address the mistakes made by previous warps wj where j < i. For the k th warp, we compute the minimum loss among all the previous warps at every pixel, written as Cascade Loss: The cascade loss computes the average loss for all warps. An additional regularization terms is enforced on the transformation parameters, so all the later warps stay close to the first warp. The cascade loss enforce a hierarchy among all warps, making it more costly for an earlier warp to make a mistake than for a later warp. This prevents possible oscillation during the training (multiple warps compete for optimal). The idea is comparable with boosting, but yet different because all the warps share gradient, making it possible for earlier warps to adjust according to later warps. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Toward Accurate, Realistic Virtual Try-on Through Shape Matching: Proposed Method

শেপ ম্যাচিং এর মাধ্যমে সঠিক, বাস্তবসম্মত ভার্চুয়াল ট্রাই-অন: প্রস্তাবিত পদ্ধতি

Auf dem Weg zur präzisen, realistischen virtuellen Anprobe durch Formabgleich: Vorgeschlagene Methode

Hacia una prueba virtual precisa y realista mediante la combinación de formas: método propuesto

Vers un essai virtuel précis et réaliste grâce à la correspondance de forme : méthode proposée

आकार मिलान के माध्यम से सटीक, यथार्थवादी आभासी प्रयास की ओर: प्रस्तावित विधि

形状マッチングによる正確でリアルな仮想試着に向けて：提案手法

형상 매칭을 통한 정확하고 사실적인 가상 시착을 위한 방법 제안

Rumo a um teste virtual preciso e realista por meio da correspondência de formas: método proposto

На пути к точной и реалистичной виртуальной примерке посредством сопоставления форм: предлагаемый метод

Şekil Eşleştirme Yoluyla Doğru, Gerçekçi Sanal Denemeye Doğru: Önerilen Yöntem

Hướng tới việc thử nghiệm ảo chính xác, thực tế thông qua việc khớp hình dạng: Phương pháp được đề xuất

通过形状匹配实现准确、逼真的虚拟试穿：建议方法

Authors:
(1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com};
(2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Table of Links Abstract and Intro Dataset Design Evaluation Methodology Experiments and Analysis Related Work Conclusion, Reproducibility, and References A. Complete list of tools B. Scenario Prompt C. Unrealistic Queries D. Nuances comparing prior work 2 DATASET DESIGN 2.1 PLUGINS AND TOOLS ToolTalk is designed for a paradigm where individual users will be able to customize a personal assistant with a number of plugins available through various online stores. This can be seen as similar to how a user might customize their phone with apps of various functionality. Each plugin contains a set of tools designed around a single purpose such as managing a calendar, buying movie tickets, or listening to music. We define a tool as a single function needed to accomplish that purpose such as creating an event, searching for movies, or playing a song. We assume that most plugins will need to contain multiple tools. For example, a theoretical “Calendar” plugin should not only have the ability to create events, but also to then search, modify, and delete these events. For our dataset, we defined 7 plugins containing a total of 28 tools (see Appendix A for the full list). Using similar domains as those in Li et al. (2023), we created the following plugins: • AccountTools: containing tools for account management such as logging in and out, updating account information, or looking up other users. • Alarm: adding, deleting, and finding alarms. • Calendar: creating, modifying, deleting, and searching events and meetings • Email: searching inbox and sending emails • Message: sending and reading messages from other users • Reminder: setting, completing, and deleting reminders on a to do list • Weather: querying current weather, weather forecasts, and historic weather data based on location To teach the LLM about how to use the tools, each tool contains a high-level description, verbose documentation about each of its parameters, and a description of its return value. To facilitate evaluation, each tool has a simulated implementation in Python, along with a method to judge whether two invocations of the same tool with different parameters should be considered equivalent. We also note for each tool whether it is considered an action (has side effects) or not. We also include accompanying databases with mock information about fictional existing users, emails, reminders, and so on, for the simulated tool implementations to use. 2.2 CREATING CONVERSATIONS To help create realistic conversations that exercise our tools and plugins, we used GPT-4. For each subset of 3 plugins from the 7 plugins we have defined, we create prompts which lists the documentation for all the tools in these 3 plugins, and instructs GPT-4 to create 3 realistic scenarios involving a user trying to accomplish a task that uses at least 5 tool calls from the random subset of plugins. We create as many prompts as the number of tools that exist in the subset of 3 plugins currently under consideration, such that each prompt instructs GPT-4 to specifically use one of the tools in the subset of 3 plugins. We provide the prompt template used in Appendix B. The above procedure results in the generation of ∼400 scenarios. We then repeatedly sampled a scenario evenly from all tools, discarding sampled scenarios that do not involve the required tool, hallucinate non-existent tools, or seem implausible. Using a sampled scenario as general guidance, we manually create a conversation, writing down all of its parts by hand. Each conversation consists of a user utterance, the tool calls that the assistant should make given that utterance, the expected return values for those calls, and the assistant’s natural language responses given the user’s utterances plus the tool calls and their results, repeating in that order until the conversation is finished. As metadata for the conversation, we also specified a timestamp for the conversation, and the user’s location and username.[1] We ensure that each conversation contains at least 3 tool calls. We repeat the above sampling of scenarios until we have written 50 conversations. Additionally, we create 28 “easy” conversations completely by hand, one for each tool. This easy version of ToolTalk consists of a few turns of user-assistant dialogue followed by a single tool call. Combined with the prior 50 “hard” examples, we create a total of 78 conversations comprising ToolTalk. After constructing conversations, we ensure that the databases used by our simulated tool implementations contain the necessary content so that when we execute the ground truth tool calls as listed in the conversations we have created, they return the same ground truth values. This paper is available on arxiv under CC 4.0 license. [1] For scenarios that use tools such as UserLogin or RegisterAccount, we omit the username to simulate a user that has yet to log in or have an account. Authors: (1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com}; (2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Authors: Authors: (1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com}; (2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Table of Links Abstract and Intro Abstract and Intro Dataset Design Dataset Design Evaluation Methodology Evaluation Methodology Experiments and Analysis Experiments and Analysis Related Work Related Work Conclusion, Reproducibility, and References Conclusion, Reproducibility, and References A. Complete list of tools A. Complete list of tools B. Scenario Prompt B. Scenario Prompt C. Unrealistic Queries C. Unrealistic Queries D. Nuances comparing prior work D. Nuances comparing prior work 2 DATASET DESIGN 2.1 PLUGINS AND TOOLS ToolTalk is designed for a paradigm where individual users will be able to customize a personal assistant with a number of plugins available through various online stores. This can be seen as similar to how a user might customize their phone with apps of various functionality. Each plugin contains a set of tools designed around a single purpose such as managing a calendar, buying movie tickets, or listening to music. We define a tool as a single function needed to accomplish that purpose such as creating an event, searching for movies, or playing a song. We assume that most plugins will need to contain multiple tools. For example, a theoretical “Calendar” plugin should not only have the ability to create events, but also to then search, modify, and delete these events. For our dataset, we defined 7 plugins containing a total of 28 tools (see Appendix A for the full list). Using similar domains as those in Li et al. (2023), we created the following plugins: • AccountTools : containing tools for account management such as logging in and out, updating account information, or looking up other users. AccountTools • Alarm : adding, deleting, and finding alarms. Alarm • Calendar : creating, modifying, deleting, and searching events and meetings Calendar • Email : searching inbox and sending emails Email • Message : sending and reading messages from other users Message • Reminder : setting, completing, and deleting reminders on a to do list Reminder • Weather : querying current weather, weather forecasts, and historic weather data based on location Weather To teach the LLM about how to use the tools, each tool contains a high-level description, verbose documentation about each of its parameters, and a description of its return value. To facilitate evaluation, each tool has a simulated implementation in Python, along with a method to judge whether two invocations of the same tool with different parameters should be considered equivalent. We also note for each tool whether it is considered an action (has side effects) or not. We also include accompanying databases with mock information about fictional existing users, emails, reminders, and so on, for the simulated tool implementations to use. 2.2 CREATING CONVERSATIONS To help create realistic conversations that exercise our tools and plugins, we used GPT-4. For each subset of 3 plugins from the 7 plugins we have defined, we create prompts which lists the documentation for all the tools in these 3 plugins, and instructs GPT-4 to create 3 realistic scenarios involving a user trying to accomplish a task that uses at least 5 tool calls from the random subset of plugins. We create as many prompts as the number of tools that exist in the subset of 3 plugins currently under consideration, such that each prompt instructs GPT-4 to specifically use one of the tools in the subset of 3 plugins. We provide the prompt template used in Appendix B. The above procedure results in the generation of ∼400 scenarios. We then repeatedly sampled a scenario evenly from all tools, discarding sampled scenarios that do not involve the required tool, hallucinate non-existent tools, or seem implausible. Using a sampled scenario as general guidance, we manually create a conversation, writing down all of its parts by hand. Each conversation consists of a user utterance, the tool calls that the assistant should make given that utterance, the expected return values for those calls, and the assistant’s natural language responses given the user’s utterances plus the tool calls and their results, repeating in that order until the conversation is finished. As metadata for the conversation, we also specified a timestamp for the conversation, and the user’s location and username.[1] We ensure that each conversation contains at least 3 tool calls. We repeat the above sampling of scenarios until we have written 50 conversations. Additionally, we create 28 “easy” conversations completely by hand, one for each tool. This easy version of ToolTalk consists of a few turns of user-assistant dialogue followed by a single tool call. Combined with the prior 50 “hard” examples, we create a total of 78 conversations comprising ToolTalk. After constructing conversations, we ensure that the databases used by our simulated tool implementations contain the necessary content so that when we execute the ground truth tool calls as listed in the conversations we have created, they return the same ground truth values. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv [1] For scenarios that use tools such as UserLogin or RegisterAccount, we omit the username to simulate a user that has yet to log in or have an account.

How to Create Realistic AI Conversations

If New Year resolutions don’t work for you, you might find this article helpful January is an insane month for regular gym-goers because the rest of the world sets a not-so-interesting NY goal — start working out. But it goes a little bit like this 👇 I intentionally wrote this article a month later. By now most resolutionists would have fallen off the wagon and begun feeling bad about themselves. That’s because their resolutions were too unrealistic or unmanageable. I will start working out 5x a week, make 100k this year, or learn Spanish are good, but not realistic goals. Not because it’s not possible but because that’s not how our minds work. Learn to gamify The reason video games are so addicting is because they offer small rewards along the way says . Anyone we deem lazy can play video games for hours. Are they lazy or are their real-life incentives boring? George Mack Video game writers are the real psychologists. They set up a big goal (winning the game), but it requires overcoming increasingly difficult obstacles (advancing through levels) whilst also offering mini-rewards within each level (the enjoyment of a bad guy screaming as he falls from your shot). Level 10 is more fun compared to level 1 because you experience increasing excitement, reward, challenge, and meaning. If you want to motivate humans, frequent rewards are more addicting than one-off rewards — George Mack Start small. Set up small goals, incentives, and rewards along the way to your desired goal. If you haven’t worked out properly don’t set out to hit the gym 5x a week. This may be the goal you’re aiming to achieve by the end of the year but start small. Imbibe a one-goal system Instead of choosing multiple goals, write down one goal for each area and only start with one goal at a time. Health (sleep, exercise, nutrition) Career/work (promotion, starting a business, career transition) Finances (investing, passive income, salary) Mental health (meditation, self-compassion, therapy) Romantic Relationship (dating, more quality time w/your partner). Learning (a new language, programming, knitting), etc. You will not be able to implement the changes all at once. It’s too difficult to leave the comfort zone. But if you pick one and work on it for a month, it’ll become a habit. You’ll then have the bandwidth to tackle another goal. For example, I’ve decided that in 2024 I’ll fix my posture. My scapulas are winging and I can’t do a single push-up. My upper back is weak which left me constantly aching. So on on the 2nd of December, I started following a daily routine. health The area of improvement: be able to do 10 push-ups at the end of the year. The big goal: targeted, 30 mins back exercises twice a day. The small daily goal: taking pictures of my scapulas monthly to see progress, no more upper back pain (incredibly rewarding), improved breathing, and a sense of self-esteem in committing to my health daily. The rewards: By the time I reached the 30-day mark, upper back exercises became part of my routine. I’ve already seen progress and reaped rewards (less pain) to keep going so I can tackle another area now. Another not-so-unexpected outcome is that I exercise more. Once I’m done with my back routine, I want to keep going. Good habits feed off of each other. P.S. The daily goal could be as small as your circumstances require. It could be doing 5 minutes of stretching if you’re a mom of three. Consistency is its own reward. Moving from head to heart This is a fancy way of explaining how to close the gap between what we know we should be doing and actually doing it. Moving from head to heart requires accepting a few basic things and you will experience pain, failure, and disappointment. You are a fallible human being but it becomes when you can trust yourself. You earn self-trust with your actions (doing what you say you would). You have intrinsic worth experiential . They are not valid nor invalid — feelings just are and they’re often irrational. Your feelings are not reality with power, agency, and responsibility. Your ‘triggers’ are yours to fix. You are an adult Mental health tools to adopt Learn to meet your own needs and self-soothe. If you’re hyper-independent, learn to ask for help and be vulnerable. Develop an awareness of your emotional needs and responses. . Recognize when you’re chasing ideals that don’t align with your true self due to societal or parental expectations. Be aware of how society’s conditions of worth impact self-esteem and behavior , where self-worth is not dependent on meeting external conditions. Set your standards. Strive for unconditional positive self-regard : Understand that everyone is striving towards something and their ‘negative’ behaviors are often influenced by unmet needs, expectations, and insecurities. Cultivate empathy and compassion for others It’s the little things that count. Perhaps there’s a local cat that needs feeding. Or a lonely neighbor who would benefit from a 10-minute chat. Maybe you could cook for a friend who’s having a busy time at work. When we do good to others, we feel about ourselves. Perform acts of service. good Good mental health is not a goal per se — it’s a foundation for fulfilling your goals. To sum up Gamify your goals by setting small, incremental rewards along the way. Pick one goal, work on it daily until it becomes a habit and then move on to the next. Review and focus on your relationship with yourself. What are your goals for 2024 and how has it been going? Go easy on yourself, Rima ♥️ P.S. This is from my , if you like to receive it directly in your inbox, go . newsletter here Also published . here

How to Set and Attain Realistic Goals

The HackerNoon editorial team has launched to celebrate their achievements and share their struggles. We need more women in technology, and by sharing stories, we can encourage many girls to follow their dreams. ! this interview series with women in tech Share your story today Tell us about yourself! First and foremost, I am a proud woman and CEO at Realiste MENA, a dynamic prop-tech company headquartered in Dubai. Secondly, I am a perfectionist, although I try to actively maintain a healthy balance between my work and personal life. I am 24 years old, and I feel younger every day as I squeeze in exercise whenever I can. I am currently at the helm of the company, shaping the strategic vision and direction for Realiste across the MENA region. Most importantly, I am grateful to have found a job that challenges me yet inspires me every day. Why did you choose this field in the first place? When I discovered Realiste AI and its mission to revolutionize real estate investment by providing comprehensive data and insights, I was instantly drawn to the potential it holds. I usually try to keep myself well-read and up to date on new innovations across multiple industries, and I simply had a feeling this company is the future. The real estate market has long been shrouded in uncertainty, and AI undoubtedly has the incredible capacity to bring transparency and efficiency to it. This is why I decided to pursue a career in this field – to be at the forefront of a technological revolution that not only benefits investors but also makes a positive impact on the real estate industry as a whole. What tech are you most excited/passionate about right now and why? I'm deeply fascinated by the transformative power of artificial intelligence. AI seamlessly integrates into our lives, simplifying complex tasks. In real estate, for example, the projected market volume for residential real estate in 2023 is a staggering US $498.60tn. This vast and complex industry is where AI's potential truly shines. With AI, we can process extensive data, analyze market trends, and provide precise forecasts at a fraction of the time it takes humans. This boosts efficiency and minimizes risk in real estate investments. My passion for AI aligns perfectly with the promising future of the real estate market, and I'm truly eager to be part of this transformative journey. What tech are you most worried about right now and why? I'm currently most concerned about the ethical implications of deepfake technology. Deepfakes have advanced to the point where they can convincingly manipulate audio and video to create fabricated content that is nearly indistinguishable from reality. While this technology has the potential for creative and positive applications, there's also the possibility of its misuse for misleading or deceptive purposes. Striking a balance between harnessing the benefits of deepfake technology and safeguarding against its potential negative consequences is a challenge that deserves a great deal of attention. What are your hobbies and interests outside of tech? Outside of the tech world, I have a range of hobbies and interests that keep me balanced and fulfilled. I'm an avid sports enthusiast, particularly enjoying playing tennis and hitting the gym to stay active. In my downtime, I love to delve into historical books, as I am a firm believer that knowing and understanding the past grants a greater control over the present. And, of course, spending quality time with my lovely dog is a cherished pastime that brings me so much peace and joy. Let's talk about breaking the glass ceiling. What were the biggest challenges you faced as a woman in tech, and how did you deal with them? One significant challenge has been the underrepresentation of women in leadership roles within tech companies. It can be disheartening to look around and see a lack of female mentors and role models. To deal with this, I actively sought out networking opportunities and connected with successful women in the industry who could offer guidance and support. It was difficult, yet now, the entire world is becoming more "women-friendly", so hopefully, younger girls won't feel as lonely when they integrate into the workforce. Any questionable misogynistic story/situation you faced/handled, and you want to share with the HackerNoon Fam? In the company where I work, in fact, almost 50% of the employees are women, and they are highly valued and listened to equally. That's why I love my team. However, we work in the Middle East, where discrimination against women's rights is still present, even on a legislative level. But the world is not standing still, and governments are making efforts to eradicate gender inequality. For example, in the past, women in Saudi Arabia were not allowed to drive, but this rule changed just a few years ago. What was the biggest setback/failure that you faced, and how did you manage it? During my previous role in the realm of communications and public relations, I faced a few hurdles. Some plans and projects did not pan out as well as I'd planned and it took some time to accept these mistakes. Nevertheless, I've made it a personal commitment to look at such situations from unconventional angles and seek out innovative solutions that weren't readily apparent at first glance. This approach has consistently proven to be a powerful tool, helping me not only overcome obstacles but also propel myself forward. What's your biggest achievement that you're really proud of? Thanks to my strong communication skills, I have assisted over 50 corporations and technological startups with their execution and communication needs. Additionally, I’ve successfully established connections with some royal families in the MENA region and orchestrated the creation of an advisory board at Realiste. This board comprises high-ranking officials and top executives from major real estate developers in the UAE and beyond. This experience has not only been a remarkable accomplishment but has also deepened my conviction in the region's future potential. In your opinion, why do we see this huge gender gap in the tech industry, and how can we reduce it? As we know, the tech industry has historically been male-dominated, which has perpetuated a lack of diversity. In addition, unconscious bias can affect hiring and promotion decisions, making it harder for women to advance in tech careers. So, to address the gender gap in the tech industry, we should promote STEM education for girls from an early age, advocate for gender equality in leadership positions, and highlight successful female role models in tech. Who is your tech idol? Why? I really respect Grace Hopper. Hopper was a trailblazing computer scientist known for her work on the first compiler and for her contributions to the development of programming languages. I remember perusing the internet in my downtime as a kid, and I just stumbled upon her Wikipedia page. To be completely honest, I was shocked I didn't know who she was sooner, seeing that she is greatly revered as both a woman and a mathematician. I read up more, and I watched a lecture or two. I didn't completely understand but I was so enamored by her aura. I think she's an inspiration to many for her achievements in a male-dominated field. Do you have any advice for aspiring girls who want to join the field? I’d recommend finding what excites girls within tech – whether it's machine learning, design, data analysis, or another area. Pursue passions, and work will feel less like a job. And don't be afraid to challenge stereotypes because unique perspectives and experiences can be assets to any industry.

Interview With Anastasia Denisova, CEO at Realiste MENA: Breaking the Glass Ceiling

https://www.youtube.com/watch?v=MVYrJJNdrEg&embedable=true&transcript=true

The Mark Zuckerberg Lex Friedman Hyper Realistic Metaverse Interview [Full Transcript] 

Stories, News & Mentions about REELIST

Reelist’s Stories on HackerNoon