Baselime

baselime

TRANSIC Demolishes Competition: 81% Success Rate vs 45% for Best Baseline

Baseline Models for Single-Cell RNA-seq Dimensionality Reduction

Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Method and 3.1 Phase 1: Taxonomy Generation 3 Method and 3.1 Phase 1: Taxonomy Generation 3.2 Phase 2: LLM-Augmented Text Classification 3.2 Phase 2: LLM-Augmented Text Classification 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4 Evaluation Suite and 4.1 Phase 1 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 4.2 Phase 2 Evaluation Strategies 5 Experiments and 5.1 Data 5 Experiments and 5.1 Data 5.2 Taxonomy Generation 5.2 Taxonomy Generation 5.3 LLM-Augmented Text Classification 5.3 LLM-Augmented Text Classification 5.4 Summary of Findings and Suggestions 5.4 Summary of Findings and Suggestions 6 Discussion and Future Work, and References 6 Discussion and Future Work, and References A. Taxonomies A. Taxonomies B. Additional Results B. Additional Results C. Implementation Details C. Implementation Details D. Prompt Templates D. Prompt Templates 5.2 Taxonomy Generation 5.2.1 Methods. To evaluate the effectiveness of TnT-LLM, we compare it with baseline methods that rely on embedding-based clustering to group conversations and then assigns LLM-generated labels to each cluster. We use two state-of-the-art LLMs, GPT4 (0613) and GPT-3.5-Turbo (0613), as label generators and evaluators, and two different embedding methods, ada2[2] and Instructor-XL [26], to represent the conversations. The methods considered in our experiments are as follows: TnT-LLM GPT4 (0613) GPT-3.5-Turbo (0613) ada2 Instructor-XL • GPT-4 (TnT-LLM): the proposed TnT-LLM with GPT-4 to perform label taxonomy generation and assignment. • GPT-4 (TnT-LLM): • GPT-3.5 (TnT-LLM): the proposed TnT-LLM with GPT-3.5- Turbo to perform label taxonomy generation and assignment. • GPT-3.5 (TnT-LLM): • ada2 + GPT-4: the embedding-based clustering approach where conversations are represented via ada2 and K-means algorithm is applied to generate clusters. We randomly sample 200 conversations within each cluster, prompt GPT-4 to summarize each conversation, then ask it to produce a label name and description from these summaries, conditioned on the use-case instruction. • ada2 + GPT-4: • ada2 + GPT-3.5-Turbo: similar to the above method, with GPT-3.5-Turbo as the label generator. • Instructor-XL + GPT-4: similar to the above embeddingbased methods, with Instructor-XL and GPT-4 as the underlying embedding and the label generator respectively. • ada2 + GPT-3.5-Turbo: • Instructor-XL + GPT-3.5-Turbo: similar to the above method, with GPT-3.5-Turbo as the label generator. • Instructor-XL + GPT-3.5-Turbo: Note that all the taxonomies evaluated in this section are fully automatic and do not involve any human intervention. 5.2.2 Implementation Details. We instruct our LLMs to generate 10 intent categories and 25 domain categories for taxonomy generation. Likewise, we learn 10 intent clusters and 25 domain clusters with our embedding-based baselines. We use a minibatch size of 200 for our proposed taxonomy generation pipeline. We also apply a minibatch version of the K-means algorithm in all embeddingbased clustering approaches, where the same batch size is used with a K-means++ [2] initialization. We run 10 different trials of the clustering algorithm and select the best one based on the Silhouette coefficient [21] on the validation set. We also devise a “model” selection prompt, which takes a batch of conversation summaries, 5.2.2 Implementation Details. multiple label taxonomies, a use-case instruction as input, then outputs the index of the taxonomy that best fits the data and the instructional desiderata. We then run TnT-LLM 10 trials and select the best outcome based on its performance on the validation set. Human Evaluation. To evaluate the quality of generated taxonomies from methods listed above, three of the authors performed the label accuracy and use-case relevance tasks; each conversation was evaluated by all three raters. While raters possessed a high degree of familiarity with the Bing Copilot system, as well as the desired use-cases, they were unaware of the correspondence between methods and their generated labels. The position of the options in the pairwise comparison label accuracy task is also fully randomized. We also use two LLM systems, GPT-4 and GPT-3.5-Turbo, to perform the same evaluation tasks as the human raters. However, we notice that the LLM systems tend to exhibit a position bias [16] for the pairwise comparison task, where they favor one option over another based on its position in the prompt. This bias is more evident when the taxonomy quality is low and the task is more challenging. To mitigate this, we average the results over multiple runs with randomized positions of the options in our experiments. TnT-LLM 5.2.3 Results. We first calculate the coverage of the LLM-generated taxonomies on the BingChat-Phase1-L-Multi dataset, where both LLM systems achieve very high coverage (>99.5%) on both user intent and conversational domain taxonomies. 5.2.3 Results. BingChat-Phase1-L-Multi We then conduct the accuracy and relevance evaluation tasks to assess the quality of the taxonomies generated by different methods on the small English-only evaluation dataset BingChat-Phase1- S-Eng. We report the inter-rater agreement (Cohen’s Kappa [6] between two raters and Fleiss’ Kappa [7] among multiple raters) in Table 1. The agreement is moderate (𝜅 > 0.4) on intent and domain accuracy as well as intent relevance, while the agreement on domain relevance is fair (Fleiss′𝜅 = 0.379).[3] Interestingly, for the tasks with moderate agreement, the GPT-4 evaluator agrees more with the human majority than the humans do among themselves. This suggests that GPT-4 can be a consistent and reliable evaluator. Figure 4a shows the main results on label accuracy and use case relevance from human evaluations on BingChat-Phase1- S-Eng. We observe our TnT-LLM using GPT-4 outperforms other methods in most cases. Compared to GPT4, we find that GPT-3.5- Turbo tends capture conversation topics (domains) well, but often fails to generate labels that are aligned with the user intent instruction. Likewise, we notice that some embedding methods (ada2 + GPT-4, instructor-xl + GPT-4) perform well in terms of producing accurate domain labels, on par with TnT-LLM instantiated with GPT-3.5-Turbo, but fail to capture the user intent behind the conversations. This is likely because the domain labels reflect the topical theme of the conversations, which can be easily derived from the semantic information captured by unsupervised embeddings, while intent labels require deeper reasoning and understanding of the use-case instruction. With regard to our baselines, we find that GPT-4 consistently outperforms GPT-3.5-Turbo in producing more accurate labels when using the same embedding method for clustering. For the intent use-case, GPT-4 generates more relevant labels than GPT-3.5-Turbo, while the difference is less noticeable for the domain use case; again, this may be because GPT-3.5-Turbo is better at capturing topical information in conversations than reasoning about user intent. Finally, given the high agreement between GPT-4 and human raters on the label accuracy task, we use GPT-4 to evaluate the label accuracy on the larger multilingual dataset BingChat-Phase1- L-Multi (Figure 4b). We observe similar patterns as those in our human evaluation, where our TnT-LLM achieves the highest accuracy, and in particular the instantation that uses GPT-4. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv [2] https://openai.com/blog/new-and-improved-embedding-model [3] Note that these evaluation tasks are cognitively challenging, especially for low-quality taxonomies (e.g., from some baseline methods). Authors:
(1) Mengting Wan, Microsoft Corporation and Microsoft Corporation;
(2) Tara Safavi (Corresponding authors), Microsoft Corporation;
(3) Sujay Kumar Jauhar, Microsoft Corporation;
(4) Yujin Kim, Microsoft Corporation;
(5) Scott Counts, Microsoft Corporation;
(6) Jennifer Neville, Microsoft Corporation;
(7) Siddharth Suri, Microsoft Corporation;
(8) Chirag Shah, University of Washington and Work done while working at Microsoft;
(9) Ryen W. White, Microsoft Corporation;
(10) Longqi Yang, Microsoft Corporation;
(11) Reid Andersen, Microsoft Corporation;
(12) Georg Buscher, Microsoft Corporation;
(13) Dhruv Joshi, Microsoft Corporation;
(14) Nagu Rangan, Microsoft Corporation. Authors: Authors: (1) Mengting Wan, Microsoft Corporation and Microsoft Corporation; (2) Tara Safavi (Corresponding authors), Microsoft Corporation; (3) Sujay Kumar Jauhar, Microsoft Corporation; (4) Yujin Kim, Microsoft Corporation; (5) Scott Counts, Microsoft Corporation; (6) Jennifer Neville, Microsoft Corporation; (7) Siddharth Suri, Microsoft Corporation; (8) Chirag Shah, University of Washington and Work done while working at Microsoft; (9) Ryen W. White, Microsoft Corporation; (10) Longqi Yang, Microsoft Corporation; (11) Reid Andersen, Microsoft Corporation; (12) Georg Buscher, Microsoft Corporation; (13) Dhruv Joshi, Microsoft Corporation; (14) Nagu Rangan, Microsoft Corporation.

TnT-LLM for Automated Taxonomy Generation: Outperforming Clustering Baselines

Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background 2 Background 3 Approach and 3.1 Differential Testing for XML Processors 3 Approach and 3.1 Differential Testing for XML Processors 3.2 XPath Expression Generation 3.2 XPath Expression Generation 3.3 XML Generation 3.3 XML Generation 4 Evaluation 4 Evaluation 4.1 Effectiveness 4.1 Effectiveness 4.2 Efficiency 4.2 Efficiency 4.3 Comparison to the State of the Art 4.3 Comparison to the State of the Art 4.4 Analysis of BaseX Historical Bug Reports 4.4 Analysis of BaseX Historical Bug Reports 5 Related Work 5 Related Work 6 Conclusion, Acknowledgments, and References 6 Conclusion, Acknowledgments, and References 4.2 Efficiency Existing-generator baselines. We considered the only two—to the best of our knowledge—approaches to generate XPath expressions. Neither of them was specifically designed to be combined with a XPath test oracle. XQgen [42] generates XPath queries for micro benchmarking. Its generated predicates only check for sub-element existence. The XQuery generator designed by Todic and Uzelac [41] generates XPath queries for automatically testing index support in DBMSs. Given that indexes apply only to sargable queries (i.e., simple comparisons), the expressions it generates are simple. Both approaches generate XPath expressions based on an XML schema, while XPress generates XPath expressions based on the actual XML document. Based on this, we expect both of them to have low applicability for our differential-testing approach. Given that neither implementations are publicly available, we re-implemented them based on the description in the papers. Existing-generator baselines. Self-constructed baselines. We also constructed our own baselines to investigate the efficiency of the separate components of XPress. XPress has two main components, namely (1) the targeted predicate generation by using the targeted node to refer to existing nodes and attributes and (2) the predicate rectification to avoid empty result sets. To evaluate the effect of the components individually, we enabled them individually to test whether they improve XPress’s bug detection efficiency. Self-constructed baselines. Configurations. We considered four configurations for our selfconstructed baselines. Apart from our proposed approach introduced in Section 3.2 as (1) Targeted, we derive configuration (2) Targeted without Rectification, (3) Untargeted with Rectification, and (4) Untargeted without Rectification. In (2) Targeted without Rectification, we disable the rectification process, which would otherwise ensure targeted node selection. Since selecting a targeted node for predicate generation guidance always requires at least one node in the result set, we stop generating new sections after an empty result set is produced. In (3) Untargeted with Rectification, we generate predicates without using targeted node information to supply parameters that reference existent context and trigger corner cases for function nodes, while keeping the rectification to ensure that at least one node from the candidate set is included in the result set. In (4) Untargeted without rectification, we remove both components to generate predicates randomly, while omitting rectification. Configurations argeted, we derive configuration Targeted without Rectification Untargeted with Rectification Untargeted without Rectification Untargeted with Rectification Untargeted without rectification, Methodology. We set each baseline to run for 24 hours [30]. We repeated each experiment 10 times to account for potential performance deviations, and report the arithmetic mean for all metrics. As our testing target, we selected BaseX 10.4, which is the BaseX version that we first started testing. The reason for selecting BaseX as a representative is that we found most bugs in BaseX and all bugs were fixed, allowing us to determine the number of unique bugs we found in a testing campaign by deduplicating bug-inducing test cases automatically. Specifically, given two bug-inducing test cases, we could determine whether they trigger the same underlying bug by identifying their fix commits; only if their associated fix commit are different, do we consider the bugs unique. This is a best-effort technique, as, for example, one fix commit might address multiple bugs. We disabled the generation of the has-children functions as well as using relative XPath expressions in predicates, as they consistently lead to crashes, triggering known bugs. Methodology Results of existing generators. Neither XQGen nor the Combined XML/XQuery generator found bugs in our experiment. This is expected, as previously proposed approaches were not designed for Results of existing generators. automated testing. As mentioned above, XQGen generates predicates that only check for element existence. The XQuery generator designed by Todic and Uzelac generates simple predicates that include at most one comparison operator. Results of different configurations. As Figure 8 shows, our proposed approach, Targeted outperforms the other configurations. Within 24 hours, it found the most number of unique bugs (namely 12.5). Both configurations with targeted generation clearly outperformed the untargeted approaches, while rectification shows a similar performance in the speed of bug detection. As shown in Table 3, both targeted generation and rectification reduce the testing throughput, as they obtain intermediate results using the XML processor under test. Despite generating only 50% of the number of test cases as compared to (4) Untargeted without Rectification, (1) Targeted detected 20× more bug-inducing test cases and 2× more unique bugs. The results show that selecting a target node to guide the XPath generation process improves testing efficiency significantly. As observed above when discussing the small-scope hypothesis, most of the bugs that we found can be reproduced using a single section, explaining the limited effectiveness of rectification. However, we still believe that rectification is an important component, since without it, bugs requiring multiple sections with non-empty results could hardly be found. Results of different configurations. Code coverage. We collected code coverage for three processors’ core modules for XPress for 24 hours [30] of execution. The result is shown in Table 4. To put the numbers in relation, we collected coverage also for the projects’ test suites; Saxon has no publicly available test suites and is therefore excluded. For the three XML processors, the line coverage ranged from 15% to 20%, and the Code coverage. branch coverage ranged from 10% to 16%. The coverage percentages are low, which is expected. The main reason for low code coverage is that XML processors typically also have other components than XPath processing. Taking BaseX as an example, around 21% of uncovered code was GUI-related, 10% was due to lack of full-text functionality support, and 5% were database commands. In Saxon, as another example, XSLT modules have not been covered. A further 18% uncovered code in BaseX involved unimplemented functions; it would be straightforward to implement many additional ones, such as math functions, but the many functions available would make this a tedious task. In Section 4.4, we detail unsupported XPath features, implementing which might allow us to find more bugs. XPress’s test-case generation process primarily aims at generating semantically valid expressions, which results in low error-checking branch coverage, quantifying which is difficult, as the relevant code is spread throughout the code base. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv Authors:
(1) Shuxin Li, Southern University of Science and Technology China and Work done during an internship at the National University of Singapore (shuxin.li.lv@gmail.com);
(2) Manuel Rigger, National University of Singapore Singapore (rigger@nus.edu.sg). Authors: Authors: (1) Shuxin Li, Southern University of Science and Technology China and Work done during an internship at the National University of Singapore (shuxin.li.lv@gmail.com); (2) Manuel Rigger, National University of Singapore Singapore (rigger@nus.edu.sg).

Finding XPath Bugs in XML Document Processors: Existing-Generator Baselines and More

Table of Links Abstract and 1 Introduction 2. Related Works 2.1. 2D Diffusion Models for 3D Generation 2.2. 3D Generative Models and 2.3. Multi-view Diffusion Models 3. Problem Formulation 3.1. Diffusion Models 3.2. The Distribution of 3D Assets 4. Method and 4.1. Consistent Multi-view Generation 4.2. Cross-Domain Diffusion 4.3. Textured Mesh Extraction 5. Experiments 5.1. Implementation Details 5.2. Baselines 5.3. Evaluation Protocol 5.4. Single View Reconstruction 5.5. Novel View Synthesis and 5.6. Discussions 6. Conclusions and Future Works, Acknowledgements and References 5.2. Baselines We adopt Zero123 [31], RealFusion [38], Magic123 [44], One-2-3-45 [30], Point-E [41], Shap-E [25] and a recent work SyncDreamer [33] as baseline methods. Given an input image, zero123 is capable of generating novel views of arbitrary viewpoints, and it can be incorporated with SDS loss [43] for 3D reconstruction (we adopt the implementation of ThreeStudio [20]). RealFusion [38] and Magic123 [44] leverage Stable Diffusion [47] and SDS loss for single-view reconstruction. One-2-3-45 [30] directly predict SDFs via SparseNeuS [36] by taking the generated multiple images of Zero123 [31]. Point-E [41] and ShapE [25] are 3D generative models trained on a large internal OpenAI 3D dataset, both of which are able to convert a single-view image into a point cloud or an implicit representation. SyncDreamer[33] aims to generate multi-view consistent images from a single image for deriving 3D geometry. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors:
(1) Xiaoxiao Long, The University of Hong Kong, VAST, MPI Informatik and Equal Contributions;
(2) Yuan-Chen Guo, Tsinghua University, VAST and Equal Contributions;
(3) Cheng Lin, The University of Hong Kong with Corresponding authors;
(4) Yuan Liu, The University of Hong Kong;
(5) Zhiyang Dou, The University of Hong Kong;
(6) Lingjie Liu, University of Pennsylvania;
(7) Yuexin Ma, Shanghai Tech University;
(8) Song-Hai Zhang, The University of Hong Kong;
(9) Marc Habermann, MPI Informatik;
(10) Christian Theobalt, MPI Informatik;
(11) Wenping Wang, Texas A&M University with Corresponding authors. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Works 2.1. 2D Diffusion Models for 3D Generation 2.1. 2D Diffusion Models for 3D Generation 2.2. 3D Generative Models and 2.3. Multi-view Diffusion Models 2.2. 3D Generative Models and 2.3. Multi-view Diffusion Models 3. Problem Formulation 3.1. Diffusion Models 3.1. Diffusion Models 3.2. The Distribution of 3D Assets 3.2. The Distribution of 3D Assets 4. Method and 4.1. Consistent Multi-view Generation 4. Method and 4.1. Consistent Multi-view Generation 4.2. Cross-Domain Diffusion 4.2. Cross-Domain Diffusion 4.3. Textured Mesh Extraction 4.3. Textured Mesh Extraction 5. Experiments 5.1. Implementation Details 5.1. Implementation Details 5.2. Baselines 5.2. Baselines 5.3. Evaluation Protocol 5.3. Evaluation Protocol 5.4. Single View Reconstruction 5.4. Single View Reconstruction 5.5. Novel View Synthesis and 5.6. Discussions 5.5. Novel View Synthesis and 5.6. Discussions 6. Conclusions and Future Works, Acknowledgements and References 6. Conclusions and Future Works, Acknowledgements and References 5.2. Baselines We adopt Zero123 [31], RealFusion [38], Magic123 [44], One-2-3-45 [30], Point-E [41], Shap-E [25] and a recent work SyncDreamer [33] as baseline methods. Given an input image, zero123 is capable of generating novel views of arbitrary viewpoints, and it can be incorporated with SDS loss [43] for 3D reconstruction (we adopt the implementation of ThreeStudio [20]). RealFusion [38] and Magic123 [44] leverage Stable Diffusion [47] and SDS loss for single-view reconstruction. One-2-3-45 [30] directly predict SDFs via SparseNeuS [36] by taking the generated multiple images of Zero123 [31]. Point-E [41] and ShapE [25] are 3D generative models trained on a large internal OpenAI 3D dataset, both of which are able to convert a single-view image into a point cloud or an implicit representation. SyncDreamer[33] aims to generate multi-view consistent images from a single image for deriving 3D geometry. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv Authors: (1) Xiaoxiao Long, The University of Hong Kong, VAST, MPI Informatik and Equal Contributions; (2) Yuan-Chen Guo, Tsinghua University, VAST and Equal Contributions; (3) Cheng Lin, The University of Hong Kong with Corresponding authors; (4) Yuan Liu, The University of Hong Kong; (5) Zhiyang Dou, The University of Hong Kong; (6) Lingjie Liu, University of Pennsylvania; (7) Yuexin Ma, Shanghai Tech University; (8) Song-Hai Zhang, The University of Hong Kong; (9) Marc Habermann, MPI Informatik; (10) Christian Theobalt, MPI Informatik; (11) Wenping Wang, Texas A&M University with Corresponding authors. Authors: Authors: (1) Xiaoxiao Long, The University of Hong Kong, VAST, MPI Informatik and Equal Contributions; (2) Yuan-Chen Guo, Tsinghua University, VAST and Equal Contributions; (3) Cheng Lin, The University of Hong Kong with Corresponding authors; (4) Yuan Liu, The University of Hong Kong; (5) Zhiyang Dou, The University of Hong Kong; (6) Lingjie Liu, University of Pennsylvania; (7) Yuexin Ma, Shanghai Tech University; (8) Song-Hai Zhang, The University of Hong Kong; (9) Marc Habermann, MPI Informatik; (10) Christian Theobalt, MPI Informatik; (11) Wenping Wang, Texas A&M University with Corresponding authors.

The Baseline Methods of Wonder3D and What They Mean

Table of Links Abstract and 1 Introduction 2. Related Work 3. Method and 3.1. Architecture 3.2. Loss and 3.3. Implementation Details 4. Data Curation 4.1. Training Dataset 4.2. Evaluation Benchmark 5. Experiments and 5.1. Metrics 5.2. Baselines 5.3. Comparison to SOTA Methods 5.4. Qualitative Results and 5.5. Ablation Study 6. Limitations and Discussion 7. Conclusion and References A. Additional Qualitative Comparison B. Inference on AI-generated Images C. Data Curation Details 5.2. Baselines We consider five state-of-the-art baselines for shape reconstruction, SS3D [4], MCC [63], Point-E [37], Shap-E [22], One-2-3-45 [30] and OpenLRM [17, 18]. SS3D learns implicit shape reconstruction by first pretraining on ShapeNet GT, and then finetuning on real-world single-view images. The finetuning is performed in a category specific way, and then a single unified model is distilled from all category-specific experts. We compare our model with their final distilled model. MCC learns shell occupancy reconstruction using multiview estimated point clouds from CO3D [47]. Their model assumes known depth and intrinsics during inference. To evaluate their model on RGB images, we use the DPTestimated depth and fixed intrinsics as MCC’s input following their pipeline. Point-E is a point cloud diffusion model that generates point clouds from text prompts or RGB images. They additionally train a separate model that converts point clouds into meshes. We compare our model with Point-E by combining their image-to-point and point-to-mesh models. Shap-E is another diffusion model that learns conditioned shape generation from text or images. Different from PointE, Shap-E uses a latent diffusion setup and can directly generate implicit shapes. The final mesh reconstruction are extracted with marching cubes. One-2-3-45 learns implicit shape reconstruction by breaking it down into a generative view synthesis step and a multiview-to-3D reconstruction step. The view synthesis is achieved with Zero-123 [31], a diffusion model that generates novel-view images conditioned on the original images and poses. Based on the synthesized multi-view images, a cost-volume-based module reconstructs the full 3D mesh of the object. LRM is a concurrent work that learns to predict NeRF [36] from single images using transformer-based architecture. Since the authors have not released the code, we use the code and weights from OpenLRM [5]. The mesh is extracted via Marching Cubes [32] from the triplane NeRF. This paper is available on arxiv under CC BY 4.0 DEED license. [5] https://github.com/3DTopia/OpenLRM Authors:
(1) Zixuan Huang, University of Illinois at Urbana-Champaign and both authors contributed equally to this work;
(2) Stefan Stojanov, Georgia Institute of Technology and both authors contributed equally to this work;
(3) Anh Thai, Georgia Institute of Technology;
(4) Varun Jampani, Stability AI;
(5) James M. Rehg, University of Illinois at Urbana-Champaign. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Architecture 3. Method and 3.1. Architecture 3.2. Loss and 3.3. Implementation Details 3.2. Loss and 3.3. Implementation Details 4. Data Curation 4.1. Training Dataset 4.1. Training Dataset 4.2. Evaluation Benchmark 4.2. Evaluation Benchmark 5. Experiments and 5.1. Metrics 5. Experiments and 5.1. Metrics 5.2. Baselines 5.2. Baselines 5.3. Comparison to SOTA Methods 5.3. Comparison to SOTA Methods 5.4. Qualitative Results and 5.5. Ablation Study 5.4. Qualitative Results and 5.5. Ablation Study 6. Limitations and Discussion 6. Limitations and Discussion 7. Conclusion and References 7. Conclusion and References A. Additional Qualitative Comparison A. Additional Qualitative Comparison B. Inference on AI-generated Images B. Inference on AI-generated Images C. Data Curation Details C. Data Curation Details 5.2. Baselines We consider five state-of-the-art baselines for shape reconstruction, SS3D [4], MCC [63], Point-E [37], Shap-E [22], One-2-3-45 [30] and OpenLRM [17, 18]. SS3D learns implicit shape reconstruction by first pretraining on ShapeNet GT, and then finetuning on real-world single-view images. The finetuning is performed in a category specific way, and then a single unified model is distilled from all category-specific experts. We compare our model with their final distilled model. SS3D MCC learns shell occupancy reconstruction using multiview estimated point clouds from CO3D [47]. Their model assumes known depth and intrinsics during inference. To evaluate their model on RGB images, we use the DPTestimated depth and fixed intrinsics as MCC’s input following their pipeline. MCC Point-E is a point cloud diffusion model that generates point clouds from text prompts or RGB images. They additionally train a separate model that converts point clouds into meshes. We compare our model with Point-E by combining their image-to-point and point-to-mesh models. Point-E Shap-E is another diffusion model that learns conditioned shape generation from text or images. Different from PointE, Shap-E uses a latent diffusion setup and can directly generate implicit shapes. The final mesh reconstruction are extracted with marching cubes. Shap-E One-2-3-45 learns implicit shape reconstruction by breaking it down into a generative view synthesis step and a multiview-to-3D reconstruction step. The view synthesis is achieved with Zero-123 [31], a diffusion model that generates novel-view images conditioned on the original images and poses. Based on the synthesized multi-view images, a cost-volume-based module reconstructs the full 3D mesh of the object. One-2-3-45 LRM is a concurrent work that learns to predict NeRF [36] from single images using transformer-based architecture. Since the authors have not released the code, we use the LRM code and weights from OpenLRM [5]. The mesh is extracted via Marching Cubes [32] from the triplane NeRF. OpenLRM This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [5] https://github.com/3DTopia/OpenLRM Authors: (1) Zixuan Huang, University of Illinois at Urbana-Champaign and both authors contributed equally to this work; (2) Stefan Stojanov, Georgia Institute of Technology and both authors contributed equally to this work; (3) Anh Thai, Georgia Institute of Technology; (4) Varun Jampani, Stability AI; (5) James M. Rehg, University of Illinois at Urbana-Champaign. Authors: Authors: (1) Zixuan Huang, University of Illinois at Urbana-Champaign and both authors contributed equally to this work; (2) Stefan Stojanov, Georgia Institute of Technology and both authors contributed equally to this work; (3) Anh Thai, Georgia Institute of Technology; (4) Varun Jampani, Stability AI; (5) James M. Rehg, University of Illinois at Urbana-Champaign.

Introducing ZeroShape's Baselines: The 5 State-of-the-Art Baselines We Considered

Table of Links Abstract and 1 Introduction 2 Related Work 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 3 Hierspeech++ and 3.1 Speech Representations 3.2 Hierarchical Speech Synthesizer 3.3 Text-to-Vec 3.4 Speech Super-resolution 3.5 Model Architecture 4 Speech Synthesis Tasks 4.1 Voice Conversion and 4.2 Text-to-Speech 4.3 Style Prompt Replication 5 Experiment and Result, and Dataset 5.2 Preprocessing and 5.3 Training 5.4 Evaluation Metrics 5.5 Ablation Study 5.6 Zero-shot Voice Conversion 5.7 High-diversity but High-fidelity Speech Synthesis 5.8 Zero-shot Text-to-Speech 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.10 Speech Super-resolution 5.11 Additional Experiments with Other Baselines 6 Limitation and Quick Fix 7 Conclusion, Acknowledgement and References 5.8 Zero-shot Text-to-Speech We compared the zero-shot TTS performance of HierSpeech++ with other baselines: 1) YourTTS, VITS-based end-to-end TTS model, 2) HierSpeech, an end-to-end TTS model using hierarchical VAE, 3) VALL-E-X, a neural codec language models-based multi-lingual zero-shot TTS model and we utilize an unofficial implementation which has an improved audio quality with Vocos decoder, 4) XTTS[12] , a TTS product XTTS v1 from Coqui Corp., and XTTS is built on a open-source TTS model, TorToise [5] which was trained with unprecedented large-scale speech dataset for the first time. For zero-shot TTS, we utilized a noisy speech prompt from the test-clean and test-other subsets of LibriTTS. HierSpeech++ synthesizes the speech with Tttv of 0.333 and Th of 0.333 in TABLE 7 and 8. The results demonstrate that our model is a strong zeroshot TTS model in terms of all subjective and objective metrics. We conducted three MOS experiments for naturalness, prosody, and similarity. Our model beats all models significantly, and our model has even surpassed the groundtruth in terms of naturalness. However, XTTS has a better performance in pMOS, and this means learning prosody requires more datasets to improve expressiveness. Although other models show limitations in synthesizing speech with noisy prompts, our model synthesizes a speech robustly. Furthermore, our model has a better CER and WER than ground-truth, and this also demonstrates the robustness of our model. In summary, all results demonstrate the superiority of our model in naturalness, expressiveness, and robustness for zero-shot TTS. obustness for zero-shot TTS. In addition, we could further improve the zero-shot TTS performance by introducing a style prompt replication (SPR) in the following subsection. Note that we do not apply the SPR in TABLE 2-8. The audio could be upsampled to 48 kHz. Lastly, we could also synthesize noise-free speech even with noisy speech. The details will be described in Section 6. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. [12]. https://github.com/coqui-ai/TTS Authors:
(1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea;
(4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.1 Neural Codec Language Models and 2.2 Non-autoregressive Models 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 2.3 Diffusion Models and 2.4 Zero-shot Voice Cloning 3 Hierspeech++ and 3.1 Speech Representations 3 Hierspeech++ and 3.1 Speech Representations 3.2 Hierarchical Speech Synthesizer 3.2 Hierarchical Speech Synthesizer 3.3 Text-to-Vec 3.3 Text-to-Vec 3.4 Speech Super-resolution 3.4 Speech Super-resolution 3.5 Model Architecture 3.5 Model Architecture 4 Speech Synthesis Tasks 4.1 Voice Conversion and 4.2 Text-to-Speech 4.1 Voice Conversion and 4.2 Text-to-Speech 4.3 Style Prompt Replication 4.3 Style Prompt Replication 5 Experiment and Result, and Dataset 5 Experiment and Result, and Dataset 5.2 Preprocessing and 5.3 Training 5.2 Preprocessing and 5.3 Training 5.4 Evaluation Metrics 5.4 Evaluation Metrics 5.5 Ablation Study 5.5 Ablation Study 5.6 Zero-shot Voice Conversion 5.6 Zero-shot Voice Conversion 5.7 High-diversity but High-fidelity Speech Synthesis 5.7 High-diversity but High-fidelity Speech Synthesis 5.8 Zero-shot Text-to-Speech 5.8 Zero-shot Text-to-Speech 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.9 Zero-shot Text-to-Speech with 1s Prompt 5.10 Speech Super-resolution 5.10 Speech Super-resolution 5.11 Additional Experiments with Other Baselines 5.11 Additional Experiments with Other Baselines 6 Limitation and Quick Fix 6 Limitation and Quick Fix 7 Conclusion, Acknowledgement and References 7 Conclusion, Acknowledgement and References 5.8 Zero-shot Text-to-Speech We compared the zero-shot TTS performance of HierSpeech++ with other baselines: 1) YourTTS, VITS-based end-to-end TTS model, 2) HierSpeech, an end-to-end TTS model using hierarchical VAE, 3) VALL-E-X, a neural codec language models-based multi-lingual zero-shot TTS model and we utilize an unofficial implementation which has an improved audio quality with Vocos decoder, 4) XTTS[12] , a TTS product XTTS v1 from Coqui Corp., and XTTS is built on a open-source TTS model, TorToise [5] which was trained with unprecedented large-scale speech dataset for the first time. For zero-shot TTS, we utilized a noisy speech prompt from the test-clean and test-other subsets of LibriTTS. HierSpeech++ synthesizes the speech with Tttv of 0.333 and Th of 0.333 in TABLE 7 and 8. The results demonstrate that our model is a strong zeroshot TTS model in terms of all subjective and objective metrics. We conducted three MOS experiments for naturalness, prosody, and similarity. Our model beats all models significantly, and our model has even surpassed the groundtruth in terms of naturalness. However, XTTS has a better performance in pMOS, and this means learning prosody requires more datasets to improve expressiveness. Although other models show limitations in synthesizing speech with noisy prompts, our model synthesizes a speech robustly. Furthermore, our model has a better CER and WER than ground-truth, and this also demonstrates the robustness of our model. In summary, all results demonstrate the superiority of our model in naturalness, expressiveness, and robustness for zero-shot TTS. obustness for zero-shot TTS. In addition, we could further improve the zero-shot TTS performance by introducing a style prompt replication (SPR) in the following subsection. Note that we do not apply the SPR in TABLE 2-8. The audio could be upsampled to 48 kHz. Lastly, we could also synthesize noise-free speech even with noisy speech. The details will be described in Section 6. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv [12]. https://github.com/coqui-ai/TTS Authors: (1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author. Authors: Authors: (1) Sang-Hoon Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (2) Ha-Yeong Choi, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (3) Seung-Bin Kim, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea; (4) Seong-Whan Lee, Fellow, IEEE with the Department of Artificial Intelligence, Korea University, Seoul 02841, South Korea and a Corresponding author.

Zero-shot Text-to-Speech: How Does the Performance of HierSpeech++ Fare With Other Baselines?

Table of Links Abstract and Introduction
Related Work
Problem Definition
Method
Experiments
Conclusion and References A. Appendix A.1. Full Prompts and A.2 ICPL Details A. 3 Baseline Details A.4 Environment Details A.5 Proxy Human Preference A.6 Human-in-the-Loop Preference A.3 BASELINE DETAILS To sample trajectories for reward learning, we employ the disagreement sampling scheme from (Lee et al., 2021b) to enhance the training process. This scheme first generates a larger batch of trajectory pairs uniformly at random and then selects a smaller batch with high variance across an ensemble of preference predictors. The selected pairs are used to update the reward model. For a fair comparison, we recorded the number of times PrefPPO queried the oracle human simulator to compare two trajectories and obtain labels during the reward learning process, using this as a measure of the human effort involved. In the proxy human experiment, we set the maximum number of human queries Q to 49, 150, 1.5k, and 15k. Once this limit is reached, the reward model ceases to update, and only the policy model is updated via PPO. Algo. 3 illustrates the pseudocode for reward learning. Algo. 4 illustrates the pseudocode for PrefPPO. Authors:
(1) Chao Yu, Tsinghua University;
(2) Hong Lu, Tsinghua University;
(3) Jiaxuan Gao, Tsinghua University;
(4) Qixin Tan, Tsinghua University;
(5) Xinting Yang, Tsinghua University;
(6) Yu Wang, with equal advising from Tsinghua University;
(7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute;
(8) Eugene Vinitsky, with equal advising from New York University (zoeyuchao@gmail.com). This paper is available on arxiv under CC 4.0 license. Table of Links Abstract and Introduction Related Work Problem Definition Method Experiments Conclusion and References Abstract and Introduction Abstract and Introduction Related Work Related Work Problem Definition Problem Definition Method Method Experiments Experiments Conclusion and References Conclusion and References A. Appendix A. Appendix A.1. Full Prompts and A.2 ICPL Details A.1. Full Prompts and A.2 ICPL Details A. 3 Baseline Details A. 3 Baseline Details A.4 Environment Details A.4 Environment Details A.5 Proxy Human Preference A.5 Proxy Human Preference A.6 Human-in-the-Loop Preference A.6 Human-in-the-Loop Preference A.3 BASELINE DETAILS To sample trajectories for reward learning, we employ the disagreement sampling scheme from (Lee et al., 2021b) to enhance the training process. This scheme first generates a larger batch of trajectory pairs uniformly at random and then selects a smaller batch with high variance across an ensemble of preference predictors. The selected pairs are used to update the reward model. For a fair comparison, we recorded the number of times PrefPPO queried the oracle human simulator to compare two trajectories and obtain labels during the reward learning process, using this as a measure of the human effort involved. In the proxy human experiment, we set the maximum number of human queries Q to 49, 150, 1.5k, and 15k. Once this limit is reached, the reward model ceases to update, and only the policy model is updated via PPO. Algo. 3 illustrates the pseudocode for reward learning. Algo. 4 illustrates the pseudocode for PrefPPO. Authors: (1) Chao Yu, Tsinghua University; (2) Hong Lu, Tsinghua University; (3) Jiaxuan Gao, Tsinghua University; (4) Qixin Tan, Tsinghua University; (5) Xinting Yang, Tsinghua University; (6) Yu Wang, with equal advising from Tsinghua University; (7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute; (8) Eugene Vinitsky, with equal advising from New York University (zoeyuchao@gmail.com). Authors: Authors: (1) Chao Yu, Tsinghua University; (2) Hong Lu, Tsinghua University; (3) Jiaxuan Gao, Tsinghua University; (4) Qixin Tan, Tsinghua University; (5) Xinting Yang, Tsinghua University; (6) Yu Wang, with equal advising from Tsinghua University; (7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute; (8) Eugene Vinitsky, with equal advising from New York University ( zoeyuchao@gmail.com ). zoeyuchao@gmail.com This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

ICPL Baseline Methods: Disagreement Sampling and PrefPPO for Reward Learning

Authors:
(1) Zhaoqing Wang, The University of Sydney and AI2Robotics;
(2) Xiaobo Xia, The University of Sydney;
(3) Ziye Chen, The University of Melbourne;
(4) Xiao He, AI2Robotics;
(5) Yandong Guo, AI2Robotics;
(6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence;
(7) Tongliang Liu, The University of Sydney. Table of Links Abstract and 1. Introduction 2. Related works 3. Method and 3.1. Problem definition 3.2. Baseline and 3.3. Uni-OVSeg framework 4. Experiments 4.1. Implementation details 4.2. Main results 4.3. Ablation study 5. Conclusion 6. Broader impacts and References A. Framework details B. Promptable segmentation C. Visualisation 3.2. Baseline We introduce a straightforward baseline using the knowledge of image-text and image-mask pairs. Specifically, we employ a CLIP model as the visual and text encoder, which is trained on a large amount of image-text pairs. Afterward, we use the image-mask pairs to obtain a branch of mask generation, predicting a set of binary masks. To perform open-vocabulary segmentation, we crop and pool the CLIP image features based on these predicted masks, which are further classified by the CLIP text embeddings. Although this straightforward baseline enables open-vocabulary segmentation, it exhibits an obvious knowledge gap between the image-level and pixel-level tasks. 3.3. Uni-OVSeg framework Overview. An overview of our framework Uni-OVSeg, for weakly-supervised open-vocabulary segmentation is illustrated in Fig. 2. On a macro level, Uni-OVSeg contains a CLIP model to extract features of both images and text descriptions. With the image-mask pairs, a branch of mask generation, including a visual prompt encoder, a pixel decoder and a mask decoder, is employed to predict a set of binary masks of an input image. With the image-text pairs, mask-text bipartite matching is used to exploit confident pairs between predicted masks and entities in text descriptions. Afterward, we adopt a multi-scale feature adapter to enhance the mask-wise visual embeddings, which are further aligned with associated entity embeddings based on the confident pairs. Finally, we perform open-vocabulary segmentation with the above-mentioned parts. More details can be found in Appendix A. Mask-text alignment. To enable the model categorising the predicted masks from an open set of vocabulary, given the image-text pairs, we build the correspondence between objects in the image and entities in the text description. Once a set of binary masks m are generated from the input image, we obtain the region embeddings ri by employing a mask pooling layer P and the CLIP visual projector Fv, Open-vocabulary inference. During inference, given the test categories Ctest, we conduct prompt engineering [21] and use the CLIP text encoder to extract text embeddings 1https://openai.com/chatgpt for open-vocabulary segmentation. For each input image, we input an uniform point grid as the visual prompt to predict a set of binary masks, and compute the cosine similarity with the text embeddings to predict the category with the maximum similarity as the label of the corresponding mask. This paper is available on arxiv under CC BY 4.0 DEED license. [1] https://openai.com/chatgpt Authors: (1) Zhaoqing Wang, The University of Sydney and AI2Robotics; (2) Xiaobo Xia, The University of Sydney; (3) Ziye Chen, The University of Melbourne; (4) Xiao He, AI2Robotics; (5) Yandong Guo, AI2Robotics; (6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence; (7) Tongliang Liu, The University of Sydney. Authors: Authors: (1) Zhaoqing Wang, The University of Sydney and AI2Robotics; (2) Xiaobo Xia, The University of Sydney; (3) Ziye Chen, The University of Melbourne; (4) Xiao He, AI2Robotics; (5) Yandong Guo, AI2Robotics; (6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence; (7) Tongliang Liu, The University of Sydney. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related works 2. Related works 3. Method and 3.1. Problem definition 3. Method and 3.1. Problem definition 3.2. Baseline and 3.3. Uni-OVSeg framework 3.2. Baseline and 3.3. Uni-OVSeg framework 4. Experiments 4.1. Implementation details 4.1. Implementation details 4.2. Main results 4.2. Main results 4.3. Ablation study 4.3. Ablation study 5. Conclusion 5. Conclusion 6. Broader impacts and References 6. Broader impacts and References A. Framework details A. Framework details B. Promptable segmentation B. Promptable segmentation C. Visualisation C. Visualisation 3.2. Baseline We introduce a straightforward baseline using the knowledge of image-text and image-mask pairs. Specifically, we employ a CLIP model as the visual and text encoder, which is trained on a large amount of image-text pairs. Afterward, we use the image-mask pairs to obtain a branch of mask generation, predicting a set of binary masks. To perform open-vocabulary segmentation, we crop and pool the CLIP image features based on these predicted masks, which are further classified by the CLIP text embeddings. Although this straightforward baseline enables open-vocabulary segmentation, it exhibits an obvious knowledge gap between the image-level and pixel-level tasks. 3.3. Uni-OVSeg framework Overview . An overview of our framework Uni-OVSeg, for weakly-supervised open-vocabulary segmentation is illustrated in Fig. 2. On a macro level, Uni-OVSeg contains a CLIP model to extract features of both images and text descriptions. With the image-mask pairs, a branch of mask generation, including a visual prompt encoder, a pixel decoder and a mask decoder, is employed to predict a set of binary masks of an input image. With the image-text pairs, mask-text bipartite matching is used to exploit confident pairs between predicted masks and entities in text descriptions. Afterward, we adopt a multi-scale feature adapter to enhance the mask-wise visual embeddings, which are further aligned with associated entity embeddings based on the confident pairs. Finally, we perform open-vocabulary segmentation with the above-mentioned parts. More details can be found in Appendix A. Overview Mask-text alignment. To enable the model categorising the predicted masks from an open set of vocabulary, given the image-text pairs, we build the correspondence between objects in the image and entities in the text description. Once a set of binary masks m are generated from the input image, we obtain the region embeddings ri by employing a mask pooling layer P and the CLIP visual projector Fv, Mask-text alignment. Open-vocabulary inference. During inference, given the test categories Ctest, we conduct prompt engineering [21] and use the CLIP text encoder to extract text embeddings 1https://openai.com/chatgpt for open-vocabulary segmentation. For each input image, we input an uniform point grid as the visual prompt to predict a set of binary masks, and compute the cosine similarity with the text embeddings to predict the category with the maximum similarity as the label of the corresponding mask. Open-vocabulary inference. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] https://openai.com/chatgpt

he Baseline and Uni-OVSeg Framework for Open-Vocabulary Segmentation

Authors:
(1) Zhaoqing Wang, The University of Sydney and AI2Robotics;
(2) Xiaobo Xia, The University of Sydney;
(3) Ziye Chen, The University of Melbourne;
(4) Xiao He, AI2Robotics;
(5) Yandong Guo, AI2Robotics;
(6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence;
(7) Tongliang Liu, The University of Sydney. Table of Links Abstract and 1. Introduction 2. Related works 3. Method and 3.1. Problem definition 3.2. Baseline and 3.3. Uni-OVSeg framework 4. Experiments 4.1. Implementation details 4.2. Main results 4.3. Ablation study 5. Conclusion 6. Broader impacts and References A. Framework details B. Promptable segmentation C. Visualisation 3. Method In this section, we first define the problem of openvocabulary segmentation in Sec. 3.1. We then introduce a straightforward baseline in Sec. 3.2. Finally, we present our proposed Uni-OVSeg framework in Sec. 3.3, including an overview, mask generation, mask-text alignment, and open-vocabulary inference. 3.1. Problem definition This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Zhaoqing Wang, The University of Sydney and AI2Robotics; (2) Xiaobo Xia, The University of Sydney; (3) Ziye Chen, The University of Melbourne; (4) Xiao He, AI2Robotics; (5) Yandong Guo, AI2Robotics; (6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence; (7) Tongliang Liu, The University of Sydney. Authors: Authors: (1) Zhaoqing Wang, The University of Sydney and AI2Robotics; (2) Xiaobo Xia, The University of Sydney; (3) Ziye Chen, The University of Melbourne; (4) Xiao He, AI2Robotics; (5) Yandong Guo, AI2Robotics; (6) Mingming Gong, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence; (7) Tongliang Liu, The University of Sydney. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related works 2. Related works 3. Method and 3.1. Problem definition 3. Method and 3.1. Problem definition 3.2. Baseline and 3.3. Uni-OVSeg framework 3.2. Baseline and 3.3. Uni-OVSeg framework 4. Experiments 4.1. Implementation details 4.1. Implementation details 4.2. Main results 4.2. Main results 4.3. Ablation study 4.3. Ablation study 5. Conclusion 5. Conclusion 6. Broader impacts and References 6. Broader impacts and References A. Framework details A. Framework details B. Promptable segmentation B. Promptable segmentation C. Visualisation C. Visualisation 3. Method In this section, we first define the problem of openvocabulary segmentation in Sec. 3.1. We then introduce a straightforward baseline in Sec. 3.2. Finally, we present our proposed Uni-OVSeg framework in Sec. 3.3, including an overview, mask generation, mask-text alignment, and open-vocabulary inference. 3.1. Problem definition This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Stories, News & Mentions about BASELIME

Categories

Trending Topics

Stories, News & Mentions about BASELIME

Baselime’s Stories on HackerNoon

Categories

Trending Topics